Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 codex-bot
|
fe0f08a550 |
@@ -50,6 +50,7 @@ jobs:
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','cvf','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','www-archive','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','aamas-proceedings','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','dblp-family','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','aamas-proceedings','--year','2025','--snapshots',str(src/'tests/fixtures/aamas2025'),'--out',str(root/'installed-aamas2025')],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','iclr-archive','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','acl','--help'],cwd=outside,check=True)
|
||||
|
||||
@@ -6,8 +6,6 @@ Paper Atlas 将 arXiv、AI 会议论文集及公开投稿决定整理到可追
|
||||
|
||||
项目仍在建设中。候选范围包含 43 个会议（含 ACM MM、MLSys），尚未完成全部历史数据和最终去重复核。实际进度以数据库、来源清单和覆盖报告为准，而不是仓库文件数量。
|
||||
|
||||
当前主动补漏从 **2020 年**起；2000 年起已发现的名单、缺口与待办保存在[补漏恢复清单](docs/coverage/backfill-handoff-2026-11-10.md)，下次从这里接续。已有旧数据保留，范围排除不算补齐。
|
||||
|
||||
历史 ICLR 2014–2016 官方录用页由 `collect iclr-archive` 适配，保留主会/workshop及重复展示证据；运行和身份核验边界见 [历史档案说明](docs/coverage/iclr-archive-2014-2016.md)。
|
||||
|
||||
## 项目结构
|
||||
@@ -72,6 +70,8 @@ EWSL、EPS、ECP、EWSP 的[具体前身卷与完整快照要求](docs/coverage/
|
||||
|
||||
AAMAS 的[官方论文集采集器](docs/aamas-proceedings.md)支持已核验的 2012、2025 版式，可从缓存或已校验快照采集元数据。2025 年目录含 476 条非 keynote 出版物，保留完整作者和机构，HTML 未提供摘要时留空；发表形式与投稿 track 分开，不根据主题推定主会轨道或研究类型。
|
||||
|
||||
旧 DBLP 候选文件可能漏掉新命名空间。`collect dblp-family` 可从已校验的固定压缩快照和完整目录中[恢复指定 family 的原始候选](docs/dblp-family-recovery.md)，例如 `ifaamas`；恢复文件保留卷、keynote 和 workshop 等原始条目，需另做来源范围与身份核验。
|
||||
|
||||
`paper-atlas collect --help` 查看来源；`paper-atlas import --help` 查看导入方式。完整批次步骤见[使用指南](docs/usage.md)。
|
||||
|
||||
共享实例名为 `foundry-postgres`，数据位于 `foundry_data` 库的 `research` schema；其他项目可按权限复用同一组表。库名通过 libpq service 配置、schema 通过 search_path 或 `--pg-schema` 指定，不写死在查询中。PostgreSQL 凭据放在服务器权限为 `0600` 的 libpq service 文件中；命令行只使用服务名称。`paper-atlas maintain postgres --help` 提供迁移入口：先完整核验 21 张业务表，再发布目标 schema。SQLite FTS5 内部表由 PostgreSQL GIN 索引替代。切换后 SQLite 为只读备份；`import records`、`import arxiv` 和 `maintain backfill` 使用 `--pg-service foundry-research` 写入主库，整个写入会话只允许一个协调写入者，会议文件失败时整体回滚并保留失败批次记录。
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
# 会议覆盖与公开决定核查
|
||||
|
||||
**当前工作范围：2020 年起主动补漏。** 2000 年起已发现资料保留，恢复工作从 [补漏范围与恢复清单（2026-11-10）](backfill-handoff-2026-11-10.md) 开始；下文旧日期快照不改写，较早年份排除不算补齐。
|
||||
|
||||
这是 2026-11-08 的首份核查基线，**完整逐年政策普查和最终去重仍未完成**。收录 43 个目标，不能把“有记录”当作“完整覆盖”。结构化清单见 [inventory-2026-11-08.json](inventory-2026-11-08.json)。清单中的 `policy_checks` 均限定所列年份和轨道；空数组表示尚未核实，绝不表示没有公开拒稿。
|
||||
|
||||
## 快照与复核
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,52 +0,0 @@
|
||||
# 补漏范围与恢复清单（2026-11-10）
|
||||
|
||||
用户确定：**后续主动补漏从 2020 年起**。2000 年起已经发现的名单、缺口、资料组合和身份核查记录保留在项目中，供下次接着做；已有旧数据继续保留。停止补采的年份标为范围外或延期，不算作已补齐。
|
||||
|
||||
本页是下一次工作的入口，结构化记录见 [backfill-handoff-2026-11-10.json](backfill-handoff-2026-11-10.json)。它保存 43 个会议的 2000–2026 已观察年份、缺年、官方目录文件入口、近期待办与恢复证据。旧的日期快照不改写，也不能当实时数据库。
|
||||
|
||||
## 当前范围和年份差集
|
||||
|
||||
依据生产只读快照 `2026-11-09 16:26:41.688509+00:00`，原全历史口径为 1266 个目标年份槽位：1208 已观察、43 个历史缺年、14 个 2026 缺年，以及 1 个 SysML 2018 非存档例外。原 135 个历史缺口中有 92 个通过数据补入解决；其余 43 个现在停止补采，不能因此宣称全部解决。
|
||||
|
||||
- **2020–2025**：没有整年完全缺记录的目标槽位，但逐篇、track、摘要、公开负面决定仍不完整。
|
||||
- **2026**：NeurIPS、ECCV、RecSys、CIKM、ICDM、SDM、IROS、RSS、CoRL、ICAPS、AAMAS、ICASSP、ISMIR、ACM MM，共 14 个槽位。须逐个核查是否已经公开出版；不能将尚未发布的资料说成漏采。
|
||||
- **2000–2019 延期线索**：WWW 2000、ICAPS 2001 两个整年缺口；官方历届记录分别见 [WWW](www-editions-2026-11-08.json)、[ICAPS](icaps-editions-2026-11-08.json)。SysML 2018 是非存档首届，见 [MLSys](mlsys-editions-2026-11-08.json)，不能制造正式出版记录。
|
||||
- **2000 年前**：剩余 41 个历史缺年停止补采。旧来源文件继续保留作证据，不再作为当前工作队列。
|
||||
|
||||
以上只判断目标会议年份是否至少有一条投稿记录，绝不等于目录完整、独立论文数量或最终去重验收。
|
||||
|
||||
## 近期资料与恢复位置
|
||||
|
||||
服务器采集根目录为 `/data/paper-index/collected`，本地证据备份为 `/Users/zzyang/Documents/Codex/2026-11-06/paper-index`。这些目录提供大体积原始文件，仓库 JSON 提供路径、状态和可用哈希；恢复前先验证文件存在、哈希和生产实际状态。
|
||||
|
||||
| 资料 | 当前状态 | 下一步与证据入口 |
|
||||
| --- | --- | --- |
|
||||
| AAMAS 2026 | 官方目录 643 条，3 个 keynote 排除；640 个非 keynote 中含 1 个明确撤稿。尚未导入 | `aamas2026-primary-evidence-v1/` 保存官方 HTML、回执、逐项目录、审计和校验清单。八个标题差异/撤稿前缀条目需核查。先扩展正式 collector 并远程验证，再做新 PostgreSQL 身份预检。 |
|
||||
| CVPR 2026 Findings | 941 条摘要已核验，尚未导入 | `cvf-2026-verified/recovery-audit.json`；原 404 与 RVLF/FVLF 路径更正证据保留。先核对 manifests、track 和实体关系。 |
|
||||
| AAMAS 2025 | 476 个非 keynote 来源中，240 已验收、236 待身份核查 | `aamas2025-import225-v1/` 与 `aamas2025-import15-v1/` 是已消费批次，禁止重放。15 条最近验收报告哈希见 JSON。余下资料从当前身份候选与新只读基线继续。 |
|
||||
| RSS 2025 | 163 条目录中 162 已验收，`2025/p117` 待核查 | 保留 spinal/RMCC 关系问题；查看工作状态键 `rss2025_held7_primary_evidence_v1`，不能重跑已消费的 6/88/54/14 批次。 |
|
||||
| 近期公开负面决定首批 | 1625 条已验收：1624 rejected、1 withdrawn | `public-negative-postgresql-import-postcheck-v1.json`；仅该批次验收，不是公开拒稿全集，不要重放。 |
|
||||
|
||||
AAMAS 2026 官方目录分区：research 338、extended abstracts 185、AAAI 19、Blue Sky 17、doctoral 34、demo 44、JAAMAS 3，另有 keynote 3。JAAMAS 是期刊展示关系，不能凭目录再造一次独立审稿录用。官方撤稿条目在 research 分区，不能把全目录都标为当前 accepted。
|
||||
|
||||
AAAI 转投政策有年份冲突：[详细征稿页](https://cyprusconferences.org/aamas2026/call-for-papers-aaai-track/) 指 AAAI-26 rejected，[总览](https://cyprusconferences.org/aamas2026/calls/) 写 AAAI-25。保留两份证据；19 个 AAAI track 条目不能直接转成 19 条有原始 note 的拒稿决定。原投稿 ID、逐篇决定和版本关系仍需核实。
|
||||
|
||||
## 2000–2019 已发现但延期的身份与名单工作
|
||||
|
||||
- **AAMAS 2012**：官方 334 条，319 已独立验收、15 个身份条目保留待查。验收报告 `aamas2012-import-v1/postgresql-postcheck-v1.json`，哈希在 JSON；已验收 319 条及被替代的 316 条计划均不得重放。来源布局与用法见 [AAMAS 适配说明](../aamas-proceedings.md)。
|
||||
- **UAI 2008**：官方 programme 与归档资料确认了四篇独立论文，但部分原始 DBLP 键的标题、作者、页码、出版链接对应仍有冲突。JSON 保留四组候选、arXiv ID、官方题名及归档报告哈希。之前仅根据页码推断题名/作者错误的结论已撤回，未应用纠正；恢复时不得据此覆盖或合并。
|
||||
- **其他名单**：本目录全部 `*-editions-2026-11-08.json` 保留各会议官方年份及来源；[历史缺口](historical-gaps-2026-11-09.md)、[来源候选](historical-source-candidates-2026-11-09.json)、[来源范围](historical-source-scope-2026-11-09.md)、[主系列](historical-primary-series-2026-11-09.md)、[前身会议](historical-predecessors-2026-11-09.md)、[ICLR 档案](iclr-archive-2014-2016.md)继续留档。这些旧报告的“缺失”可能已被后续导入解决，恢复时必须对照当前状态，不能直接当可执行队列。
|
||||
|
||||
## 公开拒稿资料不能遗漏
|
||||
|
||||
结构化清单保留最近已查的政策 URL、限定年份和可用性结论。优先继续 UAI 2024–2026 的 opt-in 公开拒稿来源、EMNLP 2023 的公开集合范围及其他目标的逐年核查；平台使用 OpenReview 不能证明 rejected 公开。访问受限如实记录，不绕过 Challenge。
|
||||
|
||||
保留 rejected、desk rejected、withdrawn、retracted、resubmit 的区别；评审建议不等于最终决定。COLM 2026 的不公开政策只适用于已核实年份，不能推广到其他会议。PaperCopilot 与原 OpenReview、镜像间有重叠，候选计数不能相加。论文先被 A 拒绝再被 B 录用时，两次投稿、决定、来源与证据全部保留。
|
||||
|
||||
## 下次启动顺序
|
||||
|
||||
1. 读取本页和 JSON，再核实服务器工作状态、明确的 systemd invocation 与终态报告。权威工作状态 `/Users/zzyang/Documents/Codex/2026-11-06/paper-index/work-state.json` 中部分旧字段会过期；以最新验收和实际查询为准。
|
||||
2. 当前主动范围为 2020 起，2000–2019 只恢复线索，不自动启动其采集。先检查新来源是否已导入，验收过的队列禁止重放。
|
||||
3. 使用现有 `foundry-postgres` 的 `foundry_data.research`。当前已验证 writer 为提交 `714f745bd5c3fbc742be1efae0b31ab37a55e21f` 的固定 release；恢复时重新确认，不使用旧 `/data/paper-index/code` 替代。服务配置仅由服务器受保护文件读取，不复制密码。
|
||||
4. 对待导入资料做新只读身份预检，处理稳定标识、短标题、作者/版本差异、track 与跨会历史，再生成小范围增量队列。每批独立验收，SQLite 只读备份保持保护。
|
||||
5. 编译、测试或 wheel 构建仅在已确认 `ces-control:host` runner 执行。代码按工作分支交付 PR，等待用户手动合并。范围内导入结束后仍须全库去重、近似标题/作者冲突、来源-年份-track、摘要、约束、实际全文查询验收；不能用单批导入或 CI 成功替代。
|
||||
@@ -0,0 +1,37 @@
|
||||
# DBLP family 候选恢复
|
||||
|
||||
DBLP 的会议命名空间会变化。旧候选过滤器包含 `atal`、`aamas`，遗漏了 `ifaamas`；2026-11-01 固定快照的完整目录实际已有 `conf/ifaamas/2025` 和 `conf/ifaamas/2026`。新目录盘点已把 `ifaamas` 纳入广泛候选范围，已有快照也可按需恢复，无需重新下载或重建完整目录。
|
||||
|
||||
## 运行
|
||||
|
||||
在已有元数据的服务器运行：
|
||||
|
||||
```sh
|
||||
paper-atlas collect dblp-family \
|
||||
--archive /data/paper-index/raw/dblp-2026-11-01.xml.gz \
|
||||
--dtd /data/paper-index/raw/dblp-2023-06-28.dtd \
|
||||
--catalogue /data/paper-index/collected/dblp/dblp-catalogue.sqlite \
|
||||
--family ifaamas \
|
||||
--out /data/paper-index/collected/ifaamas-recovered
|
||||
```
|
||||
|
||||
`--family` 可以重复，但仅接受项目明确列出的候选 family。`--out` 必须尚不存在。同名输出由文件锁防止并发恢复；任务运行中应观测原任务，不能因为日志间隔长就重复启动。完整扫描会定期打印进度。
|
||||
|
||||
命令重新验证固定压缩包 SHA256、DTD MD5 与大小，并只读打开目录。目录旁的 `dblp-ai-candidates.manifest.json` 必须属于同一个快照和 DTD，包含完整记录类型计数。旧候选文件不作为完整性依据，也不会被改写。
|
||||
|
||||
恢复扫描使用项目原有 XML 流式解析器，仅允许已校验的本地 DTD，不联网读取 XML 实体。最终会比对完整快照的记录总数、逐 family 数量和全部选中卷的原始元数据。全部一致后才原子发布输出目录；失败时不发布半成品，也不覆盖既有文件。排查明确终止的失败任务后，可用尚不存在的输出目录重新运行。
|
||||
|
||||
## 输出与后续流程
|
||||
|
||||
- `dblp-family-candidates.jsonl`：原始 `inproceedings` 和 `proceedings`，保留书目 key、原始作者、标题、年份、卷引用、出版链接等字段。
|
||||
- `recovery-manifest.json`：快照、DTD、目录凭据哈希，完整扫描数量、逐 family 数量、选中卷、旧候选文件的已知数量及输出哈希。状态为 `recovered_raw_candidates_pending_scope_review`。
|
||||
|
||||
恢复结果没有新增规范化的 `venue`、`track` 或决定标签，也不是 `import records` 的输入队列。`ifaamas` 的整卷可能混有研究论文、扩展摘要、keynote、博士生、演示和 JAAMAS 展示。目录过滤修复不会把这些条目统一标成 main；规范化的范围映射仍需第一方逐栏目证据。
|
||||
|
||||
后续先用官方目录核对标题、页码、作者及发表形式，再查数据库稳定标识和候选实体，保留每次投稿、决定和标签来源。只有审核后的新增分区才能原生导入并独立验收；已验收的来源记录不能重放。单个年份有记录不代表逐篇、track、摘要或公开拒稿覆盖完整。
|
||||
|
||||
## 2026-11-09 核查实例
|
||||
|
||||
对既有固定快照的独立原始元数据恢复发现 `ifaamas` 有 1,122 条出版物候选及 2 卷：2025 年 479 条、2026 年 643 条。2025 年的 479 条标题与起始页均对应官方目录，其中 3 条 keynote 应排除，余下 476 条出版物需逐项处理。该数字包含已经验收的 225 条来源，不是 1,122 篇可新增论文。
|
||||
|
||||
此实例说明过滤遗漏的影响，不代表恢复命令已完成生产导入。2026 年仍需核验官方出版清单、栏目、身份及逐次决定；缺少录用记录不能推断为拒稿。最终全库去重、覆盖和查询审计继续单独进行。
|
||||
@@ -27,6 +27,7 @@ COMMANDS = {
|
||||
('collect','aamas-proceedings'): ('paper_atlas.collectors.aamas', {}, '按官方目录区分 AAMAS 发表形式和摘要'),
|
||||
('collect','www-archive'): ('paper_atlas.collectors.www_archive', {}, '核验早期WWW官方档案并分开待核查条目'),
|
||||
('collect','dblp'): ('paper_atlas.collectors.dblp_records', {}, '核对并整理 DBLP 历史发表记录'),
|
||||
('collect','dblp-family'): ('paper_atlas.collectors.dblp_recovery', {}, '从固定快照恢复指定 DBLP family 的原始候选'),
|
||||
('catalogue','dblp'): ('paper_atlas.collectors.dblp', {}, '盘点 DBLP 历史会议目录'),
|
||||
('import','arxiv'): ('paper_atlas.ingestion.arxiv', {}, '导入 arXiv 元数据 ZIP'),
|
||||
('import','records'): ('paper_atlas.ingestion.records', {}, '导入核验过的会议文件'),
|
||||
|
||||
@@ -20,7 +20,7 @@ DTD_MD5='96e4283e71886dec2173718957293648'
|
||||
DTD_NAMES={'dblp.dtd','dblp-2023-06-28.dtd'}
|
||||
RELEASE='https://doi.org/10.4230/dblp.xml.2026-11-01'
|
||||
# Broad discovery candidates, including historical names; not a venue mapping.
|
||||
FAMILIES=set('nips icml iclr colm colt aistats uai clear acl emnlp naacl eacl coling conll cvpr iccv eccv aaai ijcai ecai kdd sigir www wsdm recsys cikm icdm sdm ecml pkdd ecmlpkdd icra iros rss corl aips icaps ecp ewsp kr atal aamas icassp interspeech eurospeech icslp ismir miccai fat facct lrec mm mlsys'.split())
|
||||
FAMILIES=set('nips icml iclr colm colt aistats uai clear acl emnlp naacl eacl coling conll cvpr iccv eccv aaai ijcai ecai kdd sigir www wsdm recsys cikm icdm sdm ecml pkdd ecmlpkdd icra iros rss corl aips icaps ecp ewsp kr atal aamas ifaamas icassp interspeech eurospeech icslp ismir miccai fat facct lrec mm mlsys'.split())
|
||||
|
||||
|
||||
def records(archive,dtd_bytes):
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
"""Recover raw family candidates from a pinned snapshot and its existing catalogue.
|
||||
|
||||
The old candidate file is not an exhaustive source: earlier discovery filters
|
||||
can omit an entire namespace. This command checks the complete snapshot and
|
||||
catalogue, without assigning venues, tracks, acceptance or paper identities.
|
||||
"""
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from contextlib import closing
|
||||
import fcntl
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sqlite3
|
||||
import tempfile
|
||||
|
||||
from paper_atlas.collectors.dblp import DTD_MD5, FAMILIES, RELEASE, XML_SHA, family, records
|
||||
from paper_atlas.db.submissions import timestamp
|
||||
from paper_atlas.io.snapshots import digest_file
|
||||
|
||||
|
||||
def catalogue_evidence(catalogue, families, expected_sha, expected_dtd_md5):
|
||||
manifest_path = catalogue.with_name('dblp-ai-candidates.manifest.json')
|
||||
manifest = json.loads(manifest_path.read_text())
|
||||
if (manifest.get('snapshot_sha256') != expected_sha
|
||||
or manifest.get('dtd_md5') != expected_dtd_md5):
|
||||
raise ValueError('Catalogue belongs to a different snapshot or DTD')
|
||||
totals = manifest.get('record_types', {})
|
||||
if (set(totals) != {'proceedings', 'inproceedings'}
|
||||
or any(type(n) is not int or n < 0 for n in totals.values())
|
||||
or type(manifest.get('records_scanned')) is not int
|
||||
or sum(totals.values()) != manifest['records_scanned']):
|
||||
raise ValueError('Incomplete catalogue record totals')
|
||||
counts = Counter()
|
||||
volumes = {}
|
||||
# The catalogue is evidence, never a writable recovery destination.
|
||||
with closing(sqlite3.connect(catalogue.resolve().as_uri() + '?mode=ro', uri=True)) as c:
|
||||
for name in families:
|
||||
found = c.execute('SELECT record_type,count FROM family_counts WHERE family=?',
|
||||
(name,)).fetchall()
|
||||
if not found or not any(n for _, n in found):
|
||||
raise ValueError('Family absent from complete catalogue: ' + name)
|
||||
for kind, count in found:
|
||||
if kind not in totals or type(count) is not int or count < 0:
|
||||
raise ValueError('Invalid catalogue family count')
|
||||
counts[(name, kind)] = count
|
||||
for key, raw in c.execute('SELECT key,metadata_json FROM proceedings WHERE family=?',
|
||||
(name,)):
|
||||
row = json.loads(raw)
|
||||
if (family(key) != name or row.get('key') != key
|
||||
or row.get('record_type') != 'proceedings'):
|
||||
raise ValueError('Invalid catalogue proceedings evidence')
|
||||
volumes[key] = row
|
||||
if len(volumes) != sum(counts[(name, 'proceedings')] for name in families):
|
||||
raise ValueError('Catalogue proceedings count differs from volume records')
|
||||
return manifest, digest_file(manifest_path)['sha256'], counts, volumes
|
||||
|
||||
|
||||
def recover(archive, dtd, catalogue, out, families, *, expected_sha=XML_SHA,
|
||||
expected_dtd_md5=DTD_MD5):
|
||||
families = list(families)
|
||||
if (not families or any(not isinstance(name, str)
|
||||
or not re.fullmatch(r'[a-z][a-z0-9-]*', name)
|
||||
or name not in FAMILIES for name in families)):
|
||||
raise ValueError('Select an explicitly supported discovery family')
|
||||
families = sorted(set(families))
|
||||
archive, dtd, catalogue = Path(archive), Path(dtd), Path(catalogue)
|
||||
output = Path(out).absolute()
|
||||
if output.exists() or output.is_symlink():
|
||||
raise ValueError('Recovery output must not already exist')
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
lock_path = output.parent / ('.' + output.name + '.recovery.lock')
|
||||
with lock_path.open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
if output.exists() or output.is_symlink():
|
||||
raise ValueError('Recovery output appeared before scan')
|
||||
actual = digest_file(archive)
|
||||
if actual['sha256'] != expected_sha:
|
||||
raise ValueError('DBLP snapshot checksum mismatch')
|
||||
if dtd.stat().st_size > 128 * 1024 or digest_file(dtd)['md5'] != expected_dtd_md5:
|
||||
raise ValueError('DBLP DTD checksum mismatch or size limit')
|
||||
manifest, manifest_sha, expected, volumes = catalogue_evidence(
|
||||
catalogue, families, expected_sha, expected_dtd_md5)
|
||||
totals, found = Counter(), Counter()
|
||||
seen_keys, found_volumes = set(), set()
|
||||
with tempfile.TemporaryDirectory(prefix='.dblp-family-', dir=output.parent) as tmp:
|
||||
stage = Path(tmp) / 'recovered'
|
||||
stage.mkdir()
|
||||
target = stage / 'dblp-family-candidates.jsonl'
|
||||
with target.open('w') as stream:
|
||||
for row in records(archive, dtd.read_bytes()):
|
||||
totals[row['record_type']] += 1
|
||||
name = family(row['key'])
|
||||
if name in families:
|
||||
if row['key'] in seen_keys:
|
||||
raise ValueError('Duplicate raw bibliographic key')
|
||||
seen_keys.add(row['key'])
|
||||
found[(name, row['record_type'])] += 1
|
||||
if row['record_type'] == 'proceedings':
|
||||
if volumes.get(row['key']) != row:
|
||||
raise ValueError('Raw proceedings differs from catalogue evidence')
|
||||
found_volumes.add(row['key'])
|
||||
stream.write(json.dumps(row, ensure_ascii=False) + '\n')
|
||||
scanned = sum(totals.values())
|
||||
if scanned % 250000 == 0:
|
||||
print(json.dumps({'status': 'scanning_raw_family_candidates',
|
||||
'records_scanned': scanned,
|
||||
'recovered_records': sum(found.values())}), flush=True)
|
||||
if (totals != Counter(manifest['record_types'])
|
||||
or sum(totals.values()) != manifest['records_scanned']):
|
||||
raise ValueError('Complete snapshot totals differ from catalogue')
|
||||
if found != expected or found_volumes != volumes.keys():
|
||||
raise ValueError('Recovered family totals differ from catalogue')
|
||||
report = {
|
||||
'format_version': 1, 'source': 'dblp',
|
||||
'status': 'recovered_raw_candidates_pending_scope_review',
|
||||
'finished_at': timestamp(), 'source_release': RELEASE,
|
||||
'snapshot_sha256': actual['sha256'], 'dtd_md5': expected_dtd_md5,
|
||||
'catalogue_manifest_sha256': manifest_sha, 'families': families,
|
||||
'records_scanned': sum(totals.values()), 'record_types': dict(totals),
|
||||
'recovered_records': sum(found.values()),
|
||||
'recovered_publications': sum(found[(n, 'inproceedings')] for n in families),
|
||||
'recovered_volumes': len(found_volumes),
|
||||
'family_counts': [{'family': name, 'record_type': kind, 'count': count}
|
||||
for (name, kind), count in sorted(found.items())],
|
||||
'old_candidate_family_counts': {
|
||||
name: manifest.get('candidate_family_counts', {}).get(name, 0)
|
||||
for name in families},
|
||||
'catalogue_volumes': [volumes[k] for k in sorted(volumes)],
|
||||
'file': str(output / target.name),
|
||||
'file_sha256': digest_file(target)['sha256'], 'database_written': False,
|
||||
'note': 'Raw bibliographic candidates, including volume records and possible '
|
||||
'keynotes/workshops. No venue, track, acceptance or work identity '
|
||||
'inferred. Requires primary-source scope and identity review; '
|
||||
'not a normalized import queue. Existing imported sources must '
|
||||
'not be replayed.',
|
||||
}
|
||||
# Deliberately not an import-records sidecar manifest.
|
||||
(stage / 'recovery-manifest.json').write_text(
|
||||
json.dumps(report, ensure_ascii=False, indent=2) + '\n')
|
||||
if output.exists() or output.is_symlink():
|
||||
raise ValueError('Recovery output appeared during scan')
|
||||
stage.rename(output)
|
||||
return report
|
||||
|
||||
|
||||
def main(argv=None, *, prog=None):
|
||||
parser = argparse.ArgumentParser(prog=prog, description=__doc__)
|
||||
parser.add_argument('--archive', required=True, type=Path)
|
||||
parser.add_argument('--dtd', required=True, type=Path)
|
||||
parser.add_argument('--catalogue', required=True, type=Path)
|
||||
parser.add_argument('--family', required=True, action='append', choices=sorted(FAMILIES))
|
||||
parser.add_argument('--out', required=True, type=Path)
|
||||
args = parser.parse_args(argv)
|
||||
print(json.dumps(recover(args.archive, args.dtd, args.catalogue, args.out, args.family),
|
||||
ensure_ascii=False))
|
||||
@@ -0,0 +1,141 @@
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from paper_atlas.collectors.dblp import catalogue
|
||||
from paper_atlas.collectors.dblp_recovery import recover
|
||||
from paper_atlas.collectors.dblp_scope import scope
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
|
||||
class DblpRecoveryTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.tmp.cleanup)
|
||||
self.root = Path(self.tmp.name)
|
||||
self.dtd = self.root / 'dblp.dtd'
|
||||
self.dtd.write_bytes(b'<!ENTITY eacute "é">')
|
||||
self.archive = self.root / 'dblp.xml.gz'
|
||||
# These include a keynote and a workshop, neither a main-paper inference.
|
||||
body = '''<proceedings key="conf/ifaamas/2025"><title>AAMAS 2025</title><year>2025</year></proceedings>
|
||||
<inproceedings key="conf/ifaamas/keynote"><author>René</author><title>Keynote: Agents</title><year>2025</year><booktitle>AAMAS</booktitle><crossref>conf/ifaamas/2025</crossref></inproceedings>
|
||||
<inproceedings key="conf/ifaamas/paper"><author>Alice Doe</author><title>Research <i>agents</i></title><year>2025</year><booktitle>AAMAS</booktitle><crossref>conf/ifaamas/2025</crossref><ee>https://dl.acm.org/doi/10.5555/3709347.123</ee></inproceedings>
|
||||
<proceedings key="conf/ifaamas/2025w"><title>AAMAS Workshop</title><year>2025</year></proceedings>
|
||||
<inproceedings key="conf/ifaamas/workshop"><title>Workshop paper</title><year>2025</year><booktitle>AAMAS Workshop</booktitle><crossref>conf/ifaamas/2025w</crossref></inproceedings>
|
||||
<inproceedings key="conf/unrelated/other"><title>Other family</title><year>2025</year></inproceedings>'''
|
||||
with gzip.open(self.archive, 'wb') as stream:
|
||||
stream.write(('<?xml version="1.0"?><!DOCTYPE dblp SYSTEM "dblp.dtd"><dblp>'
|
||||
+ body + '</dblp>').encode())
|
||||
self.sha = hashlib.sha256(self.archive.read_bytes()).hexdigest()
|
||||
self.md5 = hashlib.md5(self.dtd.read_bytes()).hexdigest()
|
||||
self.manifest = catalogue(self.archive, self.dtd, self.root / 'catalogue',
|
||||
self.sha, self.md5)
|
||||
self.catalogue = Path(self.manifest['catalogue'])
|
||||
self.manifest_path = self.catalogue.with_name('dblp-ai-candidates.manifest.json')
|
||||
|
||||
def run_recovery(self, **kwargs):
|
||||
options = dict(expected_sha=self.sha, expected_dtd_md5=self.md5)
|
||||
options.update(kwargs)
|
||||
return recover(self.archive, self.dtd, self.catalogue, self.root / 'recovered',
|
||||
['ifaamas'], **options)
|
||||
|
||||
def test_fresh_discovery_and_recovery_from_old_candidate_omission(self):
|
||||
self.assertEqual(self.manifest['candidate_family_counts']['ifaamas'], 3)
|
||||
# Simulate an older filter without altering its complete catalogue.
|
||||
manifest = dict(self.manifest)
|
||||
manifest['candidate_family_counts'] = {}
|
||||
self.manifest_path.write_text(json.dumps(manifest))
|
||||
candidate = Path(manifest['file'])
|
||||
candidate.write_text('old incomplete candidate file\n')
|
||||
before = {p: p.read_bytes() for p in [self.archive, self.dtd, self.catalogue,
|
||||
self.manifest_path, candidate]}
|
||||
with patch('urllib.request.urlopen', side_effect=AssertionError('No network')):
|
||||
report = self.run_recovery()
|
||||
rows = [json.loads(line) for line in Path(report['file']).read_text().splitlines()]
|
||||
self.assertEqual(report['recovered_publications'], 3)
|
||||
self.assertEqual(report['recovered_volumes'], 2)
|
||||
self.assertEqual(report['recovered_records'], 5)
|
||||
self.assertEqual(report['records_scanned'], 6)
|
||||
self.assertEqual(report['old_candidate_family_counts'], {'ifaamas': 0})
|
||||
self.assertFalse(report['database_written'])
|
||||
self.assertEqual(next(r for r in rows if r['key'].endswith('/keynote'))['fields']['author'], ['René'])
|
||||
self.assertTrue(any(r['key'].endswith('/workshop') for r in rows))
|
||||
self.assertTrue(all(not ({'source', 'status', 'venue', 'track'} & r.keys()) for r in rows))
|
||||
publication = next(r for r in rows if r['key'].endswith('/paper'))
|
||||
volume = next(r for r in rows if r['key'] == 'conf/ifaamas/2025')
|
||||
self.assertEqual(scope(publication, volume), (None, 'outside_target_family'))
|
||||
self.assertEqual({p: p.read_bytes() for p in before}, before)
|
||||
self.assertEqual(hashlib.sha256(Path(report['file']).read_bytes()).hexdigest(),
|
||||
report['file_sha256'])
|
||||
# Raw recovery files cannot accidentally enter the normalized writer.
|
||||
with patch('paper_atlas.ingestion.records.open_store') as writer:
|
||||
with self.assertRaisesRegex(ValueError, 'source manifest'):
|
||||
ingest_file(self.root / 'never.sqlite', report['file'])
|
||||
writer.assert_not_called()
|
||||
self.assertFalse((self.root / 'never.sqlite').exists())
|
||||
|
||||
def test_checksum_mismatch_does_not_publish(self):
|
||||
for options in [{'expected_sha': '0' * 64}, {'expected_dtd_md5': '0' * 32}]:
|
||||
with self.subTest(options=options), self.assertRaisesRegex(ValueError, 'checksum'):
|
||||
self.run_recovery(**options)
|
||||
self.assertFalse((self.root / 'recovered').exists())
|
||||
|
||||
def test_wrong_catalogue_snapshot_and_incomplete_totals_are_rejected(self):
|
||||
for changes in [{'snapshot_sha256': '0' * 64}, {'dtd_md5': '0' * 32},
|
||||
{'records_scanned': 99}, {'record_types': {'proceedings': 2}}]:
|
||||
manifest = {**self.manifest, **changes}
|
||||
self.manifest_path.write_text(json.dumps(manifest))
|
||||
with self.subTest(changes=changes), self.assertRaises(ValueError):
|
||||
self.run_recovery()
|
||||
self.assertFalse((self.root / 'recovered').exists())
|
||||
|
||||
def test_changed_family_count_does_not_publish_partial_result(self):
|
||||
with sqlite3.connect(self.catalogue) as c:
|
||||
c.execute("UPDATE family_counts SET count=count+1 WHERE family='ifaamas' AND record_type='inproceedings'")
|
||||
with self.assertRaisesRegex(ValueError, 'family totals'):
|
||||
self.run_recovery()
|
||||
self.assertFalse((self.root / 'recovered').exists())
|
||||
|
||||
def test_changed_volume_metadata_is_detected(self):
|
||||
with sqlite3.connect(self.catalogue) as c:
|
||||
row = json.loads(c.execute("SELECT metadata_json FROM proceedings WHERE key='conf/ifaamas/2025'").fetchone()[0])
|
||||
row['fields']['title'] = ['Wrong volume']
|
||||
c.execute("UPDATE proceedings SET metadata_json=? WHERE key='conf/ifaamas/2025'", (json.dumps(row),))
|
||||
with self.assertRaisesRegex(ValueError, 'proceedings differs'):
|
||||
self.run_recovery()
|
||||
self.assertFalse((self.root / 'recovered').exists())
|
||||
|
||||
def test_truncated_gzip_never_publishes(self):
|
||||
self.archive.write_bytes(self.archive.read_bytes()[:-8])
|
||||
revised_sha = hashlib.sha256(self.archive.read_bytes()).hexdigest()
|
||||
self.manifest_path.write_text(json.dumps({**self.manifest, 'snapshot_sha256': revised_sha}))
|
||||
with self.assertRaises((EOFError, OSError)):
|
||||
self.run_recovery(expected_sha=revised_sha)
|
||||
self.assertFalse((self.root / 'recovered').exists())
|
||||
|
||||
def test_existing_output_is_preserved_and_unknown_family_is_rejected(self):
|
||||
out = self.root / 'recovered'
|
||||
out.mkdir()
|
||||
sentinel = out / 'keep'
|
||||
sentinel.write_text('existing')
|
||||
with self.assertRaisesRegex(ValueError, 'already exist'):
|
||||
self.run_recovery()
|
||||
self.assertEqual(sentinel.read_text(), 'existing')
|
||||
for names in [[], ['../ifaamas'], ['not-a-supported-family'], [None]]:
|
||||
with self.subTest(names=names), self.assertRaises(ValueError):
|
||||
recover(self.archive, self.dtd, self.catalogue, self.root / 'other', names)
|
||||
|
||||
def test_requested_family_missing_from_catalogue_is_explicit(self):
|
||||
with self.assertRaisesRegex(ValueError, 'absent from complete catalogue'):
|
||||
recover(self.archive, self.dtd, self.catalogue, self.root / 'recovered',
|
||||
['ifaamas', 'icml'], expected_sha=self.sha, expected_dtd_md5=self.md5)
|
||||
self.assertFalse((self.root / 'recovered').exists())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user