Compare commits

...

22 Commits

Author SHA1 Message Date
codex-bot b5c0618be5 Handle standalone italic author blocks in the historical archive
Metadata integrity checks / metadata (push) Successful in 2m6s
2026-11-09 16:24:42 +08:00
codex-bot 81864e48ae Preserve complete fragmented ICLR 2014 author paragraphs
Metadata integrity checks / metadata (push) Successful in 2m9s
2026-11-09 16:19:19 +08:00
codex-bot 348ebe0695 Collect archived ICLR accepted lists with track and presentation evidence
Metadata integrity checks / metadata (push) Successful in 2m15s
2026-11-09 16:09:13 +08:00
Zayden-Jung 02a408ba8f Merge pull request '保留 AAMAS 发表来源的同届 DOI 投稿身份' (#39) from codex/aamas-publication-identity into main
Metadata integrity checks / metadata (push) Successful in 2m15s
Reviewed-on: #39
2026-11-09 06:43:15 +00:00
codex-bot 714f745bd5 Assert normalized identity tuples and exact negative-status boundaries
Metadata integrity checks / metadata (push) Successful in 2m13s
2026-11-09 14:13:12 +08:00
Zayden-Jung cc78e837d2 Merge pull request '按发表形式采集 AAMAS 官方论文集' (#38) from codex/aamas-proceedings into main
Metadata integrity checks / metadata (push) Successful in 2m26s
Reviewed-on: #38
2026-11-09 05:52:57 +00:00
codex-bot c0b2146ae8 Recognize AAMAS publisher DOI identities within submission scopes
Metadata integrity checks / metadata (push) Failing after 2m1s
2026-11-09 13:51:49 +08:00
codex-bot ae88367fac Scope AAMAS tags to their exact source namespace
Metadata integrity checks / metadata (push) Successful in 8m56s
2026-11-09 13:31:46 +08:00
codex-bot a305d00c75 Collect evidenced AAMAS proceedings metadata by publication format
Metadata integrity checks / metadata (push) Failing after 2m2s
2026-11-09 13:27:23 +08:00
Zayden-Jung 57fcfc7693 Merge pull request '按官方证据识别早期 ICML 与 IROS 主系列' (#37) from codex/historical-primary-series into main
Metadata integrity checks / metadata (push) Successful in 2m58s
Reviewed-on: #37
2026-11-09 05:19:49 +00:00
Zayden-Jung a07b1e42a8 Merge pull request '修正早期 ACL 来源误匹配并记录官方目录缺口' (#36) from codex/history-source-scope into main
Metadata integrity checks / metadata (push) Successful in 2m48s
Reviewed-on: #36
2026-11-09 05:19:43 +00:00
codex-bot 7c6f4c9f1d Recognize evidenced early ICML and IROS primary editions
Metadata integrity checks / metadata (push) Successful in 9m26s
2026-11-09 02:39:18 +08:00
codex-bot 3f0ac694a0 Correct early ACL source candidates and record official index limits
Metadata integrity checks / metadata (push) Successful in 14m52s
2026-11-09 02:28:51 +08:00
Zayden-Jung 32c7edd0a5 Merge pull request '核对135个历史年度缺口与候选论文集来源' (#35) from codex/historical-gaps into main
Metadata integrity checks / metadata (push) Successful in 2m26s
Reviewed-on: #35
2026-11-08 17:37:12 +00:00
Zayden-Jung 658d799334 Merge pull request '封装早期WWW档案采集并核实逐条poster证据' (#32) from codex/www-archive into main
Metadata integrity checks / metadata (push) Successful in 2m34s
Reviewed-on: #32
2026-11-08 17:37:03 +00:00
codex-bot 53fbe9c75c Record current historical year gaps and candidate source volumes
Metadata integrity checks / metadata (push) Successful in 3m15s
2026-11-09 00:50:15 +08:00
codex-bot dac6edbe1e Merge remote-tracking branch 'origin/main' into codex/www-archive
Metadata integrity checks / metadata (push) Successful in 3m10s
# Conflicts:
#	README.md
2026-11-09 00:48:14 +08:00
Zayden-Jung 7a674ac797 Merge pull request '按证据修正投稿轨道并保护后续来源刷新' (#34) from codex/edition-tracks into main
Metadata integrity checks / metadata (push) Successful in 2m5s
Reviewed-on: #34
2026-11-08 16:23:37 +00:00
codex-bot eacc1e3292 Resolve reviewed WWW poster entries with canonical page evidence
Metadata integrity checks / metadata (push) Successful in 1m32s
2026-11-09 00:18:32 +08:00
codex-bot 2275bc7530 Merge remote-tracking branch 'origin/main' into codex/www-archive
# Conflicts:
#	README.md
2026-11-09 00:13:37 +08:00
codex-bot 9040649320 Preserve full source listing totals beside ready archive subsets
Metadata integrity checks / metadata (push) Successful in 1m10s
2026-11-08 21:54:43 +08:00
codex-bot 64ced7af42 Package reproducible early WWW metadata collection with explicit review holds
Metadata integrity checks / metadata (push) Successful in 1m10s
2026-11-08 21:41:02 +08:00
23 changed files with 7977 additions and 1 deletions
+3
View File
@@ -48,6 +48,9 @@ jobs:
outside.mkdir()
subprocess.run([str(venv/'bin/paper-atlas'),'--help'],cwd=outside,check=True)
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','cvf','--help'],cwd=outside,check=True)
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','www-archive','--help'],cwd=outside,check=True)
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','aamas-proceedings','--help'],cwd=outside,check=True)
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','iclr-archive','--help'],cwd=outside,check=True)
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','postgres','--help'],cwd=outside,check=True)
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','edition-tracks','--help'],cwd=outside,check=True)
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','joint-editions','--help'],cwd=outside,check=True)
+7
View File
@@ -6,6 +6,8 @@ Paper Atlas 将 arXiv、AI 会议论文集及公开投稿决定整理到可追
项目仍在建设中。候选范围包含 43 个会议(含 ACM MM、MLSys),尚未完成全部历史数据和最终去重复核。实际进度以数据库、来源清单和覆盖报告为准,而不是仓库文件数量。
历史 ICLR 2014–2016 官方录用页由 `collect iclr-archive` 适配,保留主会/workshop及重复展示证据;运行和身份核验边界见 [历史档案说明](docs/coverage/iclr-archive-2014-2016.md)。
## 项目结构
```text
@@ -60,6 +62,10 @@ paper-atlas collect colm --cache /data/paper-index/raw/http-cache \
--out /data/paper-index/collected
```
早期主系列的Workshop名称按[ICML/IROS精确届次证据](docs/coverage/historical-primary-series-2026-11-09.md)识别,保留原名与范围证据;代码支持不代表已补齐生产数据。
AAMAS 的[官方论文集采集器](docs/aamas-proceedings.md)目前核验了 2012 版式,可从缓存或已校验快照采集标题、作者及页面提供的摘要;发表形式单独保存,全文/扩展摘要中未明确的投稿 track 保持待明确,不根据会话主题推定主会轨道。
`paper-atlas collect --help` 查看来源;`paper-atlas import --help` 查看导入方式。完整批次步骤见[使用指南](docs/usage.md)。
共享实例名为 `foundry-postgres`,数据位于 `foundry_data` 库的 `research` schema;其他项目可按权限复用同一组表。库名通过 libpq service 配置、schema 通过 search_path 或 `--pg-schema` 指定,不写死在查询中。PostgreSQL 凭据放在服务器权限为 `0600` 的 libpq service 文件中;命令行只使用服务名称。`paper-atlas maintain postgres --help` 提供迁移入口:先完整核验 21 张业务表,再发布目标 schema。SQLite FTS5 内部表由 PostgreSQL GIN 索引替代。切换后 SQLite 为只读备份;`import records`、`import arxiv` 和 `maintain backfill` 使用 `--pg-service foundry-research` 写入主库,整个写入会话只允许一个协调写入者,会议文件失败时整体回滚并保留失败批次记录。
@@ -88,6 +94,7 @@ CI 在已授权的远程 runner 上运行测试、构建 wheel,并从源码目
开发通过工作分支和 PR 交付,由用户手动合并。
早期WWW档案的可重放元数据采集、论文/海报分轨与待核查条目见 [WWW档案采集](docs/www-archive.md)。
历史联合会议的既有届次归属可通过 [`maintain joint-editions`](docs/joint-editions.md) 先只读规划,再按计划哈希执行;保留论文与投稿决定来源,不重导入。
同年多届的 `event_key`、显式迁移、旧文件刷新与查询边界见 [会议事件身份](docs/event-identity.md)。
研究类型与自定义标签可通过 [`annotate`](docs/annotations.md) 显式初始化、添加和撤回;两者独立查询,保留证据及完整更正历史,不从会议轨道推断研究类型。
+43
View File
@@ -0,0 +1,43 @@
# AAMAS 官方论文集元数据
`paper_atlas.collectors.aamas` 是按需运行的采集适配器,通过统一入口 `collect aamas-proceedings` 使用。目前只支持已独立核验的 **AAMAS 2012** HTML 版式;其他年份需先核对官方目录和逐篇结构,再增加适配,不能仅替换年份。
## 来源与范围
[官方论文集介绍](https://www.ifaamas.org/Proceedings/aamas2012/)及其 Full Papers、Extended Abstracts、Demos、Doctorial Consortium Abstracts 四个栏目分别列出 136、146、28、24 条,共 334 条。旧 DBLP 候选卷的 310 条不能替代这四类官方目录,也不能全部标记为主会全文。栏目数量来自 2026-11-09 冻结的原始页面;页面结构或数量发生变化时,采集器停止,留待重新核验。
采集标题、逐篇作者、页面实际提供的摘要、官方条目 ID、目录锚点和 PDF 地址。**只保存 PDF 链接,不下载 PDF。** 全文页面的摘要藏在标题链接内部的 `abstract` 元素,作者链接里的 “has N papers” 是界面徽标,二者不会混入标题或姓名。未提供的摘要保存为 `null`,不会用标题、说明文字或其他论文填充。
官方介绍说明了 general、robotics、virtual agents、innovative applications 等投稿轨道,但逐篇目录的主题会话不能独立证明某篇论文的投稿轨道。因此全文和扩展摘要均使用 `proceedings_unspecified`;演示、博士栏目使用 `demo`、`doctoral`。发表形式保留在 `metadata.publication_format` 和 `aamas_proceedings:publication_format` 标签中,会话原文单独保存。它们不是研究类型标注,也不会写入自定义 annotation。
目录条目仅提供发表录用证据,不是原始决定信。`observed_at` 是页面采集时间,不能当作 2012 年的决定日期;官方介绍中的投稿、撤稿、录用汇总数不能识别逐篇拒稿或撤稿,采集器不会由总量相减生成这些决定。
## 运行
在已确认的远程运行环境中,使用 HTTP 缓存采集:
```sh
paper-atlas collect aamas-proceedings --year 2012 \
--cache /data/paper-index/cache/aamas \
--out /data/paper-index/collected/aamas-2012
```
也可使用已冻结的官方页面:
```sh
paper-atlas collect aamas-proceedings --year 2012 \
--snapshots /data/paper-index/collected/historical-next-source-evidence-v1 \
--out /data/paper-index/collected/aamas-2012
```
快照目录需含 `receipts.json`,其中每个响应包含原始 `url`、HTTP `status`、相对 `file`、`bytes`、`sha256` 和 `finished_at`。要求五个准确官方 URL 均唯一匹配且返回 200;验证全部字节数和哈希后才解析,不接受目录穿越或绝对文件路径。快照中其他来源的响应不会被当成 AAMAS 页面。
输出为按数据库 track 分组的三个 JSONL 文件和对应 manifest、五份原始 HTML 证据,以及最后写出的 `aamas-proceedings-2012.json` 采集报告。锁文件避免同一输出目录并发运行。manifest 始终为 `partial`,记录发表形式数量和缺失摘要数;**采集命令不连接数据库,也不导入。**
## 复核与恢复
每条记录携带对应官方页面的 URL、采集时间、字节数、SHA-256 与输出内证据路径。重新运行前先检查已有采集报告和文件哈希;采集中断不等于主库已导入,报告缺失或哈希不一致的目录不能直接作为导入队列。输出中的原始证据应随文件一起保存。
生产导入前仍需单独与主库核验稳定标识、标题、作者及已有投稿,按证据补充 DOI/arXiv 等标识;目录页上的作者计数徽标和会话 ID 不能作为这些标识。需要另建经过核验的增量清单、保护旧历史,完成独立验收后才能计入覆盖。跨来源同一论文不得仅因同题就新增或合并投稿。
本功能的代码、测试或采集成功均不代表 AAMAS 2012 已导入生产,也不证明拒稿全集完整。全库最终去重、摘要和来源覆盖验收属于独立后续阶段。
+4
View File
@@ -190,3 +190,7 @@ WWW早期逐来源核查见 `www-early-proceedings-2026-11-08.json`:1995春季
WWW事件标识生产迁移已独立验收(报告SHA `f5dbb8a8`):保留1399个原届次字段及IDs、59个Fall投稿和完整全库历史哈希。服务器已缓存46个官方HTML来源(约223KB),只作为后续维护式采集器的输入;尚未补导早期WWW论文,也未启用可选注解表。
IJCAI 联办届次的逐条后续核查见 [2018/2022 分区与目录差集](ijcai-sections-2026-11-08.md) 和 [稳定 ID 映射](ijcai-sections-2026-11-08.json)。已定位 345 条非 main 分类及 2022 官方 HTML 缺失但官方文件仍存在的一条记录;生产分类尚未修正,不能把整卷数量作为 main 覆盖率。
2026年11月9日的[当前历史年度缺口核对](historical-gaps-2026-11-09.md)单列135个历史会议—年份、14个2026年度及85个经范围修正的初步候选来源年度;它使用当前生产edition/别名快照,保留原固定快照不变。年度有记录不等于逐篇、逐track或公开拒稿历史完整。
[早期NLP来源范围复核](historical-source-scope-2026-11-09.md)保留3份官方网页的响应哈希,排除4个COLING卷对ACL缺口的误匹配,并区分会议举办记录、目录可获取性与论文级覆盖;尚未导入或关闭这些年度缺口。
@@ -0,0 +1,867 @@
{
"snapshot_at": "2026-11-08 16:34:40.741334+00:00",
"scope": "Official first-pass conference-year census vs current populated editions and real aliases. Missing year means no imported attempt in that year, not missing paper count or proof all other years complete. Includes announced/nonarchival exceptions below.",
"venues": [
{
"venue": "NeurIPS",
"expected_years": 40,
"observed_expected_years": 39,
"missing_years": [
2026
],
"missing_count": 1,
"canonical_matches": [
"NeurIPS"
]
},
{
"venue": "ICML",
"expected_years": 43,
"observed_expected_years": 14,
"missing_years": [
1980,
1983,
1985,
1987,
1988,
1989,
1990,
1991,
1992,
1993,
1994,
1995,
1996,
1997,
1998,
1999,
2000,
2001,
2002,
2003,
2004,
2005,
2006,
2007,
2008,
2009,
2010,
2011,
2012
],
"missing_count": 29,
"canonical_matches": [
"ICML"
]
},
{
"venue": "ICLR",
"expected_years": 14,
"observed_expected_years": 11,
"missing_years": [
2014,
2015,
2016
],
"missing_count": 3,
"canonical_matches": [
"ICLR"
]
},
{
"venue": "COLM",
"expected_years": 3,
"observed_expected_years": 3,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"COLM"
]
},
{
"venue": "COLT",
"expected_years": 39,
"observed_expected_years": 39,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"COLT"
]
},
{
"venue": "AISTATS",
"expected_years": 29,
"observed_expected_years": 25,
"missing_years": [
1985,
1989,
1991,
1993
],
"missing_count": 4,
"canonical_matches": [
"AISTATS"
]
},
{
"venue": "UAI",
"expected_years": 42,
"observed_expected_years": 42,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"UAI"
]
},
{
"venue": "CLeaR",
"expected_years": 5,
"observed_expected_years": 5,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"CLEAR"
]
},
{
"venue": "ACL",
"expected_years": 64,
"observed_expected_years": 48,
"missing_years": [
1963,
1964,
1965,
1966,
1967,
1968,
1969,
1970,
1971,
1972,
1973,
1974,
1975,
1976,
1977,
1978
],
"missing_count": 16,
"canonical_matches": [
"ACL",
"ACL+COLING",
"ACL+EACL"
]
},
{
"venue": "EMNLP",
"expected_years": 31,
"observed_expected_years": 31,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"CoNLL+EMNLP",
"EMNLP"
]
},
{
"venue": "NAACL",
"expected_years": 18,
"observed_expected_years": 18,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"NAACL"
]
},
{
"venue": "EACL",
"expected_years": 19,
"observed_expected_years": 19,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"ACL+EACL",
"EACL"
]
},
{
"venue": "COLING",
"expected_years": 31,
"observed_expected_years": 28,
"missing_years": [
1971,
1976,
1978
],
"missing_count": 3,
"canonical_matches": [
"ACL+COLING",
"COLING",
"COLING+LREC"
]
},
{
"venue": "CoNLL",
"expected_years": 30,
"observed_expected_years": 30,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"CoNLL",
"CoNLL+EMNLP"
]
},
{
"venue": "CVPR",
"expected_years": 39,
"observed_expected_years": 14,
"missing_years": [
1983,
1985,
1986,
1988,
1989,
1991,
1992,
1993,
1994,
1996,
1997,
1998,
1999,
2000,
2001,
2003,
2004,
2005,
2006,
2007,
2008,
2009,
2010,
2011,
2012
],
"missing_count": 25,
"canonical_matches": [
"CVPR"
]
},
{
"venue": "ICCV",
"expected_years": 20,
"observed_expected_years": 7,
"missing_years": [
1987,
1988,
1990,
1993,
1995,
1998,
1999,
2001,
2003,
2005,
2007,
2009,
2011
],
"missing_count": 13,
"canonical_matches": [
"ICCV"
]
},
{
"venue": "ECCV",
"expected_years": 19,
"observed_expected_years": 4,
"missing_years": [
1990,
1992,
1994,
1996,
1998,
2000,
2002,
2004,
2006,
2008,
2010,
2012,
2014,
2016,
2026
],
"missing_count": 15,
"canonical_matches": [
"ECCV"
]
},
{
"venue": "AAAI",
"expected_years": 40,
"observed_expected_years": 40,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"AAAI",
"AAAI+IAAI"
]
},
{
"venue": "IJCAI",
"expected_years": 35,
"observed_expected_years": 35,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"ECAI+IJCAI",
"IJCAI"
]
},
{
"venue": "ECAI",
"expected_years": 29,
"observed_expected_years": 27,
"missing_years": [
1974,
1980
],
"missing_count": 2,
"canonical_matches": [
"ECAI",
"ECAI+IJCAI"
]
},
{
"venue": "KDD",
"expected_years": 32,
"observed_expected_years": 32,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"KDD"
]
},
{
"venue": "SIGIR",
"expected_years": 50,
"observed_expected_years": 50,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"SIGIR"
]
},
{
"venue": "WWW",
"expected_years": 33,
"observed_expected_years": 27,
"missing_years": [
1994,
1996,
1997,
1998,
1999,
2000
],
"missing_count": 6,
"canonical_matches": [
"WWW"
]
},
{
"venue": "WSDM",
"expected_years": 19,
"observed_expected_years": 19,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"WSDM"
]
},
{
"venue": "RecSys",
"expected_years": 20,
"observed_expected_years": 19,
"missing_years": [
2026
],
"missing_count": 1,
"canonical_matches": [
"RecSys"
]
},
{
"venue": "CIKM",
"expected_years": 35,
"observed_expected_years": 34,
"missing_years": [
2026
],
"missing_count": 1,
"canonical_matches": [
"CIKM"
]
},
{
"venue": "ICDM",
"expected_years": 26,
"observed_expected_years": 25,
"missing_years": [
2026
],
"missing_count": 1,
"canonical_matches": [
"ICDM"
]
},
{
"venue": "SDM",
"expected_years": 26,
"observed_expected_years": 25,
"missing_years": [
2026
],
"missing_count": 1,
"canonical_matches": [
"SDM"
]
},
{
"venue": "ECML-PKDD",
"expected_years": 38,
"observed_expected_years": 33,
"missing_years": [
1986,
1987,
1988,
1989,
1991
],
"missing_count": 5,
"canonical_matches": [
"ECML",
"ECML-PKDD",
"PKDD"
]
},
{
"venue": "ICRA",
"expected_years": 43,
"observed_expected_years": 43,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"ICRA"
]
},
{
"venue": "IROS",
"expected_years": 39,
"observed_expected_years": 35,
"missing_years": [
1988,
1989,
1990,
2026
],
"missing_count": 4,
"canonical_matches": [
"IROS"
]
},
{
"venue": "RSS",
"expected_years": 22,
"observed_expected_years": 20,
"missing_years": [
2025,
2026
],
"missing_count": 2,
"canonical_matches": [
"RSS"
]
},
{
"venue": "CoRL",
"expected_years": 10,
"observed_expected_years": 9,
"missing_years": [
2026
],
"missing_count": 1,
"canonical_matches": [
"CORL"
]
},
{
"venue": "ICAPS",
"expected_years": 37,
"observed_expected_years": 28,
"missing_years": [
1990,
1991,
1992,
1993,
1995,
1997,
1999,
2001,
2026
],
"missing_count": 9,
"canonical_matches": [
"AIPS",
"ICAPS"
]
},
{
"venue": "KR",
"expected_years": 23,
"observed_expected_years": 23,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"KR"
]
},
{
"venue": "AAMAS",
"expected_years": 25,
"observed_expected_years": 22,
"missing_years": [
2012,
2025,
2026
],
"missing_count": 3,
"canonical_matches": [
"AAMAS"
]
},
{
"venue": "ICASSP",
"expected_years": 51,
"observed_expected_years": 49,
"missing_years": [
1977,
2026
],
"missing_count": 2,
"canonical_matches": [
"ICASSP"
]
},
{
"venue": "INTERSPEECH",
"expected_years": 39,
"observed_expected_years": 39,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"ECST",
"EUROSPEECH",
"ICSLP",
"INTERSPEECH"
]
},
{
"venue": "ISMIR",
"expected_years": 27,
"observed_expected_years": 26,
"missing_years": [
2026
],
"missing_count": 1,
"canonical_matches": [
"ISMIR"
]
},
{
"venue": "MICCAI",
"expected_years": 29,
"observed_expected_years": 29,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"MICCAI"
]
},
{
"venue": "FAccT",
"expected_years": 9,
"observed_expected_years": 9,
"missing_years": [],
"missing_count": 0,
"canonical_matches": [
"FAccT"
]
},
{
"venue": "MLSys",
"expected_years": 9,
"observed_expected_years": 8,
"missing_years": [
2018
],
"missing_count": 1,
"canonical_matches": [
"MLSys"
]
},
{
"venue": "MM",
"expected_years": 34,
"observed_expected_years": 33,
"missing_years": [
2026
],
"missing_count": 1,
"canonical_matches": [
"MM"
]
}
],
"expected_venue_years": 1266,
"missing_venue_years": 150,
"venues_with_missing_years": 25,
"known_nonpublication_exceptions": [
{
"venue": "NeurIPS",
"year": 2026,
"reason": "announced;not in proceedings index at census"
},
{
"venue": "CoRL",
"year": 2026,
"reason": "not in publisher proceedings index at census"
},
{
"venue": "MLSys",
"year": 2018,
"reason": "first non-archival SysML meeting; not a missing formal publication volume"
}
],
"historical_missing_years_excluding_nonarchival_2018": 135,
"historical_venues_with_missing_years": 16,
"historical_gaps": [
{
"venue": "ICML",
"years": [
1980,
1983,
1985,
1987,
1988,
1989,
1990,
1991,
1992,
1993,
1994,
1995,
1996,
1997,
1998,
1999,
2000,
2001,
2002,
2003,
2004,
2005,
2006,
2007,
2008,
2009,
2010,
2011,
2012
]
},
{
"venue": "ICLR",
"years": [
2014,
2015,
2016
]
},
{
"venue": "AISTATS",
"years": [
1985,
1989,
1991,
1993
]
},
{
"venue": "ACL",
"years": [
1963,
1964,
1965,
1966,
1967,
1968,
1969,
1970,
1971,
1972,
1973,
1974,
1975,
1976,
1977,
1978
]
},
{
"venue": "COLING",
"years": [
1971,
1976,
1978
]
},
{
"venue": "CVPR",
"years": [
1983,
1985,
1986,
1988,
1989,
1991,
1992,
1993,
1994,
1996,
1997,
1998,
1999,
2000,
2001,
2003,
2004,
2005,
2006,
2007,
2008,
2009,
2010,
2011,
2012
]
},
{
"venue": "ICCV",
"years": [
1987,
1988,
1990,
1993,
1995,
1998,
1999,
2001,
2003,
2005,
2007,
2009,
2011
]
},
{
"venue": "ECCV",
"years": [
1990,
1992,
1994,
1996,
1998,
2000,
2002,
2004,
2006,
2008,
2010,
2012,
2014,
2016
]
},
{
"venue": "ECAI",
"years": [
1974,
1980
]
},
{
"venue": "WWW",
"years": [
1994,
1996,
1997,
1998,
1999,
2000
]
},
{
"venue": "ECML-PKDD",
"years": [
1986,
1987,
1988,
1989,
1991
]
},
{
"venue": "IROS",
"years": [
1988,
1989,
1990
]
},
{
"venue": "RSS",
"years": [
2025
]
},
{
"venue": "ICAPS",
"years": [
1990,
1991,
1992,
1993,
1995,
1997,
1999,
2001
]
},
{
"venue": "AAMAS",
"years": [
2012,
2025
]
},
{
"venue": "ICASSP",
"years": [
1977
]
}
],
"current_2026_gaps": 14
}
@@ -0,0 +1,58 @@
# 历史年度缺口的当前核对
生产只读快照:2026-11-08 16:34:40.741334 UTC(北京时间11月9日00:34),foundry_data/research。对比已核实的43份首轮官方会议年度清单,使用当前有投稿记录的edition及真实venue_aliases,不沿用旧固定快照的missing_years。
- 清单共1266个会议—年份位置,当前有150个位置尚无投稿记录,涉及25个目标会议。
- 其中2025年及以前136个;排除已明确非存档的SysML2018首届,剩**135个历史年度待补采或核实,涉及16个会议**。
- 另有14个2026年度待核实。已知NeurIPS2026和CoRL2026在首轮核查时尚未进入论文集索引,不能一律当已公开论文遗漏;其他年度也需逐个核实发布时间。
这些是“整年没有任何投稿记录”的数量,不是缺论文篇数,也不是完整性百分比。一个年份只有少数记录、只有公开拒稿或缺了某个track,仍会被计作“已有年度”;这些更细缺口及公开拒稿的逐年可获得性尚须继续核验。会议别名共享的联合届次不能当成两批独立论文,未见录用也不能推断拒稿。
| 会议 | 历史缺口年份数 |
| --- | ---: |
| ICML | 29 |
| CVPR | 25 |
| ACL | 16 |
| ECCV | 14 |
| ICCV | 13 |
| ICAPS | 8 |
| WWW | 6 |
| ECML-PKDD | 5 |
| AISTATS | 4 |
| ICLR、COLING、IROS | 各3 |
| ECAI、AAMAS | 各2 |
| RSS、ICASSP | 各1 |
[完整逐年结果](historical-gaps-2026-11-09.json) 保留43目标的官方预期、现有年份数、缺失年份与规范会议匹配。原始只读生产导出保存在 `/data/paper-index/collected/historical-gap-current-20261109-v1.json`。 导出SHA256 `580cdea6dd9869adafe9bc28fb75cf3854bb290ad65aa5a600d9bc368a528b93`。年度对照清单为 `docs/coverage/inventory-2026-11-08.json`;候选来源文件绑定的输入哈希与本目录缺口JSON逐字节一致。来源清单的年份和哈希沿用相应 `*-editions-2026-11-08.json`,不是本轮重新访问所有会议网站;2026发布状态需要后续更新。
## 候选来源与下一步
另一次只读目录核查在现有、已校验的DBLP快照中,为135个历史年度中的**85个**匹配到初步候选论文集,另**50个**尚未按本轮规则匹配到,需要继续识别目录键或寻找官方来源。未匹配不等于会议没举办、未出版或没有公开来源。
[候选卷清单](historical-source-candidates-2026-11-09.json) 保留卷key、标题、出版年份、原始bibliographic counts和出版者链接。初筛使用目标family映射内的四位卷key年份或卷标题唯一显式年份,再按具体会议身份复核;两位键、出版年份与会议年份差异、前身系列及其他目录命名可能漏匹配。原始计数可能含workshop、前言、索引或多个卷,**不能当可直接导入的录用论文数量**。
后续[来源范围复核](historical-source-scope-2026-11-09.md)排除了4个误归ACL的COLING卷,因此候选年度由89修正为85、未匹配年度由46增至50。原卷及排除理由保留在JSON中;135个历史年度缺口不因此改变。ACL1963–1978与COLING1971/1976/1978在当前Anthology目录未列出,继续记为来源待核实,不能一律称作已公开但漏采的论文集。
优先核查已有候选卷较集中的CVPR22年、ICCV12年、ECCV14年和ICML25年。其中ICML早期workshop名称可能属于官方主系列前身,也可能是附属workshop,须对照官方届次证据逐卷区分,不能靠标题包含workshop一刀切。所有候选须经过范围、稳定标识、重复/冲突、原始字段和覆盖预检;这一步没有导入候选卷。
## 复核方法
只读获取当前有记录的edition,不连接papers大文本或标签表:
```sql
WITH counts AS (
SELECT edition_id,count(*) AS submissions
FROM submissions GROUP BY edition_id
)
SELECT e.id,e.venue_id,v.acronym,e.year,e.track,e.event_key,counts.submissions
FROM editions e JOIN venues v ON v.id=e.venue_id
JOIN counts ON counts.edition_id=e.id ORDER BY e.id;
SELECT venue_id,alias FROM venue_aliases ORDER BY venue_id,alias;
```
逐目标取规范会议名或别名相等的edition年份集合,与该目标edition_census.expected_years做集合差;再把2026与明确非存档的SysML2018单列。保留原始导出与校验哈希,不把每个matched year误当逐篇/逐track完整。
与本清单同一阶段的WWW补采只补1995年现有poster轨道,不新增缺失年度,所以即使其15条增量已验收,135这个年度计数也不会随之下降。早期1994年的50条正式展示候选、307条混合节目候选共357条仍需核实;其中并非每条都是论文。
本清单用于明确第14步剩余范围,不代表第14步完成,也不替代所有导入结束后的第16步全库去重、摘要、约束和实际查询验收。
@@ -0,0 +1,33 @@
# ICML 与 IROS 早期主系列的来源范围
本轮修正采集阶段的届次识别,不修改任何现有生产投稿轨道或历史。正式主系列曾以Workshop命名,不等于附属workshop;也不能反过来把所有旧workshop升为主会。
## 已核实的精确范围
| 会议 | 论文集键 | 依据与行为 |
| --- | --- | --- |
| ICML | `conf/icml/1988`、`1989`、`1990`、`1991`、`1992` | [官方历史](https://icml.cc/Conferences/2008/past_icmls.shtml.html)将其列为第5–9届,年份与举办地对应固定目录中的完整标题。1989/1991/1992的Workshop名称不再导致主系列被默认排除;1988/1990保留main并增加来源范围证据 |
| IROS | `conf/iros/1988`、`1989`、`1990` | [IEEE RAS官方历届目录](https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/)列出三届与论文集链接,地点及日期对应固定目录标题;这些是IROS主系列 |
规则同时要求family、booktitle基础别名(ML或IROS)、空轨道后缀、完整精确论文集键、匹配的事件年份及固定完整标题。仍执行原有书目年份核查:与事件年不同且标题无明确事件年时拒绝推断。ML91原始标题没有四位年份,使用精确1991卷键、完整标题、逐篇书目年和官方第8届记录,不从两位标题代码猜世纪。
`1991w`/`1988w`等卷键、booktitle中的Workshops后缀、不同届次/地点/完整标题、未列入的年份均不会获得上述主系列豁免。AISTATS/COLT既有精确规则保留;真正的附属workshop仍受默认不采集workshop的边界约束。
## 原始字段、决定与后续导入
采集器仍保存原始proceedings标题(包括Workshop)、书目年、原始作者名、完整来源书目字段及稳定标识;新增 `track_scope_note` 与官方 `track_scope_evidence_url`。接受状态仍只源自该论文的inproceedings书目证据;不从历史目录生成占位论文或拒稿,也不重新导入已经存在的记录。
新规则的输出必须重新做原始来源/标题/作者/稳定标识与现有库冲突预检,再按增量导入。此前被隔离的ICML1988/1990记录不能仅凭代码规则而直接解除全部冲突。当前正在运行的历史批次使用原固定版本和文件,不能在运行中替换其源码或输入。
## 验证与证据
新增回归覆盖:ICML三届Workshop名称、IROS最初三届、ICML原Conference名称的两届、错误卷键/后缀/标题/未知年份,以及归一化记录经过原生文件导入后保留轨道证据。构建与测试仅在已授权的ces-control远程runner执行,本机仅做语法和文本/JSON一致性检查;结果以相应commit的CI回执为准。
官方HTML在工作证据目录 `paper-index/historical-primary-series-evidence-v1/`,附响应状态、字节数、时间与SHA256;仅取元数据网页,没有下载论文全文。
| 来源 | UTC采样时间 | HTTP | 字节数 | SHA256 |
| --- | --- | ---: | ---: | --- |
| [icml-history](https://icml.cc/Conferences/2008/past_icmls.shtml.html) | 2026-11-08T18:36:43.043135+00:00 | 200 | 11993 | `80cce886fe985551339a52b13c4077acf827e8fcc01555f854dc24077bdebdf0` |
| [iros-history](https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/) | 2026-11-08T18:36:45.687171+00:00 | 200 | 466059 | `fdb4594621e68567e1099f5c88aae53de835b7b8146025148c3b1968df990697` |
本次不关闭历史年度缺口,也不替代全部导入后的全库去重、来源/年度/track覆盖、摘要和实际查询验收。
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,32 @@
# 早期 NLP 来源范围复核
核查时间:2026-11-08 18:27 UTC(北京时间11月9日)。本轮只修正来源候选和记录可获得性,没有操作生产数据库。
## 跨会议候选修正
先前候选清单将 `conf/coling/1965`、`1967`、`1969`、`1973` 匹配到 ACL 年会缺口。这四卷标题明确属于 COLING;[COLING 官方 Anthology 目录](https://aclanthology.org/venues/coling/)将它们列在 COLING 下。出版者写作 ACL 或目标映射包含 COLING,不足以证明某一年是 ACL 联办届次。
从 ACL 候选中排除上述四卷,原始卷字段和排除理由保留在[候选JSON](historical-source-candidates-2026-11-09.json)的 `excluded_cross_venue_candidates` 中。135个历史会议—年份位置仍未关闭;有初步候选来源的年度由89修正为85,尚未匹配的由46增至50。这些数字是目录初筛结果,不是已验证可导入的论文数。
联合届次仍按具体年度的明确会议身份处理,例如有官方联办证据的卷可以属于多个会议。不能一律排除跨family卷,也不能把联合卷重复导入为两批论文。
## 来源待核实的早期年份
| 范围 | 本轮官方来源观察 | 处理 |
| --- | --- | --- |
| ACL 1963–1978,共16年 | [ACL 官方 Anthology 目录](https://aclanthology.org/venues/acl/)目前从1979年列起 | 保留历史举办清单;逐年核实是否有可索引发表材料,不能据会议举办就断言漏采正式论文集,也不据目录缺失断言未出版 |
| COLING 1971、1976、1978 | [主办委员会历史](https://ufal.mff.cuni.cz/iccl-old)列出这三届;当前Anthology COLING目录未列出 | 来源缺口继续开放,需要主办方或出版者逐篇索引;不生成占位论文或录用/拒稿记录 |
这19个位置不能用其他会议的卷补齐,也不能因本轮未找到目录而删除。公开拒稿可获得性是独立核查,不从论文集或录用清单缺失推断。
## 证据与复核
三份原始HTML及 `receipts.json` 保存在工作证据目录 `paper-index/nlp-history-source-review-v1/`;结构化响应URL、时间、状态、字节数和完整SHA256收录于候选JSON的 `scope_review.evidence`。三次响应均为HTTP200;未获取论文全文。
| 官方来源 | 响应SHA256 |
| --- | --- |
| [acl-venue](https://aclanthology.org/venues/acl/) | `d04c188c0769e065f18ca9c0246f70086e2f7027ce3efe5ea0f2cdc641796b72` |
| [coling-venue](https://aclanthology.org/venues/coling/) | `8e7ec0a2915bfbccaa26b7a7558a9cf977c4aadc119794516ecebb826f61623f` |
| [iccl-history](https://ufal.mff.cuni.cz/iccl-old) | `c5286dc51236c90685d22543ff1f7b209adcb3eedb82a7b1ac2115c8dd0169fc` |
复核时先校验保存的原始响应哈希,再读取目录中的会议身份和年度。实时网页若已变化,应另存新响应和时间,不能把新内容套用旧哈希。本次不改最初会议年度清单或生产年度快照,不代表第14步覆盖验收或第16步最终全库审计完成。
+44
View File
@@ -0,0 +1,44 @@
# ICLR 2014–2016 官方历史录用档案
此适配器是长期维护工具的一部分,按需运行。只识别已经独立核对的三种页面布局;后续页面发生变化必须重新核查,不能放松计数或轨道边界以使采集成功。
| 官方页面 | 页面链接次数 | 整理后的不同记录 |
|---|---:|---:|
| [2014 Conference Proceedings](https://iclr.cc/archive/2014/conference-proceedings/) | 35 | main 35 |
| [2015 accepted-main](https://iclr.cc/archive/www/doku.php%3Fid=iclr2015:accepted-main.html) | 116 | main 31、workshop 74 |
| [2016 accepted-main](https://iclr.cc/archive/www/doku.php%3Fid=iclr2016:accepted-main.html) | 80 | main 80 |
2015 的 11 篇 oral 全部再次出现在 poster 段落中。相同年份、轨道及 arXiv 标识的重复展示只生成一条来源记录,但每次展示的标题、作者显示字符串、链接、版本和段落均保留在 `source_variants`。两处标题或解析作者冲突时停止采集、等待人工核查。不同年份或不同轨道不合并投稿。
2014 的作者以分号列出,一条有末尾分号,一条标题和作者斜体块不在普通段落中,部分作者段落拆成多个格式片段;必须读取完整作者段落或独立斜体块,不能只读首个格式片段。2016 的代码链接不属于作者。保留原文,解析结果不替代来源证据。oral、poster 作为 `iclr_archive:presentation` 标签,不改成独立投稿轨道。arXiv 版本留在来源变体中,稳定标识使用去版本号的编号。
## 来源快照与运行
2026-11-09 已冻结的官方响应 SHA256:
- 2014:`a33858176fbffae0066ed3c10310f7dd98a66d12a36334b6339ef1f915ef07af`
- 2015:`131491663742cb084662a4804009b8318e0fa7bc184d4a9044d16dd28f05b888`
- 2016:`7320b8652431793b6ea0743f871ee446bf08b6361ccd4e421b6b53449e71f4f6`
在服务器运行,输出新目录,不下载论文 PDF:
```sh
paper-atlas collect iclr-archive --year 2015 \
--snapshots /data/paper-index/collected/historical-next-source-evidence-v1 \
--out /data/paper-index/collected/iclr-archive-source-v1/2015
# 或重新下载官方 HTML,并保留 HTTP 缓存与收据
paper-atlas collect iclr-archive --year 2016 \
--cache /data/paper-index/cache/iclr-archive \
--out /data/paper-index/collected/iclr-archive-source-v1/2016
```
冻结输入目录必须包含 `receipts.json`,每项有精确 `url`、HTTP `status: 200`、相对 `file`、`bytes`、`sha256` 和带时区的 `finished_at`。读取时重新核对尺寸、哈希和路径;拒绝越界及指向目录外的软链接。已完成输出不可覆盖,应检查报告后另建明确版本目录。解析或计数失败时不生成完成报告,也不导入数据库。
输出分轨道 JSONL、manifest、原始 HTML 证据和年度采集报告。仅支持这三年已审查的页面。2014 workshop、其他未列轨道、逐篇摘要及公开拒稿范围尚未通过这些页面核验,不能据此声称完整。缺席不代表 rejected、withdrawn 或从未被录用。
## 导入前后的边界
这三页合计 220 条不同来源记录,不等于新增 220 篇论文;同一工作可能已存在于 arXiv 或其他会议。先核验来源字段与原始页面,再执行 PostgreSQL 只读预检,比较稳定标识、标题、作者和原始来源。标题/作者差异、版本关联及冲突必须保留证据并明确评估;不能仅靠标题猜测身份,也不能覆盖原有摘要。
采集器不连接数据库。正式导入使用既有 `import records`,必须有新的生产保护基线和独立验收;全部导入结束后仍需全库去重和历史覆盖审计。来自官方录用清单的 accepted 是清单观察,不是原始审稿决定签名或决定日期。
@@ -0,0 +1,201 @@
{
"format": 1,
"source_cache_sha256": "8a65562f3490497cada2919ec77827134beabd5ec9c3690dec3625e2318aa54e",
"records": [
{
"entry_id": "45",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/45/index.html",
"individual_sha256": "83bede6afc9a7039fae2dd1f28274a394418b34f5d5632590b913617580bda1a",
"title": "WISE - World Wide Information System for R&D; Efforts",
"authors": [
"Fritz Loseries",
"Veronika Samara",
"Bodo Urban"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "16",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/16/index.html",
"individual_sha256": "cf76da903eaeaa27d19857fdaa13220f6821559bf4ac22844fae10abbaa6f3f3",
"title": "The World Wide Web as a Tool for Collaborative Inter-Agency Projects: Creation of an Environmental Metadata Database for the Analysis of the Venice Lagoon Ecosystem.",
"authors": [
"Mirko Labbri"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "17",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/17/index.html",
"individual_sha256": "0578fec9c82e8f2f98c4de0f1bcc44011ebb84c53353fdcc423a69cf70673bc2",
"title": "NetRepreneur",
"authors": [
"Kari-Pekka Turtiainen",
"Antti Auer"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "29",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/29/index.html",
"individual_sha256": "624ad3f685de1163411e33e90488975d848a63243fd7ffbed0862c6fb58c4140",
"title": "Mosaic TueV 2.4.3: Multilingual GUI and Improvements towards User Friendliness",
"authors": [
"Mike Bretz",
"Uwe Koch",
"Paul de Bra"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "13",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/13/index.html",
"individual_sha256": "02409321a3403a94d8362a9862f4960386ea9c344fb47f4bcdf9eae128e8b46e",
"title": "Using a WWW Browser as an Alternative User Interface for Interactive Applications",
"authors": [
"Frank Lonczewski"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "36",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/36/index.html",
"individual_sha256": "ed9dada4ea5903cb8f0a4e75f2958416a6e32647fbf3530e58edd9abf26070ce",
"title": "HyperSource: A Hypermedia Program Development And Documentation System",
"authors": [
"Jan Oliver Borchers"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "57",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/57/index.html",
"individual_sha256": "3ad4206ad28a19823172cd2845f636fe7a461c251940e9a93deb8f6484b8a984",
"title": "Progressive Image Transmission Capability (PROTRAC) on the World Wide Web",
"authors": [
"Vojislav Lalich-Petrich",
"Gaurav Bhatia",
"Larry Davis"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "38",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/38/index.html",
"individual_sha256": "940370acc5e6a954b197053fe451e466a904d1f0ac82c8fe4e6ca455faf2c927",
"title": "Measuring the Web with Lycos",
"authors": [
"Michael L. Mauldin"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": "Official abstract listing has no abstract. Canonical HTML has body sections but no explicit abstract; body is not substituted as a summary."
},
{
"entry_id": "05",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/05/index.html",
"individual_sha256": "1858be87d6e7f47476e1cb8fb4793c0fce3b8a6ff653bcfd4a35089e0520f177",
"title": "Virtual Places - Adding people to the web",
"authors": [
"Yosi Mass"
],
"author_source": "author_index",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": "Official abstract listing and canonical HTML have no abstract. Canonical page identifies a contact and links PostScript; no full-text download. Author identity comes from author index, not inferred from contact role."
},
{
"entry_id": "37",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/37/index.html",
"individual_sha256": "b700a0a51b7a82a5f88373ae0283bb369d1cf6f455acff7d91cd7313fa8b3adf",
"title": "The Internet Public Library: Serving a Global Community",
"authors": [
"Susan Schweitzer"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "41",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/41/index.html",
"individual_sha256": "397b474109f9c1240a77d0f490f2b02dd35fae93febbb7397d1b2a31759a8ec1",
"title": "Graphical Browsing in the WorldWideWeb and the MultiMedia Forum",
"authors": [
"Rainer Pagé",
"Klaus Süllow"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "47",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/47/index.html",
"individual_sha256": "52eb4bfdd658730a222c98170b72dabf9cbcc5ffd2ada1e900b826baf2b02833",
"title": "Interoperating Advanced Hypermedia Systems with WWW",
"authors": [
"Jian Zhang",
"Martin Fruehauf",
"Thomas Kirste"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "50",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/50/index.html",
"individual_sha256": "8dc8b0561bd8b1b106c5b209a973c695c3cc9df5288450d4a01fe32541476b7e",
"title": "Web Enabled Agile Manufacturing",
"authors": [
"S. V. Iyer",
"T. Singh",
"U. Ravaioli",
"R. E. DeVor",
"S. G. Kapoor",
"G. Veknatasubramian"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "55",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/55/index.html",
"individual_sha256": "21780cd4047f513e8c9aeced5247002e27cf71719f08de85c6531b57a9e5058d",
"title": "Weblets: Fundamental Building Blocks for WWW Tools",
"authors": [
"Thomas J. Watt Jr."
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
},
{
"entry_id": "59",
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/59/index.html",
"individual_sha256": "fed4cc9dcaee350a12c5e4b6fa9c99be9bead25a81632512d31566400d3d0fad",
"title": "Usenet and the World-Wide Web",
"authors": [
"M. L. Grant"
],
"author_source": "canonical_byline",
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
"missing_abstract_reason": ""
}
]
}
+111
View File
@@ -0,0 +1,111 @@
# Early WWW archive collector
`paper-atlas collect www-archive` is a maintained metadata adapter. It reads official
IW3C2 archive HTML, keeps exact source hashes, and separates importable rows from
program entries that still need review. It fetches no PDF, follows no paper-body or
reference link, and makes no database connection.
For a fresh source snapshot on the server:
```sh
paper-atlas collect www-archive \
--http-cache /data/paper-index/cache/www-archive \
--source-cache /data/paper-index/collected/www-archive-source-v2 \
--out /data/paper-index/collected/www-archive-normalized-v2
```
For reproducible offline replay of the already verified snapshot:
```sh
paper-atlas collect www-archive \
--source-cache /data/paper-index/collected/www-early-archive-source-v1 \
--source-cache-sha256 c9388ccef44486e0684c9ee9e8620a465cdc4574e6bc9ea9391d743e01a3312c \
--out /data/paper-index/collected/www-archive-normalized-v1
```
Use a new output directory for another batch. Existing snapshots and normalized
batches are never overwritten. The manifest hash binds every source URL, byte count,
response hash and cache filename. Plain files must remain directly within the cache;
symlinks, escaping paths, duplicate URLs, non-200 responses and altered bytes fail.
The command uses UTF-8 where valid and explicitly records an ISO-8859-1 fallback for
older pages; original bytes remain authoritative for encoding corrections.
## Evidence and outputs
- WWW3 Spring 1995 papers use `event_key=www-1995-spring`, `track=main`. The author,
session and abstract indexes must have the same nonempty entry-ID set. The ID is
local to this exact event and proceedings section, never a DOI or OpenReview ID.
- Posters remain `track=poster`. An abstract without a linked author-index entry
stays in review even when another index has a placeholder `#` link for it.
- Paper URLs must agree between author and session indexes. A malformed whitespace
URL in the abstract list may use the exact event/entry-ID match only when both
independent indexes corroborate the same clean URL. The original href is retained.
- Canonical names come from the explicitly labeled author index. The source lists
them alphabetically; this is **not** original byline order. Both the raw index names
and the full abstract-page byline/affiliation text remain in metadata. No person ID
or cross-source identity is inferred from a name. Conflicting titles or missing
abstracts stay in review. Topic keywords are not treated as an abstract.
- WWW1 1994 formal presentations and WWW2 1994 mixed categories remain review
candidates. The first list contains uncertain business-track annotations and a
count discrepancy; the second contains panels, tutorials, products and other
activities. Neither presentation listing nor a missing full-text link establishes
a regular-paper acceptance or a rejection. Raw category/title/name claims are kept.
The main and poster JSONL files have independent manifests with explicit event key,
track, count and SHA256. `source_listing_count` records the full recognized abstract
listing, while `held_entry_count` records excluded entries; neither is a ready-row
count. Their status stays `partial`: a verified linked subset does
not prove complete event coverage. `review-candidates.jsonl` contains nested
candidates and reasons, with no importable decision status. It is excluded from
`import-queue.json`. `report.json` accounts for ready/held records and every source
used. Newlines delimit JSONL; Unicode U+2028/U+2029 inside metadata are not line breaks.
The output is published only after all required source and scope checks pass.
## Before importing
This independent collector branch starts from main; it does not include or silently
install the event-aware writer. Its output requires the reviewed integrated runtime
from PR31 (or a main revision containing that implementation). The production event
schema migration has been independently accepted, preserving existing edition1212
and its 59 Fall submissions; that is not an import of the Spring proceedings.
Inspect held entries and original author names, preflight every ready row against
current PostgreSQL stable identifiers and candidate identities, and verify the
source/year/track/event coverage before running `import records`. When a ready subset
excludes held entries, preserve the independently known listing total in the existing
incremental-coverage protocol: bind a freshly checked parent count and expected count,
and set the full expected count from this manifest. A new source uses previous count
0 and previous expected count null; an existing source requires its actual current
values. These live parent values are not guessed by the offline collector. Do not automatically
run this queue against an older writer. Preserve extended versions and independent
submission/decision evidence, and keep unresolved identities held. A normalization
report or passing tests alone do not establish production import, complete historical
coverage or final global deduplication.
Initial fixed-cache replay (64ced7a, CI1057): 38 main papers and38 posters ready;
372 entries held (50 first-conference fields,307 mixed program entries,15 posters).
Independent report44dfd070 verified source hashes,counts and author evidence, finding
one existing exact-title candidate. That publication is explicitly linked to
arXiv1811.12161 by its WWW1995/journal-reference declaration; arXiv1810.07232 is a
different1997 invited chapter and remains separate despite the overlap note. This
is a preparation observation, not evidence of completed production import or final
global deduplication.
## 以单篇官方页面核实待处理 poster
作者索引漏链接、摘要目录使用短标题或标题存在拼写差异时,不放宽全局标题匹配。可以先取得单篇官方 HTML,人工核对 H1、署名区与原目录同一 entry URL,再提供带 SHA256 的逐条核查文件:
```sh
paper-atlas collect www-archive \
--source-cache /data/paper-index/collected/www-poster-reviewed-source-v2 \
--source-cache-sha256 8a65562f3490497cada2919ec77827134beabd5ec9c3690dec3625e2318aa54e \
--reviewed-posters docs/coverage/www-1995-posters-reviewed-2026-11-09.json \
--reviewed-posters-sha256 <核查文件的SHA256> \
--out <新的输出目录>
```
[1995 年15条poster的核查输入](coverage/www-1995-posters-reviewed-2026-11-09.json) 绑定61页来源快照:原46页和补取的15页单篇HTML。程序验证输入文件及快照哈希、单篇页面哈希、完全一致的 canonical H1、原摘要目录中的同一URL/entry ID及作者证据,不通过相似标题猜身份。规范标题采用单篇H1,摘要目录标题及作者索引标题均在 metadata 中保留。
`author_source=canonical_byline` 表示名字出现在H1之后、摘要/关键词或下一节标题之前的署名区域,由逐条核查明确选出,不从全文引用或机构猜作者;保留原始缩写、顺序、重音和拼写。只有 `author_source=author_index` 时使用原作者索引顺序并逐项核对;05的单篇页面写的是联系人,作者身份使用原作者索引证据,不由联系人角色推断。核查输入是审阅后的数据声明,程序的字符串/哈希检查不代替人工识别署名语义。
38和05的来源没有可明确引用的摘要。保留 `abstract=null` 和逐条缺失原因,manifest准确报告 missing_abstract_count,继续标partial;正文、联系方式及PostScript链接不当作摘要,也不下载正文。缺摘要不等于拒稿或未发表。本批输出仍包含原先已验收记录,正式导入前必须只取新增external ID,检查实时覆盖父记录并按incremental模式准备,不能直接重导入全部输出。
+3
View File
@@ -13,6 +13,7 @@ COMMANDS = {
('download',): ('paper_atlas.io.snapshots', {}, '下载并校验固定版本快照'),
('collect','pmlr'): ('paper_atlas.collectors.pmlr', {}, 'PMLR 官方元数据'),
('collect','acl'): ('paper_atlas.collectors.acl', {}, 'ACL Anthology XML'),
('collect','iclr-archive'): ('paper_atlas.collectors.iclr_archive', {}, '核对ICLR 2014–2016官方历史录用档案'),
('collect','iclr-program'): ('paper_atlas.collectors.iclr', {}, 'ICLR 官方公开程序'),
('collect','iclr-snapshot'): ('paper_atlas.collectors.openreview', {}, 'ICLR 公开投稿与决定快照'),
('collect','public-openreview'): ('paper_atlas.collectors.public_openreview', {}, '核验其他会议的公开拒稿及撤稿标签'),
@@ -23,6 +24,8 @@ COMMANDS = {
('collect','mlsys'): ('paper_atlas.collectors.proceedings', {'source':'mlsys'}, 'MLSys 官方历年标题与摘要'),
('collect','cvf'): ('paper_atlas.collectors.vision', {'source':'cvf'}, 'CVPR / ICCV 官方开放目录'),
('collect','ecva'): ('paper_atlas.collectors.vision', {'source':'ecva'}, 'ECCV 官方开放目录'),
('collect','aamas-proceedings'): ('paper_atlas.collectors.aamas', {}, '按官方目录区分 AAMAS 发表形式和摘要'),
('collect','www-archive'): ('paper_atlas.collectors.www_archive', {}, '核验早期WWW官方档案并分开待核查条目'),
('collect','dblp'): ('paper_atlas.collectors.dblp_records', {}, '核对并整理 DBLP 历史发表记录'),
('catalogue','dblp'): ('paper_atlas.collectors.dblp', {}, '盘点 DBLP 历史会议目录'),
('import','arxiv'): ('paper_atlas.ingestion.arxiv', {}, '导入 arXiv 元数据 ZIP'),
+210
View File
@@ -0,0 +1,210 @@
"""Official AAMAS proceedings metadata, with publication format separate from track."""
import argparse
from collections import Counter, defaultdict
import fcntl
import hashlib
import json
from pathlib import Path
import re
import tempfile
from urllib.parse import urljoin, urlsplit
from paper_atlas.db.submissions import timestamp
from paper_atlas.io.html import Document, Node, clean
from paper_atlas.io.http import fetch
SOURCE = 'aamas_proceedings'
ROOT = 'https://www.ifaamas.org/Proceedings/aamas2012/'
# Counts independently checked against the frozen publisher sections. This
# adapter deliberately refuses other editions/layouts until they are reviewed.
SECTIONS = {
'fullpapers': ('Full Papers', 'full_paper', 'proceedings_unspecified', 136),
'extabstracts': ('Extended Abstracts', 'extended_abstract', 'proceedings_unspecified', 146),
'demos': ('Demos', 'demo', 'demo', 28),
'doctorals': ('Doctorial Consortium Abstracts', 'doctoral_abstract', 'doctoral', 24),
}
def classes(node):
return set(node.attrs.get('class', '').split())
def text_without(node, omitted):
if node.tag in ('script', 'style') or classes(node) & omitted:
return ''
parts = [text_without(child, omitted) if isinstance(child, Node) else child
for child in node.children]
value = ''.join(parts)
return ' '+value+' ' if node.tag in ('p', 'div', 'section', 'li', 'br') else value
def one(nodes, description):
if len(nodes) != 1:
raise ValueError('Expected one '+description)
return nodes[0]
def parse_section(html, section, observed_at, evidence):
heading, publication_format, track, _ = SECTIONS[section]
document = Document(html)
titles = [clean(n.text()) for n in document.root.walk() if n.tag == 'title']
if titles != [heading+' - AAMAS 2012']:
raise ValueError('Unexpected proceedings edition or section title')
headings = [clean(n.text()) for n in document.root.walk() if n.tag == 'h1']
if headings != [heading]:
raise ValueError('Unexpected proceedings section heading')
url = ROOT+'resources/'+section+'.html'
records = []
seen = set()
session = None
for node in document.root.walk():
if node.tag == 'h2' and 'section' in classes(node):
session = {'id': node.attrs.get('id'), 'title': clean(node.text())}
if node.tag != 'div' or 'paper' not in classes(node):
continue
paper_id = node.attrs.get('id', '')
if not re.fullmatch(r'[A-Za-z0-9]+_[0-9]+', paper_id) or paper_id in seen:
raise ValueError('Missing or repeated publisher paper ID')
seen.add(paper_id)
title_node = one([n for n in node.walk() if n.tag == 'span' and 'title' in classes(n)], 'paper title')
link = one([n for n in title_node.children if isinstance(n, Node) and n.tag == 'a'], 'paper link')
paper_url = urljoin(url, link.attrs.get('href', ''))
if paper_url != ROOT+'papers/'+paper_id+'.pdf':
raise ValueError('Paper link differs from publisher ID or official host')
title = clean(text_without(link, {'abstract'}))
abstract_nodes = [n for n in link.walk() if n.tag == 'span' and 'abstract' in classes(n)]
if len(abstract_nodes) > 1:
raise ValueError('Multiple abstracts for one paper')
abstract = (clean(abstract_nodes[0].text()) or None) if abstract_nodes else None
author_node = one([n for n in node.walk() if n.tag == 'span' and 'authors' in classes(n)], 'author list')
author_display = clean(text_without(author_node, {'amount'}))
authors = [clean(name) for name in author_display.split(',')]
if not title or not author_display or any(not name for name in authors):
raise ValueError('Empty paper title or author')
tags = [{'namespace': 'aamas_proceedings:publication_format', 'value': publication_format}]
if session:
tags.append({'namespace': 'aamas_proceedings:session', 'value': session['title']})
records.append({
'source': SOURCE, 'external_id': '2012/'+paper_id,
'url': url+'#'+paper_id, 'title': title, 'abstract': abstract,
'authors': authors, 'identifiers': {}, 'venue': 'AAMAS', 'venue_aliases': [],
'year': 2012, 'track': track, 'status': 'accepted',
'observed_at': observed_at,
'raw_decision': 'listed in official AAMAS 2012 proceedings '+heading,
'metadata': {'publisher_paper_id': paper_id, 'paper_url': paper_url,
'section_url': url, 'section_heading': heading,
'publication_format': publication_format, 'session': session,
'authors_display': author_display, 'source_evidence': evidence,
'publication_evidence': 'official published proceedings entry; not an original decision note',
'track_scope_note': ('Full papers and extended abstracts include general and special submission tracks; '
'session topics do not establish an individual submission track.'
if track == 'proceedings_unspecified' else 'Explicit publisher section.'),
'decision_date_precision': 'not_supplied'},
'tags': tags})
if not records:
raise ValueError('No publisher paper entries found')
return records
def read_pages(cache=None, snapshots=None):
if bool(cache) == bool(snapshots):
raise ValueError('Choose exactly one HTTP cache or frozen snapshot directory')
urls = {'introduction': ROOT, **{key: ROOT+'resources/'+key+'.html' for key in SECTIONS}}
pages = {}
receipts = []
frozen = json.loads((Path(snapshots)/'receipts.json').read_text())['receipts'] if snapshots else []
for key, url in urls.items():
if snapshots:
receipt = one([r for r in frozen if r['url'] == url], 'frozen response receipt')
filename = Path(receipt['file'])
if filename.is_absolute() or '..' in filename.parts or receipt.get('status') != 200:
raise ValueError('Unsafe or unsuccessful frozen response')
data = (Path(snapshots)/filename).read_bytes()
if len(data) > 8*1024**2 or hashlib.sha256(data).hexdigest() != receipt['sha256'] or len(data) != receipt['bytes']:
raise ValueError('Frozen publisher response checksum or size mismatch')
observed = receipt['finished_at']
else:
data = fetch(url, cache, 8*1024**2, delay=1)
receipt = json.loads((Path(cache)/hashlib.sha256(url.encode()).hexdigest()).with_suffix('.json').read_text())
observed = receipt['retrieved_at']
evidence = {'url': url, 'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data),
'file': 'evidence/'+hashlib.sha256(data).hexdigest()+'.html', 'observed_at': observed,
'http_status': 200, 'response_receipt': receipt}
pages[key] = (data, evidence)
receipts.append(evidence)
return pages, receipts
def _collect(year, out, *, cache=None, snapshots=None):
if year != 2012:
raise ValueError('Only the independently reviewed AAMAS 2012 layout is supported')
pages, receipts = read_pages(cache, snapshots)
intro = Document(pages['introduction'][0].decode('utf-8'))
if [clean(n.text()) for n in intro.root.walk() if n.tag == 'title'] != ['AAMAS 2012']:
raise ValueError('Unexpected proceedings introduction')
records = []
for section, (_, _, _, expected) in SECTIONS.items():
data, evidence = pages[section]
group = parse_section(data.decode('utf-8'), section, evidence['observed_at'], evidence)
if len(group) != expected:
raise ValueError('Publisher section count differs from reviewed inventory: '+section)
records.extend(group)
if len({r['external_id'] for r in records}) != len(records):
raise ValueError('Paper IDs collide across publisher sections')
grouped = defaultdict(list)
for record in records:
grouped[record['track']].append(record)
out = Path(out)
out.mkdir(parents=True, exist_ok=True)
files = []
with tempfile.TemporaryDirectory(prefix='.aamas-', dir=out) as temporary:
stage = Path(temporary)
(stage/'evidence').mkdir()
for data, evidence in pages.values():
(stage/evidence['file']).write_bytes(data)
for track, group in sorted(grouped.items()):
path = stage/f'aamas-proceedings-2012-{track}.jsonl'
path.write_text(''.join(json.dumps(r, ensure_ascii=False)+'\n' for r in group))
manifest = {'source': SOURCE, 'parser_version': 1, 'venue': 'AAMAS', 'year': 2012, 'track': track,
'count': len(group), 'file': str(out/path.name), 'file_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
'list_url': ROOT, 'status': 'partial', 'missing_abstract_count': sum(not r['abstract'] for r in group),
'publication_format_counts': dict(Counter(r['metadata']['publication_format'] for r in group)),
'note': 'Publisher sections verified; individual full/extended submission track, decision history and independent identities still need review.'}
path.with_suffix('.manifest.json').write_text(json.dumps(manifest, ensure_ascii=False, indent=2)+'\n')
files.append(manifest)
result = {'source': SOURCE, 'year': year, 'finished_at': timestamp(), 'status': 'collected_pending_identity_review',
'records': len(records), 'files': files, 'source_receipts': receipts,
'publication_format_counts': dict(Counter(r['metadata']['publication_format'] for r in records)),
'database_written': False, 'pdfs_downloaded': False,
'note': 'Aggregate submission/withdrawal totals do not identify individual rejected or withdrawn papers.'}
(stage/'aamas-proceedings-2012.json').write_text(json.dumps(result, ensure_ascii=False, indent=2)+'\n')
# Validated files first; write the completed collection report last.
(out/'evidence').mkdir(exist_ok=True)
for path in (stage/'evidence').iterdir():
path.replace(out/'evidence'/path.name)
for path in stage.iterdir():
if path.is_file() and path.name != 'aamas-proceedings-2012.json':
path.replace(out/path.name)
(stage/'aamas-proceedings-2012.json').replace(out/'aamas-proceedings-2012.json')
return result
def collect(year, out, *, cache=None, snapshots=None):
if year != 2012:
raise ValueError('Only the independently reviewed AAMAS 2012 layout is supported')
out = Path(out)
out.mkdir(parents=True, exist_ok=True)
with (out/'aamas.collect.lock').open('a') as lock:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
return _collect(year, out, cache=cache, snapshots=snapshots)
def main(argv=None, *, prog=None):
parser = argparse.ArgumentParser(prog=prog)
parser.add_argument('--year', type=int, choices=[2012], required=True)
parser.add_argument('--out', required=True)
source = parser.add_mutually_exclusive_group(required=True)
source.add_argument('--cache')
source.add_argument('--snapshots')
args = parser.parse_args(argv)
print(json.dumps(collect(args.year, args.out, cache=args.cache, snapshots=args.snapshots), ensure_ascii=False))
+27
View File
@@ -54,7 +54,34 @@ HISTORICAL_WORKSHOP_TITLES = {
}
# Exact pinned volume titles corroborated by official series histories. The
# original title (including Workshop) remains in source metadata. In particular
# ML91 has no four-digit year in its title; its exact key/title and bibliographic
# year are checked independently, rather than guessing a title year.
HISTORICAL_PRIMARY_EVIDENCE = {
'icml': ('ML', 'https://icml.cc/Conferences/2008/past_icmls.shtml.html'),
'iros': ('IROS', 'https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/'),
}
HISTORICAL_PRIMARY_TITLES = {
('icml', 1988): 'Machine Learning, Proceedings of the Fifth International Conference on Machine Learning, Ann Arbor, Michigan, USA, June 12-14, 1988',
('icml', 1989): 'Proceedings of the Sixth International Workshop on Machine Learning (ML 1989), Cornell University, Ithaca, New York, USA, June 26-27, 1989',
('icml', 1990): 'Machine Learning, Proceedings of the Seventh International Conference on Machine Learning, Austin, Texas, USA, June 21-23, 1990',
('icml', 1991): 'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA',
('icml', 1992): 'Proceedings of the Ninth International Workshop on Machine Learning (ML 1992), Aberdeen, Scotland, UK, July 1-3, 1992',
('iros', 1988): "IEEE International Workshop on Intelligent Robots and Systems '88, Proceedings. IROS 1988, Tokyo, Japan, October 31 - November 2, 1988",
('iros', 1989): "IEEE International Workshop on Intelligent Robots and Systems '89, Proceedings. IROS 1989, September 4-6, 1989, Tsukuba, Japan",
('iros', 1990): "IEEE International Workshop on Intelligent Robots and Systems '90, Towards a New Frontier of Applications, Proceedings. IROS 1990, July 3-6, 1990, Ibaraki, Japan",
}
def historical_track(family, base, suffix, volume_key, year, title):
primary = HISTORICAL_PRIMARY_EVIDENCE.get(family)
if (primary and not suffix and base.casefold() == primary[0].casefold()
and volume_key == f'conf/{family}/{year}'
and title == HISTORICAL_PRIMARY_TITLES.get((family, year))):
return ('main',
'Historical primary series; exact proceedings key, complete title and bibliographic/event year corroborated by official series history. Workshop in this original title is not an affiliated track.',
primary[1])
if suffix or base.casefold() != family or volume_key != f'conf/{family}/{year}':
return None
prefix = HISTORICAL_WORKSHOP_TITLES.get((family, year))
+225
View File
@@ -0,0 +1,225 @@
"""Frozen official ICLR 2014–2016 lists; presentation evidence is not a track."""
import argparse
from collections import Counter, defaultdict
import fcntl
import hashlib
import json
from pathlib import Path
import re
import tempfile
from urllib.parse import urlsplit
from paper_atlas.db.submissions import timestamp
from paper_atlas.io.html import Document, Node, clean
from paper_atlas.io.http import fetch
SOURCE = 'iclr_archive'
PAGES = {
2014: ('https://iclr.cc/archive/2014/conference-proceedings/', {'main': 35}),
2015: ('https://iclr.cc/archive/www/doku.php%3Fid=iclr2015:accepted-main.html', {'main': 31, 'workshop': 74}),
2016: ('https://iclr.cc/archive/www/doku.php%3Fid=iclr2016:accepted-main.html', {'main': 80}),
}
SECTIONS = {
2015: {'Main Conference - Oral Presentations': ('main', 'oral'),
'Main Conference - Poster Presentations': ('main', 'poster'),
'Workshop Papers': ('workshop', 'workshop')},
2016: {'Oral Presentations': ('main', 'oral'), 'Poster Presentations': ('main', 'poster')},
}
def arxiv_link(url):
parsed = urlsplit(url)
if parsed.hostname != 'arxiv.org':
return None
match = re.fullmatch(r'/(?:abs|pdf)/(\d{4}\.\d{4,5})(v[1-9]\d*)?(?:\.pdf)?', parsed.path)
if (parsed.scheme not in ('http', 'https') or parsed.netloc != 'arxiv.org'
or parsed.query or parsed.fragment or not match):
raise ValueError('Unexpected official arXiv link')
return match[1], match[2] or None
def without_links(node):
if node.tag in ('a', 'script', 'style'):
return ''
value = ''.join(without_links(n) if isinstance(n, Node) else n for n in node.children)
return ' '+value+' ' if node.tag in ('div', 'li', 'p', 'br') else value
def authors(display, year):
parts = display.rstrip(';').split(';') if year == 2014 else re.split(r',\s*(?:and\s+)?|\s+and\s+', display)
names = [clean(p) for p in parts]
if not names or any(not name for name in names):
raise ValueError('Missing author or unsupported author delimiter')
return names
def parse_page(html, year, evidence):
if year not in PAGES:
raise ValueError('Unreviewed ICLR archive year')
document = Document(html)
nodes = list(document.root.walk())
headings = [clean(n.text()) for n in nodes if n.tag in ('h1', 'h2', 'h3')]
if year == 2014:
if 'Conference Proceedings' not in headings or 'conference papers accepted' not in clean(document.root.text()):
raise ValueError('Unexpected ICLR 2014 accepted list')
elif (set(SECTIONS[year]) - set(headings)
or (year == 2016 and 'Accepted Papers (Conference Track)' not in headings)):
raise ValueError('Unexpected ICLR accepted sections')
entries = []
section = None
for index, node in enumerate(nodes):
if node.tag in ('h1', 'h2', 'h3'):
section = clean(node.text())
if year == 2014:
if node.tag != 'a' or not arxiv_link(node.attrs.get('href', '')):
continue
link = node
# This archived Google Sites page has an entry outside a paragraph.
# Authors can span several differently formatted italic spans.
# Read their whole paragraph, never only the first styled fragment.
display = None
for following in nodes[index+sum(1 for _ in link.walk()):]:
if following.tag == 'a' and arxiv_link(following.attrs.get('href', '')):
break
if following.tag in ('i', 'em') or (following.tag == 'p' and any(
n.tag in ('i', 'em') or 'font-style:italic' in n.attrs.get('style', '').replace(' ', '')
for n in following.walk())):
text = clean(following.text())
if text:
display = text
break
if not display:
raise ValueError('Missing explicit ICLR 2014 author span')
track, presentation = 'main', 'conference'
else:
if node.tag != 'li':
continue
links = [n for n in node.walk() if n.tag == 'a' and arxiv_link(n.attrs.get('href', ''))]
if not links:
continue
if len(links) != 1 or section not in SECTIONS[year]:
raise ValueError('Ambiguous paper link or unknown accepted section')
link, = links
display = clean(without_links(node)).lstrip(',').strip()
track, presentation = SECTIONS[year][section]
arxiv, version = arxiv_link(link.attrs['href'])
title = clean(link.text())
if not title:
raise ValueError('Empty official title')
entries.append({'arxiv': arxiv, 'version': version, 'title': title,
'authors': authors(display, year), 'authors_display': display,
'paper_url': link.attrs['href'], 'track': track,
'section': section, 'presentation': presentation})
# Check every arXiv occurrence, including links outside the expected layout.
occurrences = sum(n.tag == 'a' and bool(arxiv_link(n.attrs.get('href', ''))) for n in nodes)
if len(entries) != occurrences or not entries:
raise ValueError('Some official paper links were not parsed')
grouped = defaultdict(list)
for entry in entries:
grouped[(entry['track'], entry['arxiv'])].append(entry)
result = []
for (track, arxiv), variants in grouped.items():
first = variants[0]
if any((v['title'], v['authors']) != (first['title'], first['authors']) for v in variants):
raise ValueError('Repeated presentation has conflicting title or authors; review required')
presentations = sorted({v['presentation'] for v in variants})
result.append({'source': SOURCE, 'external_id': f'{year}/{track}/{arxiv}',
'url': PAGES[year][0], 'title': first['title'], 'abstract': None,
'authors': first['authors'], 'identifiers': {'arxiv': arxiv},
'venue': 'ICLR', 'venue_aliases': [], 'year': year, 'track': track,
'status': 'accepted', 'observed_at': evidence['observed_at'],
'raw_decision': 'listed in official ICLR accepted '+track+' list',
'metadata': {'source_evidence': evidence, 'source_variants': variants,
'presentations': presentations, 'decision_date_precision': 'not_supplied',
'identity_note': 'Official arXiv link, not title matching; validate canonical identity before import.',
'decision_evidence': 'Archived accepted list, not an original decision note.'},
'tags': [{'namespace': SOURCE+':presentation', 'value': value} for value in presentations]})
return result
def read_page(year, *, cache=None, snapshots=None):
if bool(cache) == bool(snapshots):
raise ValueError('Choose exactly one cache or frozen snapshot directory')
url = PAGES[year][0]
if snapshots:
root = Path(snapshots).resolve()
receipts = json.loads((root/'receipts.json').read_bytes())['receipts']
matches = [r for r in receipts if r['url'] == url]
if len(matches) != 1:
raise ValueError('Expected one exact official response receipt')
receipt, = matches
filename = Path(receipt['file'])
path = root/filename
if (filename.is_absolute() or '..' in filename.parts or not path.resolve().is_relative_to(root)
or receipt['status'] != 200):
raise ValueError('Unsafe or unsuccessful archive snapshot')
data = path.read_bytes()
if (len(data) > 8*1024**2 or len(data) != receipt['bytes']
or hashlib.sha256(data).hexdigest() != receipt['sha256']):
raise ValueError('Archive snapshot checksum or size differs')
observed = timestamp(receipt['finished_at'])
else:
data = fetch(url, cache, 8*1024**2, delay=1)
receipt = json.loads((Path(cache)/hashlib.sha256(url.encode()).hexdigest()).with_suffix('.json').read_bytes())
observed = timestamp(receipt['retrieved_at'])
evidence = {'url': url, 'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data),
'file': 'evidence/'+hashlib.sha256(data).hexdigest()+'.html',
'observed_at': observed, 'http_status': 200, 'response_receipt': receipt}
return data, evidence
def collect(year, out, *, cache=None, snapshots=None):
if year not in PAGES:
raise ValueError('Only independently reviewed ICLR archive years are supported')
out = Path(out)
out.mkdir(parents=True, exist_ok=True)
with (out/'iclr-archive.collect.lock').open('a') as lock:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
report = out/f'iclr-archive-{year}.json'
if report.exists():
raise FileExistsError('Existing completed archive output must be inspected, not overwritten')
data, evidence = read_page(year, cache=cache, snapshots=snapshots)
records = parse_page(data.decode('utf-8'), year, evidence)
counts = Counter(r['track'] for r in records)
if dict(counts) != PAGES[year][1]:
raise ValueError('Distinct archive entries differ from reviewed inventory')
files = []
with tempfile.TemporaryDirectory(prefix='.iclr-archive-', dir=out) as temporary:
stage = Path(temporary)
(stage/'evidence').mkdir()
(stage/evidence['file']).write_bytes(data)
for track in sorted(counts):
group = [r for r in records if r['track'] == track]
path = stage/f'iclr-archive-{year}-{track}.jsonl'
path.write_text(''.join(json.dumps(r, ensure_ascii=False)+'\n' for r in group))
manifest = {'source': SOURCE, 'parser_version': 1, 'venue': 'ICLR', 'year': year,
'track': track, 'count': len(group), 'file': str(out/path.name),
'file_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
'list_url': PAGES[year][0], 'status': 'partial', 'missing_abstract_count': len(group)}
path.with_suffix('.manifest.json').write_text(json.dumps(manifest, ensure_ascii=False, indent=2)+'\n')
files.append(manifest)
result = {'source': SOURCE, 'year': year, 'records': len(records), 'files': files,
'source_evidence': evidence, 'database_written': False, 'pdfs_downloaded': False,
'status': 'collected_pending_identity_review',
'note': 'Accepted list only. Oral/poster are presentation evidence; absent papers are not rejected. '
'ICLR2014 workshop and other unlisted tracks remain outside these reviewed pages.'}
(stage/report.name).write_text(json.dumps(result, ensure_ascii=False, indent=2)+'\n')
(out/'evidence').mkdir(exist_ok=True)
for path in (stage/'evidence').iterdir():
path.replace(out/'evidence'/path.name)
for path in stage.iterdir():
if path.is_file() and path.name != report.name:
path.replace(out/path.name)
(stage/report.name).replace(report)
return result
def main(argv=None, *, prog=None):
parser = argparse.ArgumentParser(prog=prog)
parser.add_argument('--year', type=int, choices=sorted(PAGES), required=True)
parser.add_argument('--out', required=True)
source = parser.add_mutually_exclusive_group(required=True)
source.add_argument('--cache')
source.add_argument('--snapshots')
args = parser.parse_args(argv)
print(json.dumps(collect(args.year, args.out, cache=args.cache, snapshots=args.snapshots), ensure_ascii=False))
+340
View File
@@ -0,0 +1,340 @@
"""Reproducible early WWW metadata; unresolved program entries stay outside imports."""
import argparse
from collections import defaultdict
import hashlib
from html.parser import HTMLParser
import json
from pathlib import Path
import re
import tempfile
from urllib.parse import urljoin, urlsplit
from paper_atlas.io.html import Document, clean
from paper_atlas.io.http import fetch
ROOT = 'https://archives.iw3c2.org/'
SPRING = ROOT+'www3/proceedings/'
FALL = ROOT+'www2/Proceedings/WWW2_Proceedings.html'
SEEDS = [ROOT+'www1/PrelimProcs.html', ROOT+'www1/Proceedings.html', FALL,
SPRING+'proceedings.html', *[SPRING+'papers/'+x+'.html' for x in ('author','session','abstract')],
*[SPRING+'posters/'+x+'.html' for x in ('author','abstract')]]
def decode(raw):
try:
return raw.decode('utf-8-sig'), 'utf-8'
except UnicodeDecodeError:
return raw.decode('iso-8859-1'), 'iso-8859-1 fallback; original bytes retained'
def text(html):
return re.sub(r'[ \t\r\n\f\v]+',' ',Document(html).root.text()).strip(' \t\r\n\f\v')
def anchors(html):
return [(n.attrs.get('href',''),clean(n.text())) for n in Document(html).root.walk() if n.tag=='a']
def source_url(url):
p=urlsplit(url)
if p.scheme!='https' or p.netloc!='archives.iw3c2.org' or p.query or p.fragment or not p.path.endswith('.html'):
raise ValueError('Only exact official archive HTML source URLs are allowed')
return url
class Sources:
def __init__(self, directory, expected_sha256):
self.root=Path(directory).resolve();manifest=self.root/'manifest.json';raw=manifest.read_bytes()
if hashlib.sha256(raw).hexdigest()!=expected_sha256:
raise ValueError('Source cache manifest checksum mismatch')
self.rows={};self.used=set()
for row in json.loads(raw)['sources']:
url=source_url(row['url'])
if url in self.rows or row.get('status')!=200:
raise ValueError('Duplicate or incomplete source response')
p=self.root/row['file']
if p.is_symlink() or p.parent!=self.root or not p.is_file():
raise ValueError('Source cache paths must be plain files directly within the cache')
data=p.read_bytes()
if len(data)>2*1024**2 or len(data)!=row['bytes'] or hashlib.sha256(data).hexdigest()!=row['sha256']:
raise ValueError('Source response checksum or size mismatch')
self.rows[url]={'raw':data,'receipt':row}
def read(self,url):
source_url(url)
if url not in self.rows:
raise ValueError('Required source missing from cache: '+url)
self.used.add(url)
return decode(self.rows[url]['raw'])[0]
def proof(self,urls):
return [{'url':url,'sha256':self.rows[url]['receipt']['sha256'],
'encoding':decode(self.rows[url]['raw'])[1]} for url in sorted(set(urls))]
def download_sources(http_cache, directory):
directory=Path(directory)
if directory.exists():
raise ValueError('Refusing to overwrite a source snapshot')
directory.mkdir(parents=True)
responses={url:fetch(url,http_cache,2*1024**2) for url in SEEDS}
html=decode(responses[FALL])[0]
for href,label in anchors(html):
if href!='Author_Inst.html' and href.endswith('.html'):
url=source_url(urljoin(FALL,href))
if not url.startswith(ROOT+'www2/Proceedings/'):
raise ValueError('Unexpected fall category source')
responses[url]=fetch(url,http_cache,2*1024**2)
receipts=[]
for i,(url,raw) in enumerate(sorted(responses.items())):
name='source-'+str(i).zfill(2)+'.html';(directory/name).write_bytes(raw)
receipts.append({'url':url,'status':200,'file':name,'bytes':len(raw),'sha256':hashlib.sha256(raw).hexdigest()})
manifest=directory/'manifest.json';manifest.write_text(json.dumps({'sources':receipts},indent=2)+'\n')
return hashlib.sha256(manifest.read_bytes()).hexdigest()
def identity(href,track):
url=urljoin(SPRING+track+'/',href);p=urlsplit(url)
match=re.fullmatch(r'/www3/proceedings/'+track+r'/(\d+)/[^?#]+',p.path)
if p.scheme=='https' and p.netloc=='archives.iw3c2.org' and not p.query and not p.fragment and match:
return match[1],url
return None
class AuthorIndex(HTMLParser):
def __init__(self,html,track):
super().__init__(convert_charrefs=True);self.track=track;self.name='';self.bold=None;self.link=None;self.link_text=[]
self.entries=defaultdict(lambda:{'authors':[],'urls':set(),'titles':set()});self.feed(html)
def handle_starttag(self,tag,attrs):
if tag=='b':self.bold=[]
if tag=='a':self.link=dict(attrs).get('href');self.link_text=[]
def handle_data(self,data):
if self.bold is not None:self.bold.append(data)
if self.link is not None:self.link_text.append(data)
def handle_endtag(self,tag):
if tag=='b' and self.bold is not None:
self.name=clean(''.join(self.bold));self.bold=None
if tag=='a' and self.link is not None:
own=identity(self.link,self.track)
if own:
key,url=own;entry=self.entries[key];entry['urls'].add(url);entry['titles'].add(clean(''.join(self.link_text)))
if not self.name:raise ValueError('Paper link has no archive author context')
if self.name not in entry['authors']:entry['authors'].append(self.name)
self.link=None
def sections(html):
"""Use explicit h3 boundaries; old HTML omits many p/li closing tags."""
result=[]
for chunk in re.split(r'<h3\b[^>]*>',html,flags=re.I)[1:]:
parts=re.split(r'</h3\s*>',chunk,maxsplit=1,flags=re.I)
if len(parts)!=2:raise ValueError('Unclosed archive entry heading')
heading,body=parts
body=re.split(r'<(?:hr|h2|address)\b',body,maxsplit=1,flags=re.I)[0]
author=re.search(r'<i\b[^>]*>(.*?)</i\s*>',body,re.I|re.S)
paragraphs=[]
for paragraph in re.findall(r'<p\b[^>]*>(.*?)(?=</p\s*>|<p\b|$)',body,re.I|re.S):
value=text(paragraph)
if value and not value.casefold().startswith('keywords:'):paragraphs.append(value)
result.append({'heading':heading,'title':text(heading),'abstract':'\n\n'.join(paragraphs)or None,
'author_byline_raw':text(author[1])if author else None})
if not result:raise ValueError('No explicit archive entry headings')
return result
def reviewed_posters(sources, path, expected_sha256, cache_sha256):
if path is None:
if expected_sha256:raise ValueError('Reviewed poster path required')
return {},None
raw=Path(path).read_bytes()
if hashlib.sha256(raw).hexdigest()!=expected_sha256:raise ValueError('Reviewed poster checksum mismatch')
review=json.loads(raw)
if set(review)!={'format','source_cache_sha256','records'} or review['format']!=1 or review['source_cache_sha256']!=cache_sha256:
raise ValueError('Reviewed poster snapshot binding differs')
if not isinstance(review['records'],list) or not 0<len(review['records'])<=100:
raise ValueError('Bounded nonempty poster review required')
result={}
for row in review['records']:
if set(row)!={'entry_id','individual_url','individual_sha256','title','authors','author_source','reason','missing_abstract_reason'}:
raise ValueError('Unexpected reviewed poster fields')
key=row['entry_id'];own=identity(row['individual_url'],'posters')
if not isinstance(key,str) or not own or own[0]!=key or key in result:
raise ValueError('Reviewed poster identity duplicated or outside canonical entry')
if row['author_source'] not in ('canonical_byline','author_index') or not isinstance(row['authors'],list) or not row['authors'] or any(not isinstance(a,str) or not a.strip()for a in row['authors']):
raise ValueError('Explicit author evidence required')
if len(set(row['authors']))!=len(row['authors']) or not isinstance(row['reason'],str) or not row['reason'].strip():
raise ValueError('Reviewed author order/reason invalid')
if sources.rows.get(own[1],{}).get('receipt',{}).get('sha256')!=row['individual_sha256']:
raise ValueError('Individual poster page checksum differs')
page=sources.read(own[1]);headings=re.findall(r'<h1\b[^>]*>(.*?)</h1\s*>',page,re.I|re.S)
if len(headings)!=1 or text(headings[0])!=row['title']:
raise ValueError('Reviewed title differs from exact canonical H1')
header=re.split(r'</h1\s*>',page,maxsplit=1,flags=re.I)[1]
header=re.split(r'<h[23]\b',header,maxsplit=1,flags=re.I)[0]
header=re.split(r'\b(?:keywords|abstract)\s*:',text(header),maxsplit=1,flags=re.I)[0]
if row['author_source']=='canonical_byline' and any(a.casefold() not in header.casefold()for a in row['authors']):
raise ValueError('Reviewed author absent from canonical byline region')
result[key]={**row,'canonical_byline_region_raw':header}
return result,hashlib.sha256(raw).hexdigest()
def reviewed_row(sources,key,url,item,authors,review):
if url!=review['individual_url']:
raise ValueError('Reviewed canonical page differs from abstract-entry URL')
if authors and authors['urls']!={url}:
raise ValueError('Reviewed poster author-index URL disagrees')
if review['author_source']=='author_index':
names=[]
for name in authors['authors']if authors else []:
parts=name.split(',');names.append(clean(parts[1]+' '+parts[0])if len(parts)==2 else name)
if names!=review['authors']:raise ValueError('Reviewed author-index evidence differs')
missing=review['missing_abstract_reason']
if not isinstance(missing,str) or bool(missing.strip())==bool(item['abstract']):
raise ValueError('Missing abstract requires explicit reason; never invent a summary')
urls=[SPRING+'posters/author.html',SPRING+'posters/abstract.html',url]
return {'source':'www_archive','external_id':'www3:posters:'+key,'url':url,'title':review['title'],
'abstract':item['abstract'],'authors':review['authors'],'identifiers':{},'venue':'WWW','year':1995,
'track':'poster','event_key':'www-1995-spring','status':'accepted','raw_decision':'listed in official posters proceedings',
'metadata':{'entry_id':key,'source_evidence':sources.proof(urls),'abstract_heading_title_raw':item['title'],
'author_byline_raw':item['author_byline_raw'],'author_index_raw':authors['authors']if authors else [],
'author_index_titles_raw':sorted(authors['titles'])if authors else [],'abstract_href_raw':url,
'author_order':review['author_source'],'reviewed_poster_identity':review,
'missing_abstract_reason':missing or None},
'tags':[{'namespace':'www_archive:proceedings_section','value':'posters'}]}
def spring_rows(sources,track,reviews=None):
base=SPRING+track+'/';author_url=base+'author.html';abstract_url=base+'abstract.html'
authors=AuthorIndex(sources.read(author_url),track).entries
listing=sections(sources.read(abstract_url));by_id={};held=[];session={}
if track=='papers':
session_url=base+'session.html'
for href,title in anchors(sources.read(session_url)):
own=identity(href,track)
if own:
key,url=own
if key in session:raise ValueError('Duplicate main session entry ID')
session[key]=url
for item in listing:
links=[identity(h,track)for h,t in anchors(item['heading'])if identity(h,track)]
if len(links)!=1:
held.append({'reason':'unverified_heading_identity','evidence':abstract_url,'candidate':item});continue
key,url=links[0]
if key in by_id:raise ValueError('Duplicate abstract entry ID')
by_id[key]=(url,item)
if not by_id:raise ValueError('No stable abstract entry IDs')
if track=='papers' and (set(authors)!=set(by_id) or set(session)!=set(by_id)):
raise ValueError('Main author/session/abstract entry ID sets disagree')
if set(authors)-set(by_id):raise ValueError('Author index has entries outside the abstract listing')
reviews=reviews or {}
if set(reviews)-set(by_id):raise ValueError('Reviewed poster outside abstract listing')
rows=[]
for key,(url,item) in by_id.items():
if key in reviews:
if track!='posters':raise ValueError('Reviews only apply to poster entries')
rows.append(reviewed_row(sources,key,url,item,authors.get(key),reviews[key]));continue
if key not in authors:
held.append({'reason':'abstract_entry_missing_linked_author_index','entry_id':key,'evidence':abstract_url,'candidate':item});continue
alternatives=authors[key]['urls']
if len(alternatives)!=1:raise ValueError('Ambiguous author-index paper URL')
canonical=next(iter(alternatives))
if track=='papers' and session[key]!=canonical:raise ValueError('Main author/session paper URLs disagree')
if url!=canonical and (track!='papers' or not re.search(r'\s',url)):
raise ValueError('Uncorroborated archive URL discrepancy')
if not item['abstract']:
held.append({'reason':'missing_abstract','entry_id':key,'evidence':abstract_url,'candidate':item});continue
if item['title'].casefold() not in {t.casefold() for t in authors[key]['titles']}:
held.append({'reason':'abstract_and_author_index_titles_disagree','entry_id':key,'evidence':abstract_url,'candidate':item,'author_index_titles':sorted(authors[key]['titles'])});continue
names=[]
for name in authors[key]['authors']:
parts=name.split(',')
names.append(clean(parts[1]+' '+parts[0])if len(parts)==2 else name)
proof_urls=[author_url,abstract_url]+([session_url]if track=='papers'else[])
rows.append({'source':'www_archive','external_id':'www3:'+track+':'+key,'url':canonical,
'title':item['title'],'abstract':item['abstract'],'authors':names,'identifiers':{},
'venue':'WWW','year':1995,'track':'main'if track=='papers'else'poster','event_key':'www-1995-spring',
'status':'accepted','raw_decision':'listed in official '+track+' proceedings',
'metadata':{'entry_id':key,'source_evidence':sources.proof(proof_urls),
'author_index_raw':authors[key]['authors'],'author_index_titles_raw':sorted(authors[key]['titles']),'author_order':'archive author-index order, not original byline rank',
'author_byline_raw':item['author_byline_raw'],'abstract_href_raw':url,
'url_corroborated_by_author_and_session':url!=canonical},
'tags':[{'namespace':'www_archive:proceedings_section','value':track}]})
return rows,held,{'abstract_entries':len(by_id),'linked_author_entries':len(authors),'ready':len(rows),'held':len(held)}
def early_review(sources):
held=[];url=ROOT+'www1/PrelimProcs.html';html=sources.read(url)
for ordinal,chunk in enumerate(re.split(r'<li\b[^>]*>',html,flags=re.I)[1:],1):
title=re.search(r'<em>Title:</em>\s*<b>(.*?)</b>',chunk,re.I|re.S)
if not title:raise ValueError('Unclassified first-conference entry')
names=re.findall(r'<em>Author:</em>(.*?)(?=<br\b)',chunk,re.I|re.S)
held.append({'reason':'formal_presentation_scope_and_count_need_review','event_key':'www-1994-spring',
'ordinal':ordinal,'evidence':url,'candidate':{'title':text(title[1]),
'authors_raw':[re.sub(r'\s*<[^>]+>','',text(n)).strip()for n in names],
'keywords_raw':text(re.search(r'<em>Keywords:</em>(.*?)(?=<br\b)',chunk,re.I|re.S)[1]) if re.search(r'<em>Keywords:</em>(.*?)(?=<br\b)',chunk,re.I|re.S)else None},
'source_evidence':sources.proof([url])})
if not held:raise ValueError('First-conference presentation listing empty')
category_urls={}
for href,label in anchors(sources.read(FALL)):
if href!='Author_Inst.html' and href.endswith('.html'):
category_url=source_url(urljoin(FALL,href))
if not category_url.startswith(ROOT+'www2/Proceedings/'):raise ValueError('Unexpected fall category')
category_urls[category_url]=label
if not category_urls:raise ValueError('Fall archive category listing empty')
for url,label in sorted(category_urls.items()):
html=sources.read(url)
for ordinal,chunk in enumerate(re.split(r'<dt\b[^>]*>',html,flags=re.I)[1:],1):
fields=re.split(r'<dd\b[^>]*>',chunk,maxsplit=1,flags=re.I)
title=text(fields[0]);raw_byline=text(re.split(r'</dl\b',fields[1],maxsplit=1,flags=re.I)[0]) if len(fields)==2 else None
if title:
held.append({'reason':'mixed_program_category_not_verified_paper_acceptance','event_key':'www-1994-fall',
'category':label,'ordinal':ordinal,'evidence':url,'candidate':{'title':title,'authors_raw':raw_byline},
'source_evidence':sources.proof([FALL,url])})
return held,{'first_conference_fields':sum(r['event_key']=='www-1994-spring'for r in held),'fall_categories':len(category_urls),
'fall_program_entries':sum(r['event_key']=='www-1994-fall'for r in held)}
def collect(source_cache,cache_sha256,out,*,reviewed=None,reviewed_sha256=None):
out=Path(out)
if out.exists():raise ValueError('Refusing to overwrite an existing normalized batch')
sources=Sources(source_cache,cache_sha256);groups={};held=[];counts={}
reviews,review_sha=reviewed_posters(sources,reviewed,reviewed_sha256,cache_sha256)
for track in ('papers','posters'):
rows,review,counts[track]=spring_rows(sources,track,reviews if track=='posters'else None);groups[track]=rows;held.extend(review)
review,counts['earlier']=early_review(sources);held.extend(review)
for item in held:
if 'source_evidence' not in item:item['source_evidence']=sources.proof([item['evidence']])
out.parent.mkdir(parents=True,exist_ok=True)
with tempfile.TemporaryDirectory(prefix='.www-archive-',dir=out.parent)as tmp:
stage=Path(tmp);queue=[]
for track,rows in groups.items():
if not rows:continue
filename='www-1995-spring-'+('main'if track=='papers'else'poster')+'.jsonl';raw=''.join(json.dumps(r,ensure_ascii=False)+'\n'for r in rows).encode();(stage/filename).write_bytes(raw)
manifest={'source':'www_archive','parser_version':2 if review_sha else 1,'venue':'WWW','year':1995,'track':rows[0]['track'],'event_key':'www-1995-spring',
'list_url':SPRING+track+'/abstract.html','count':len(rows),'status':'partial','file_sha256':hashlib.sha256(raw).hexdigest(),
'missing_abstract_count':sum(not bool((r['abstract']or '').strip())for r in rows),'source_listing_count':counts[track]['abstract_entries'],'held_entry_count':counts[track]['held'],'source_cache_sha256':cache_sha256,'scope':'Verified linked archive entries; held entries and other event tracks excluded, not complete event coverage.'}
(stage/filename).with_suffix('.manifest.json').write_text(json.dumps(manifest,indent=2)+'\n');queue.append(str(out/filename))
(stage/'review-candidates.jsonl').write_text(''.join(json.dumps(r,ensure_ascii=False)+'\n'for r in held))
report={'status':'normalized_not_imported','source_cache_sha256':cache_sha256,'counts':counts,'ready':sum(len(r)for r in groups.values()),'held':len(held),
'source_evidence':sources.proof(sources.used),'files':queue,'scope':'Early WWW archive only; no PDF fetching, no rejected/withdrawn inference, no database access. Event-aware importer required.'}
if review_sha:report['reviewed_posters_sha256']=review_sha
(stage/'report.json').write_text(json.dumps(report,ensure_ascii=False,indent=2)+'\n')
(stage/'import-queue.json').write_text(json.dumps({'files':queue},indent=2)+'\n')
stage.rename(out)
return report
def main(argv=None,*,prog=None):
p=argparse.ArgumentParser(prog=prog);p.add_argument('--source-cache',required=True);p.add_argument('--source-cache-sha256');p.add_argument('--http-cache');p.add_argument('--out',required=True)
p.add_argument('--reviewed-posters');p.add_argument('--reviewed-posters-sha256')
args=p.parse_args(argv)
if bool(args.http_cache)==bool(args.source_cache_sha256):p.error('Choose a fresh --http-cache download or an existing --source-cache-sha256 snapshot')
sha=download_sources(args.http_cache,args.source_cache)if args.http_cache else args.source_cache_sha256
if bool(args.reviewed_posters)!=bool(args.reviewed_posters_sha256):p.error('Reviewed poster file and SHA256 must be supplied together')
print(json.dumps(collect(args.source_cache,sha,args.out,reviewed=args.reviewed_posters,reviewed_sha256=args.reviewed_posters_sha256),ensure_ascii=False),flush=True)
if __name__=='__main__':main()
+1 -1
View File
@@ -4,7 +4,7 @@ import re
from paper_atlas.io.publication_ids import mlsys_id
PUBLICATIONS={'pmlr','acl_anthology','colm','neurips','mlsys','cvf','ecva','dblp'}
PUBLICATIONS={'pmlr','acl_anthology','colm','neurips','mlsys','cvf','ecva','dblp','aamas_proceedings'}
def timestamp(value=None):
+151
View File
@@ -0,0 +1,151 @@
"""Publisher evidence, section boundaries and native-ingestion regression checks."""
import hashlib
import json
import re
from pathlib import Path
import sqlite3
import tempfile
import unittest
from unittest.mock import patch
from paper_atlas.collectors import aamas
from paper_atlas.ingestion.records import ingest_file
def paper(pid, abstract=True):
summary = '<span class="abstract">Research with <i>evidence</i>.<br>More detail.</span>' if abstract else ''
return f'''<div class="paper" id="{pid}"><span class="id">{pid}</span>
<p class="paper"><span class="title"><a href="../papers/{pid}.pdf">{summary}
A structured &amp; auditable paper {pid}</a></span><span class="authors">
<a href="authors.html#Jane">Jane Doe<span class="amount">has 7 papers</span></a>, Bob Smith
</span></p></div>'''
def page(section, pid, abstract=True):
heading = aamas.SECTIONS[section][0]
return (f'<html><head><title>{heading} - AAMAS 2012</title></head><body>'
f'<h1>{heading}</h1><h2 class="section" id="s-1A">Session 1A – Robotics I</h2>'
+paper(pid, abstract)+'</body></html>')
class AamasProceedingsTests(unittest.TestCase):
def snapshots(self, root):
root.mkdir()
receipts = []
items = {'introduction': ('<title>AAMAS 2012</title><p>Aggregate withdrawals do not identify papers.</p>', aamas.ROOT)}
for index, section in enumerate(aamas.SECTIONS):
items[section] = (page(section, f'P{index}_1', section != 'extabstracts'), aamas.ROOT+'resources/'+section+'.html')
for key, (html, url) in items.items():
data = html.encode()
(root/(key+'.html')).write_bytes(data)
receipts.append({'url': url, 'status': 200, 'file': key+'.html', 'bytes': len(data),
'sha256': hashlib.sha256(data).hexdigest(), 'finished_at': '2026-11-09T00:00:00+00:00'})
(root/'receipts.json').write_text(json.dumps({'receipts': receipts}))
return root
def test_hidden_abstract_and_author_badges_are_not_title_or_authors(self):
rows = aamas.parse_section(page('fullpapers', '1A_1'), 'fullpapers', '2026-11-09T00:00:00Z', {'sha256': 'proof'})
row, = rows
self.assertEqual(row['title'], 'A structured & auditable paper 1A_1')
self.assertEqual(row['abstract'], 'Research with evidence. More detail.')
self.assertEqual(row['authors'], ['Jane Doe', 'Bob Smith'])
self.assertEqual(row['metadata']['authors_display'], 'Jane Doe, Bob Smith')
self.assertEqual(row['url'], aamas.ROOT+'resources/fullpapers.html#1A_1')
self.assertEqual(row['metadata']['paper_url'], aamas.ROOT+'papers/1A_1.pdf')
self.assertEqual(row['metadata']['source_evidence'], {'sha256': 'proof'})
def test_publication_formats_and_session_topics_do_not_invent_submission_tracks(self):
for section, (_, fmt, track, _) in aamas.SECTIONS.items():
row, = aamas.parse_section(page(section, 'P_1', False), section, '2026-11-09T00:00:00Z', {})
self.assertEqual((row['track'], row['metadata']['publication_format']), (track, fmt))
self.assertIsNone(row['abstract'])
self.assertEqual(row['status'], 'accepted')
self.assertEqual(row['metadata']['decision_date_precision'], 'not_supplied')
self.assertEqual(row['metadata']['session']['title'], 'Session 1A – Robotics I')
self.assertIn({'namespace': 'aamas_proceedings:publication_format', 'value': fmt}, row['tags'])
self.assertEqual(aamas.SECTIONS['fullpapers'][2], 'proceedings_unspecified')
self.assertEqual(aamas.SECTIONS['extabstracts'][2], 'proceedings_unspecified')
def test_changed_edition_section_duplicate_id_or_pdf_link_is_rejected(self):
html = page('fullpapers', '1A_1')
for changed in [html.replace('AAMAS 2012', 'AAMAS 2025'),
html.replace('<h1>Full Papers</h1>', '<h1>Rejected Papers</h1>'),
html.replace('</body>', paper('1A_1')+'</body>'),
html.replace('../papers/1A_1.pdf', 'https://example.org/1A_1.pdf'),
html.replace('../papers/1A_1.pdf', '../papers/1A_2.pdf')]:
with self.subTest(changed=changed):
with self.assertRaises(ValueError):
aamas.parse_section(changed, 'fullpapers', '2026-11-09T00:00:00Z', {})
def test_frozen_responses_require_checksums_success_and_safe_paths(self):
for failure in ['checksum', 'status', 'path']:
with self.subTest(failure=failure), tempfile.TemporaryDirectory() as temporary:
root = self.snapshots(Path(temporary)/'source')
path = root/'receipts.json'
receipts = json.loads(path.read_text())
item = receipts['receipts'][0]
if failure == 'checksum': item['sha256'] = '0'*64
elif failure == 'status': item['status'] = 404
else: item['file'] = '../outside.html'
path.write_text(json.dumps(receipts))
with self.assertRaises(ValueError):
aamas.read_pages(snapshots=root)
def test_reviewed_section_counts_and_year_are_required(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
source = self.snapshots(root/'source')
with self.assertRaises(ValueError):
aamas.collect(2012, root/'out', snapshots=source)
self.assertFalse((root/'out/aamas-proceedings-2012.json').exists())
with self.assertRaises(ValueError):
aamas.collect(2025, root/'future', snapshots=source)
self.assertFalse((root/'future').exists())
def test_ids_colliding_across_sections_do_not_produce_a_completed_collection(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
source = self.snapshots(root/'source')
receipts_path = source/'receipts.json'
receipts = json.loads(receipts_path.read_text())
for item in receipts['receipts'][1:]:
path = source/item['file']
data = re.sub(r'P[0-9]_1', 'SAME_1', path.read_text()).encode()
path.write_bytes(data)
item.update(bytes=len(data), sha256=hashlib.sha256(data).hexdigest())
receipts_path.write_text(json.dumps(receipts))
small = {key: (*value[:3], 1) for key, value in aamas.SECTIONS.items()}
with patch.dict(aamas.SECTIONS, small, clear=True), self.assertRaises(ValueError):
aamas.collect(2012, root/'out', snapshots=source)
self.assertFalse((root/'out/aamas-proceedings-2012.json').exists())
def test_collection_and_native_ingestion_preserve_formats_evidence_and_missing_abstracts(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
source = self.snapshots(root/'source')
small = {key: (*value[:3], 1) for key, value in aamas.SECTIONS.items()}
with patch.dict(aamas.SECTIONS, small, clear=True):
result = aamas.collect(2012, root/'out', snapshots=source)
self.assertEqual(result['records'], 4)
self.assertEqual(len(result['files']), 3)
self.assertFalse(result['database_written'])
self.assertFalse(result['pdfs_downloaded'])
db = root/'sample.sqlite'
for manifest in result['files']:
path = Path(manifest['file'])
self.assertEqual(hashlib.sha256(path.read_bytes()).hexdigest(), manifest['file_sha256'])
for raw in path.read_text().splitlines():
record = json.loads(raw)
evidence = record['metadata']['source_evidence']
self.assertEqual(hashlib.sha256((root/'out'/evidence['file']).read_bytes()).hexdigest(), evidence['sha256'])
ingest_file(db, path)
with sqlite3.connect(db) as connection:
self.assertEqual(connection.execute('SELECT count(*) FROM papers').fetchone()[0], 4)
self.assertEqual(connection.execute('SELECT count(*) FROM submissions').fetchone()[0], 4)
self.assertEqual(connection.execute("SELECT count(*) FROM decisions WHERE status='accepted'").fetchone()[0], 4)
self.assertEqual(connection.execute('SELECT count(*) FROM papers WHERE abstract IS NULL').fetchone()[0], 1)
metadata = [json.loads(r[0]) for r in connection.execute('SELECT metadata_json FROM source_records')]
self.assertEqual({r['publication_format'] for r in metadata}, {'full_paper','extended_abstract','demo','doctoral_abstract'})
namespaces = {r[0] for r in connection.execute('SELECT DISTINCT namespace FROM tags')}
self.assertEqual(namespaces, {'aamas_proceedings:publication_format', 'aamas_proceedings:session'})
self.assertEqual(connection.execute('PRAGMA foreign_key_check').fetchall(), [])
+89
View File
@@ -0,0 +1,89 @@
"""Cross-source publisher identities must preserve attempts and decision history."""
import hashlib
import json
from pathlib import Path
import sqlite3
import tempfile
import unittest
from paper_atlas.db.submissions import identities_for_row
from paper_atlas.ingestion.records import ingest_file
DOI = '10.1234/aamas.identity.example'
def row(source, external_id, *, venue='AAMAS', year=2012, track='demo', status='accepted', identifiers=None):
return {'source': source, 'external_id': external_id,
'url': 'https://example.org/'+source+'/'+external_id,
'title': 'A shared publication with independently recorded evidence',
'abstract': 'An independently recorded source abstract.',
'authors': ['Jane Doe', 'Bob Smith'],
'identifiers': identifiers if identifiers is not None else {'doi': DOI},
'venue': venue, 'year': year, 'track': track, 'status': status,
'observed_at': '2026-11-09T00:00:00Z',
'raw_decision': source+' observed '+status,
'metadata': {'original_source': source}, 'tags': []}
def ingest(db, root, record):
path = root/(record['source']+'-'+record['external_id'].replace('/', '-')+'.jsonl')
path.write_text(json.dumps(record)+'\n')
path.with_suffix('.manifest.json').write_text(json.dumps({
'source': record['source'], 'parser_version': 1, 'venue': record['venue'],
'year': record['year'], 'track': record['track'], 'count': 1,
'list_url': 'https://example.org/'+record['source'], 'status': 'partial',
'file_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
'missing_abstract_count': 0}))
ingest_file(db, path)
class AamasPublicationIdentityTests(unittest.TestCase):
def test_only_accepted_known_publisher_rows_supply_publication_doi_identity(self):
record = row('aamas_proceedings', '2012/D1_1', identifiers={'doi': 'https://doi.org/'+DOI.upper()})
self.assertEqual(identities_for_row(record), [('publication_doi', DOI)])
for status in ['rejected', 'desk_rejected', 'withdrawn', 'under_review', 'unknown']:
with self.subTest(status=status):
self.assertEqual(identities_for_row({**record, 'status': status}), [])
self.assertEqual(identities_for_row({**record, 'source': 'unverified_source'}), [])
def test_same_edition_doi_joins_sources_in_both_orders_without_duplicate_attempts(self):
publisher = row('aamas_proceedings', '2012/D1_1')
bibliography = row('dblp', 'conf/atal/Example12')
for order in [(publisher, bibliography), (bibliography, publisher)]:
with self.subTest(first=order[0]['source']), tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
db = root/'sample.sqlite'
for record in order:
ingest(db, root, record)
ingest(db, root, publisher)
with sqlite3.connect(db) as connection:
self.assertEqual(connection.execute('SELECT count(*) FROM papers').fetchone()[0], 1)
self.assertEqual(connection.execute('SELECT count(*) FROM source_records').fetchone()[0], 2)
self.assertEqual(connection.execute('SELECT count(*) FROM submissions').fetchone()[0], 1)
self.assertEqual(connection.execute('SELECT count(*) FROM submission_sources').fetchone()[0], 2)
self.assertEqual(connection.execute('SELECT count(*) FROM decision_sources').fetchone()[0], 2)
self.assertEqual(connection.execute('SELECT scheme,value FROM submission_identifiers').fetchall(), [('publication_doi', DOI)])
metadata = [json.loads(r[0]) for r in connection.execute('SELECT metadata_json FROM source_records')]
self.assertEqual({r['original_source'] for r in metadata}, {'aamas_proceedings', 'dblp'})
self.assertEqual(connection.execute('PRAGMA foreign_key_check').fetchall(), [])
def test_cross_venue_rejection_and_distinct_tracks_remain_separate_attempts(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
db = root/'sample.sqlite'
rejected = row('openreview_snapshot', 'original-forum', venue='AAAI', year=2011,
track='main', status='rejected', identifiers={'arxiv': '1101.00001', 'openreview': 'original-forum'})
accepted = row('aamas_proceedings', '2012/D1_1', identifiers={'arxiv': '1101.00001', 'doi': DOI})
another_track = row('aamas_proceedings', '2012/F1_1', track='proceedings_unspecified', identifiers={'doi': DOI})
for record in [rejected, accepted, another_track]:
ingest(db, root, record)
with sqlite3.connect(db) as connection:
self.assertEqual(connection.execute('SELECT count(*) FROM papers').fetchone()[0], 1)
self.assertEqual(connection.execute('SELECT count(*) FROM submissions').fetchone()[0], 3)
history = connection.execute('''SELECT v.acronym,e.year,e.track,d.status FROM decisions d
JOIN submissions s ON s.id=d.submission_id JOIN editions e ON e.id=s.edition_id
JOIN venues v ON v.id=e.venue_id ORDER BY v.acronym,e.track''').fetchall()
self.assertEqual(set(history), {('AAAI',2011,'main','rejected'),('AAMAS',2012,'demo','accepted'),('AAMAS',2012,'proceedings_unspecified','accepted')})
self.assertEqual(connection.execute('SELECT count(*) FROM decision_sources').fetchone()[0], 3)
self.assertEqual(connection.execute('PRAGMA foreign_key_check').fetchall(), [])
+115
View File
@@ -0,0 +1,115 @@
"""Regression coverage for primary series once named Workshop, with strict scope boundaries."""
from contextlib import closing
import hashlib
import json
from pathlib import Path
import sqlite3
import tempfile
import unittest
from paper_atlas.collectors.dblp_records import normalize_row
from paper_atlas.collectors.dblp_scope import scope
from paper_atlas.ingestion.records import ingest_file
ICML_HISTORY = 'https://icml.cc/Conferences/2008/past_icmls.shtml.html'
IROS_HISTORY = 'https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/'
class HistoricalPrimarySeriesTests(unittest.TestCase):
def example(self, family, year, title, book=None, key=None):
key = key or f'conf/{family}/{year}'
book = book or ('ML' if family == 'icml' else 'IROS')
row = {'record_type':'inproceedings', 'key':f'conf/{family}/Paper{year}', 'publtype':None,
'fields':{'title':['An independently identified historical research paper'],
'author':['Jane Doe'], 'booktitle':[book], 'year':[str(year)],
'crossref':[key], 'ee':[f'https://doi.org/10.1234/{family}.{year}']}}
volume = {'record_type':'proceedings', 'key':key,
'fields':{'title':[title], 'year':[str(year)]}}
return row, volume
def test_icml_workshop_primary_editions_are_not_silently_filtered(self):
examples = {
1989:'Proceedings of the Sixth International Workshop on Machine Learning (ML 1989), Cornell University, Ithaca, New York, USA, June 26-27, 1989',
1991:'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA',
1992:'Proceedings of the Ninth International Workshop on Machine Learning (ML 1992), Aberdeen, Scotland, UK, July 1-3, 1992',
}
for year, title in examples.items():
with self.subTest(year=year):
row, volume = self.example('icml', year, title)
record, reason = normalize_row(row, {volume['key']:volume}, {'ICML'})
self.assertIsNone(reason)
self.assertEqual((record['venue'],record['year'],record['track'],record['status']), ('ICML',year,'main','accepted'))
self.assertEqual(record['metadata']['proceedings_title'], title)
self.assertEqual(record['metadata']['track_scope_evidence_url'], ICML_HISTORY)
self.assertEqual(record['metadata']['bibliographic_year'], year)
def test_iros_first_three_editions_are_primary_series(self):
examples = {
1988:"IEEE International Workshop on Intelligent Robots and Systems '88, Proceedings. IROS 1988, Tokyo, Japan, October 31 - November 2, 1988",
1989:"IEEE International Workshop on Intelligent Robots and Systems '89, Proceedings. IROS 1989, September 4-6, 1989, Tsukuba, Japan",
1990:"IEEE International Workshop on Intelligent Robots and Systems '90, Towards a New Frontier of Applications, Proceedings. IROS 1990, July 3-6, 1990, Ibaraki, Japan",
}
for year, title in examples.items():
with self.subTest(year=year):
row, volume = self.example('iros', year, title)
record, reason = normalize_row(row, {volume['key']:volume}, {'IROS'})
self.assertIsNone(reason)
self.assertEqual((record['venue'],record['year'],record['track']), ('IROS',year,'main'))
self.assertEqual(record['metadata']['proceedings_title'], title)
self.assertEqual(record['metadata']['track_scope_evidence_url'], IROS_HISTORY)
def test_main_named_icml_predecessors_also_carry_primary_scope_provenance(self):
examples = {
1988:'Machine Learning, Proceedings of the Fifth International Conference on Machine Learning, Ann Arbor, Michigan, USA, June 12-14, 1988',
1990:'Machine Learning, Proceedings of the Seventh International Conference on Machine Learning, Austin, Texas, USA, June 21-23, 1990',
}
for year, title in examples.items():
row, volume = self.example('icml', year, title)
edition, reason = scope(row, volume)
self.assertIsNone(reason)
self.assertEqual(edition['track'], 'main')
self.assertEqual(edition['track_scope_evidence_url'], ICML_HISTORY)
def test_wrong_keys_titles_suffixes_and_unknown_years_keep_workshop_boundary(self):
title = 'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA'
for year, changed, book, key in [
(1991,title,'ML','conf/icml/1991w'),
(1991,title,'ML (Workshops)','conf/icml/1991'),
(1991,title.replace('Eighth','Ninth'),'ML','conf/icml/1991'),
(1991,title + ', Affiliated Workshop','ML','conf/icml/1991'),
(1987,'Machine Learning Workshop 1987, Irvine','ML','conf/icml/1987'),
]:
with self.subTest(year=year, book=book, key=key, title=changed):
row, volume = self.example('icml',year,changed,book,key)
record, reason = normalize_row(row,{volume['key']:volume},{'ICML'})
self.assertIsNone(record)
self.assertEqual(reason,'separate_workshop')
self.assertNotIn('track_scope_evidence_url',scope(row,volume)[0])
title = "IEEE International Workshop on Intelligent Robots and Systems '88, Proceedings. IROS 1988, Tokyo, Japan, October 31 - November 2, 1988"
for changed, book, key in [
(title,'IROS','conf/iros/1988w'),
(title,'IROS (Workshops)','conf/iros/1988'),
(title.replace('Tokyo','Another city'),'IROS','conf/iros/1988'),
]:
row, volume = self.example('iros',1988,changed,book,key)
self.assertEqual(normalize_row(row,{volume['key']:volume},{'IROS'})[1],'separate_workshop')
def test_primary_scope_proof_survives_native_file_ingestion(self):
title = 'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA'
row, volume = self.example('icml',1991,title)
record, reason = normalize_row(row,{volume['key']:volume},{'ICML'})
self.assertIsNone(reason)
with tempfile.TemporaryDirectory() as temporary:
root=Path(temporary);path=root/'records.jsonl';path.write_text(json.dumps(record)+'\n')
path.with_suffix('.manifest.json').write_text(json.dumps({'source':'dblp','count':1,'list_url':ICML_HISTORY,'file_sha256':hashlib.sha256(path.read_bytes()).hexdigest()}))
ingest_file(root/'test.sqlite',path)
with closing(sqlite3.connect(root/'test.sqlite')) as connection:
self.assertEqual(connection.execute('SELECT track FROM editions').fetchone()[0],'main')
self.assertEqual(connection.execute('SELECT status FROM decisions').fetchone()[0],'accepted')
metadata=json.loads(connection.execute('SELECT metadata_json FROM source_records').fetchone()[0])
self.assertEqual(metadata['proceedings_title'],title)
self.assertEqual(metadata['track_scope_evidence_url'],ICML_HISTORY)
if __name__ == '__main__':
unittest.main()
+148
View File
@@ -0,0 +1,148 @@
"""Historical accepted lists, repeated presentations and preserved work histories."""
import hashlib
import json
from pathlib import Path
import sqlite3
import tempfile
import unittest
from unittest.mock import patch
from paper_atlas.collectors import iclr_archive as archive
from paper_atlas.db.store import Store
from paper_atlas.ingestion.records import ingest_file
OBSERVED = '2026-11-09T00:00:00Z'
def entry(identifier, title='A shared research paper', display='Jane Doe and Bob Smith'):
return f'<li><div><a href="http://arxiv.org/abs/{identifier}">{title}</a>, {display}</div></li>'
def page2015():
return ('<h3>Main Conference - Oral Presentations</h3><ol>'+entry('1412.0001v2')+'</ol>'
'<h3>Main Conference - Poster Presentations</h3><ol>'+entry('1412.0001v2')+'</ol>'
'<h3>Workshop Papers</h3><ol>'+entry('1412.0002', 'A workshop paper')+'</ol>')
class IclrArchiveTests(unittest.TestCase):
def frozen(self, root, year, html):
root.mkdir()
data = html.encode()
(root/'page.html').write_bytes(data)
receipt = {'url': archive.PAGES[year][0], 'file': 'page.html', 'status': 200,
'bytes': len(data), 'sha256': hashlib.sha256(data).hexdigest(), 'finished_at': OBSERVED}
(root/'receipts.json').write_text(json.dumps({'receipts': [receipt]}))
return root
def test_oral_poster_repeat_is_one_attempt_with_all_source_variants(self):
rows = archive.parse_page(page2015(), 2015, {'observed_at': OBSERVED})
main, workshop = rows
self.assertEqual((main['track'], workshop['track']), ('main', 'workshop'))
self.assertEqual(main['identifiers'], {'arxiv': '1412.0001'})
self.assertEqual(main['external_id'], '2015/main/1412.0001')
self.assertEqual(main['metadata']['presentations'], ['oral', 'poster'])
self.assertEqual(len(main['metadata']['source_variants']), 2)
self.assertEqual(main['metadata']['source_variants'][0]['version'], 'v2')
self.assertEqual(main['authors'], ['Jane Doe', 'Bob Smith'])
self.assertTrue(all(r['status'] == 'accepted' and r['abstract'] is None for r in rows))
def test_2014_author_span_outside_paragraph_and_trailing_separator(self):
html = ('<h3>Conference Proceedings</h3><p>conference papers accepted</p>'
'<div><b><a href="http://arxiv.org/abs/1312.0001">A <i>research</i> title</a></b></div>'
'<p><span style="font-style: italic">Jane Doe; </span>'
'<span style="font-style:italic">Bob Smith; Carol Wu;</span></p>'
'<div><a href="http://arxiv.org/abs/1312.0002">Another title</a>'
'<div><i><span>First Author; </span><span>Second Author</span></i></div></div>')
row, unwrapped = archive.parse_page(html, 2014, {'observed_at': OBSERVED})
self.assertEqual(row['title'], 'A research title')
self.assertEqual(row['authors'], ['Jane Doe', 'Bob Smith', 'Carol Wu'])
self.assertEqual(row['metadata']['source_variants'][0]['authors_display'], 'Jane Doe; Bob Smith; Carol Wu;')
self.assertEqual(row['track'], 'main')
self.assertEqual(unwrapped['authors'], ['First Author', 'Second Author'])
def test_same_arxiv_in_different_tracks_keeps_separate_source_attempts(self):
html = page2015().replace('1412.0002', '1412.0001v3')
rows = archive.parse_page(html, 2015, {'observed_at': OBSERVED})
self.assertEqual(len(rows), 2)
self.assertEqual({r['external_id'] for r in rows}, {'2015/main/1412.0001', '2015/workshop/1412.0001'})
self.assertEqual(rows[1]['metadata']['source_variants'][0]['version'], 'v3')
def test_2016_code_links_and_breaks_do_not_become_authors(self):
html = ('<h1>Accepted Papers (Conference Track)</h1><h3>Oral Presentations</h3><ol>'
'<li><a href="http://arxiv.org/abs/1511.00001">A title</a>'
'<a href="https://example.org/code">[code]</a><br>Jane Doe, Bob Smith</li></ol>'
'<h3>Poster Presentations</h3>')
row, = archive.parse_page(html, 2016, {'observed_at': OBSERVED})
self.assertEqual(row['authors'], ['Jane Doe', 'Bob Smith'])
self.assertEqual(row['metadata']['presentations'], ['oral'])
def test_conflicting_repeat_unknown_sections_and_unparsed_links_fail(self):
variants = [page2015().replace('Poster Presentations', 'Other Presentations'),
page2015().replace('</ol>', '</ol><a href="http://arxiv.org/abs/1412.0003">stray</a>', 1),
page2015().replace('Jane Doe and Bob Smith', 'Different Author', 1)]
for html in variants:
with self.subTest(html=html), self.assertRaises(ValueError):
archive.parse_page(html, 2015, {'observed_at': OBSERVED})
def test_arxiv_versions_are_preserved_and_unsafe_links_rejected(self):
self.assertEqual(archive.arxiv_link('https://arxiv.org/pdf/1511.00001v3.pdf'), ('1511.00001', 'v3'))
self.assertIsNone(archive.arxiv_link('https://arxiv.org.example.org/abs/1511.00001'))
for url in ['https://arxiv.org/abs/1511.00001?other=1', 'ftp://arxiv.org/abs/1511.00001',
'https://user@arxiv.org/abs/1511.00001', 'https://arxiv.org/abs/not-a-paper']:
with self.subTest(url=url), self.assertRaises(ValueError):
archive.arxiv_link(url)
def test_receipt_hash_status_path_and_symlink_cannot_escape_snapshot(self):
for failure in ['hash', 'status', 'path', 'symlink']:
with self.subTest(failure=failure), tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
source = self.frozen(root/'source', 2015, page2015())
file = source/'receipts.json'
data = json.loads(file.read_text())
receipt = data['receipts'][0]
if failure == 'hash': receipt['sha256'] = '0'*64
elif failure == 'status': receipt['status'] = 404
elif failure == 'path': receipt['file'] = '../outside.html'
else:
(root/'outside.html').write_text(page2015())
(source/'page.html').unlink()
(source/'page.html').symlink_to(root/'outside.html')
file.write_text(json.dumps(data))
with self.assertRaises(ValueError):
archive.read_page(2015, snapshots=source)
def test_changed_inventory_leaves_no_completed_output(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
source = self.frozen(root/'source', 2015, page2015())
with self.assertRaises(ValueError):
archive.collect(2015, root/'out', snapshots=source)
self.assertFalse((root/'out/iclr-archive-2015.json').exists())
def test_native_import_keeps_rejected_to_accepted_history_and_abstract(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
source = self.frozen(root/'source', 2015, page2015())
with patch.dict(archive.PAGES, {2015: (archive.PAGES[2015][0], {'main': 1, 'workshop': 1})}):
result = archive.collect(2015, root/'out', snapshots=source)
with self.assertRaises(FileExistsError):
archive.collect(2015, root/'out', snapshots=source)
db = root/'sample.sqlite'
store = Store(db)
pid, rid = store.record('arxiv', '1412.0001', 'https://arxiv.org/abs/1412.0001',
'A shared research paper', 'Keep this canonical abstract.',
['Jane Doe', 'Bob Smith'], {'arxiv': '1412.0001'}, {})
store.decision(pid, rid, 'Earlier Conference', 2014, 'main', 'original-attempt',
'https://example.org/rejected', 'rejected', 'explicit earlier decision', observed_at=OBSERVED)
store.conn.commit()
store.conn.close()
for manifest in result['files']:
ingest_file(db, manifest['file'])
ingest_file(db, manifest['file'])
with sqlite3.connect(db) as c:
self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0], 2)
self.assertEqual(c.execute('SELECT abstract FROM papers WHERE id=?', (pid,)).fetchone()[0], 'Keep this canonical abstract.')
history = c.execute('SELECT v.acronym,e.year,d.status FROM decisions d JOIN submissions s ON s.id=d.submission_id JOIN editions e ON e.id=s.edition_id JOIN venues v ON v.id=e.venue_id WHERE s.paper_id=? ORDER BY e.year', (pid,)).fetchall()
self.assertEqual(history, [('Earlier Conference', 2014, 'rejected'), ('ICLR', 2015, 'accepted')])
self.assertEqual(c.execute('SELECT count(*) FROM submissions').fetchone()[0], 3)
self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(), [])
+155
View File
@@ -0,0 +1,155 @@
import hashlib
import json
from pathlib import Path
import tempfile
import unittest
from paper_atlas.collectors.www_archive import (ROOT, SPRING, FALL, Sources, collect, decode,
early_review, spring_rows)
class ArchiveTests(unittest.TestCase):
def setUp(self):
t=tempfile.TemporaryDirectory();self.addCleanup(t.cleanup);self.root=Path(t.name);self.cache=self.root/'cache';self.cache.mkdir()
self.pages={ROOT+'www1/PrelimProcs.html':'''<h1>Formal presentations</h1><ol><li><em>Keywords:</em> BUSINESS TRACK?<br><em>Title:</em><b> First contribution</b><br><em>Author:</em>Jane Doe &lt;jane@example.org&gt;<br><li><em>Title:</em><b>Unlinked contribution</b><br><em>Author:</em>John Roe<br></ol>''',
FALL:'<h1>Conference Tracks</h1><ol><li><a href="Agents.html">Agents</a><li><a href="Tutor.html">Tutorials</a></ol>',
ROOT+'www2/Proceedings/Agents.html':'<h3>Panel</h3><dl><dt><a href="a.html">Linked talk</a><dd>Jane Doe<dt>Unlinked talk</a><dd>John Roe</dl>',
ROOT+'www2/Proceedings/Tutor.html':'<h1>Tutorials</h1><dl><dt>Training session<dd>Teacher</dl>'}
for track in ['papers','posters']:
self.pages[SPRING+track+'/author.html']='<b>Doe, Jane</b><ul><li><a href="01/paper.html">Same contribution</a></li></ul><b>Roe, John</b><a href="01/paper.html">Same contribution</a>'
self.pages[SPRING+track+'/abstract.html']='''<h1>Abstracts</h1><a href="#one">Same contribution</a><h3><a href="01/paper.html">Same contribution</a></h3><i>Author: Jane Doe, John Roe<br>Institute</i><p>Actual abstract with <i>inline emphasis</i> and a separator\u2028 preserved.</p><hr><h2>Another session</h2>'''
self.pages[SPRING+'papers/session.html']='<a href="01/paper.html">Same contribution</a>'
def sources(self):
rows=[]
for i,(url,html) in enumerate(sorted(self.pages.items())):
raw=html.encode()if isinstance(html,str)else html;name='source-'+str(i)+'.html';(self.cache/name).write_bytes(raw)
rows.append(dict(url=url,status=200,file=name,bytes=len(raw),sha256=hashlib.sha256(raw).hexdigest()))
p=self.cache/'manifest.json';p.write_text(json.dumps({'sources':rows}));sha=hashlib.sha256(p.read_bytes()).hexdigest()
return Sources(self.cache,sha),sha
def test_offline_batch_separates_tracks_review_and_importable_rows(self):
_,sha=self.sources();out=self.root/'batch';report=collect(self.cache,sha,out)
self.assertEqual(report['ready'],2);self.assertEqual(report['held'],5)
queue=json.loads((out/'import-queue.json').read_text())['files'];self.assertEqual(len(queue),2)
for name in queue:
p=Path(name);rows=[json.loads(l)for l in p.read_text().split('\n') if l];self.assertEqual(len(rows),1)
row=rows[0];self.assertEqual(row['event_key'],'www-1995-spring');self.assertEqual(row['authors'],['Jane Doe','John Roe'])
self.assertIn(row['track'],['main','poster']);self.assertIn('inline emphasis',row['abstract']);self.assertNotIn('Another session',row['abstract'])
self.assertIn('\u2028',row['abstract']);manifest=json.loads(p.with_suffix('.manifest.json').read_text());self.assertEqual(manifest['file_sha256'],hashlib.sha256(p.read_bytes()).hexdigest());self.assertEqual(manifest['status'],'partial')
held=[json.loads(l)for l in(out/'review-candidates.jsonl').read_text().split('\n') if l]
self.assertTrue(all('status'not in r and 'source_evidence'in r for r in held))
self.assertFalse(any('review-candidates'in p for p in queue))
def test_partial_manifest_preserves_full_listing_size_separately_from_ready_count(self):
self.pages[SPRING+'posters/abstract.html']+='<h3><a href="02/other.html">Unlinked poster</a></h3><i>Author: Other Person</i><p>Actual abstract</p>'
_,sha=self.sources();out=self.root/'batch';collect(self.cache,sha,out)
manifest=json.loads((out/'www-1995-spring-poster.manifest.json').read_text())
self.assertEqual((manifest['count'],manifest['source_listing_count'],manifest['held_entry_count']),(1,2,1))
self.assertNotIn('coverage_previous_count',manifest) # Live parent must be independently preflighted.
def test_malformed_href_requires_exact_id_and_two_clean_index_anchors(self):
self.pages[SPRING+'papers/abstract.html']=self.pages[SPRING+'papers/abstract.html'].replace('01/paper.html','01/ paper.html')
sources,_=self.sources();rows,held,_=spring_rows(sources,'papers');self.assertEqual(held,[])
self.assertEqual(rows[0]['url'],SPRING+'papers/01/paper.html');self.assertTrue(rows[0]['metadata']['url_corroborated_by_author_and_session'])
self.assertIn(' ',rows[0]['metadata']['abstract_href_raw'])
def test_main_index_disagreement_fails_without_publishing_batch(self):
self.pages[SPRING+'papers/session.html']='<a href="02/other.html">Other work</a>';_,sha=self.sources();out=self.root/'batch'
with self.assertRaisesRegex(ValueError,'ID sets disagree'):collect(self.cache,sha,out)
self.assertFalse(out.exists())
def test_same_entry_id_with_conflicting_clean_url_is_rejected(self):
self.pages[SPRING+'papers/session.html']='<a href="01/other.html">Same contribution</a>';sources,_=self.sources()
with self.assertRaisesRegex(ValueError,'URLs disagree'):spring_rows(sources,'papers')
def test_poster_without_linked_author_entry_is_held_even_with_abstract(self):
self.pages[SPRING+'posters/author.html']='<b>Doe, Jane</b><a href="#">Same contribution</a>';sources,_=self.sources();rows,held,counts=spring_rows(sources,'posters')
self.assertEqual(rows,[]);self.assertEqual(counts['abstract_entries'],1);self.assertEqual(held[0]['reason'],'abstract_entry_missing_linked_author_index')
def test_unlinked_and_panel_entries_never_become_acceptance_or_rejection(self):
sources,_=self.sources();held,counts=early_review(sources);self.assertEqual(counts,{'first_conference_fields':2,'fall_categories':2,'fall_program_entries':3})
self.assertTrue(any(r['candidate']['title']=='Unlinked talk'for r in held));self.assertTrue(all('status'not in r for r in held))
self.assertEqual(held[0]['candidate']['authors_raw'],['Jane Doe'])
def test_title_disagreement_goes_to_review_with_source_claims(self):
self.pages[SPRING+'papers/abstract.html']=self.pages[SPRING+'papers/abstract.html'].replace('Same contribution','Different contribution');sources,_=self.sources();rows,held,_=spring_rows(sources,'papers')
self.assertEqual(rows,[]);self.assertEqual(held[0]['reason'],'abstract_and_author_index_titles_disagree')
def test_cache_tampering_and_manifest_hash_mismatch_fail_closed(self):
_,sha=self.sources()
with self.assertRaisesRegex(ValueError,'manifest checksum'):Sources(self.cache,'0'*64)
p=next(self.cache.glob('source-*.html'));p.write_bytes(p.read_bytes()+b'x')
with self.assertRaisesRegex(ValueError,'checksum or size'):Sources(self.cache,sha)
def test_cache_cannot_read_paths_outside_snapshot(self):
_,sha=self.sources();p=self.cache/'manifest.json';d=json.loads(p.read_text());d['sources'][0]['file']='../outside.html';p.write_text(json.dumps(d))
with self.assertRaisesRegex(ValueError,'plain files'):Sources(self.cache,hashlib.sha256(p.read_bytes()).hexdigest())
def test_keywords_only_is_missing_abstract_not_invented_summary(self):
self.pages[SPRING+'papers/abstract.html']='<h3><a href="01/paper.html">Same contribution</a></h3><i>Author: Jane Doe</i><p>Keywords: networking, hypertext</p>';sources,_=self.sources();rows,held,_=spring_rows(sources,'papers')
self.assertEqual(rows,[]);self.assertEqual(held[0]['reason'],'missing_abstract')
def test_old_encoding_and_duplicate_abstract_identity(self):
self.assertEqual(decode(b'M\xfcller')[0],'Müller')
self.pages[SPRING+'papers/abstract.html']*=2;sources,_=self.sources()
with self.assertRaisesRegex(ValueError,'Duplicate abstract'):spring_rows(sources,'papers')
def test_existing_output_is_never_overwritten(self):
_,sha=self.sources();out=self.root/'batch';out.mkdir();(out/'keep').write_text('original')
with self.assertRaisesRegex(ValueError,'overwrite'):collect(self.cache,sha,out)
self.assertEqual((out/'keep').read_text(),'original')
def review(self,*,authors=None,author_source='canonical_byline',missing='',page=None):
url=SPRING+'posters/01/paper.html'
self.pages[url]=page or '<h1>Canonical full title</h1><p>Jane Doe, John Roe</p><p>Keywords: web</p><h2>Abstract</h2><p>Body</p>'
_,sha=self.sources();raw=self.pages[url].encode()if isinstance(self.pages[url],str)else self.pages[url]
row=dict(entry_id='01',individual_url=url,individual_sha256=hashlib.sha256(raw).hexdigest(),
title='Canonical full title',authors=authors or ['Jane Doe','John Roe'],author_source=author_source,
reason='Reviewed exact canonical entry and its explicit authorship; retain original title variants',missing_abstract_reason=missing)
p=self.root/'review.json';p.write_text(json.dumps({'format':1,'source_cache_sha256':sha,'records':[row]}));return sha,p,hashlib.sha256(p.read_bytes()).hexdigest()
def test_reviewed_variant_keeps_primary_title_and_all_old_title_claims(self):
sha,p,h=self.review();out=self.root/'reviewed';r=collect(self.cache,sha,out,reviewed=p,reviewed_sha256=h)
row=json.loads((out/'www-1995-spring-poster.jsonl').read_text());self.assertEqual(row['title'],'Canonical full title')
self.assertEqual(row['metadata']['abstract_heading_title_raw'],'Same contribution');self.assertEqual(row['metadata']['author_index_titles_raw'],['Same contribution'])
self.assertEqual(len(row['metadata']['source_evidence']),3);self.assertEqual(r['reviewed_posters_sha256'],h)
self.assertEqual(row['external_id'],'www3:posters:01');self.assertEqual(row['status'],'accepted')
def test_canonical_byline_can_resolve_unlinked_author_index_without_guessing(self):
self.pages[SPRING+'posters/author.html']='<b>Doe, Jane</b><a href="#">Broken index link</a>'
sha,p,h=self.review();out=self.root/'reviewed';collect(self.cache,sha,out,reviewed=p,reviewed_sha256=h)
row=json.loads((out/'www-1995-spring-poster.jsonl').read_text());self.assertEqual(row['authors'],['Jane Doe','John Roe'])
self.assertEqual(row['metadata']['author_index_raw'],[]);self.assertEqual(row['metadata']['author_order'],'canonical_byline')
def test_reviewed_missing_abstract_is_explicit_partial_not_body_summary(self):
self.pages[SPRING+'posters/abstract.html']='<h3><a href="01/paper.html">Same contribution</a></h3><i>Author: Jane Doe</i><p>Keywords: web</p>'
sha,p,h=self.review(missing='Neither listing nor canonical page supplies an explicit abstract')
out=self.root/'reviewed';collect(self.cache,sha,out,reviewed=p,reviewed_sha256=h)
row=json.loads((out/'www-1995-spring-poster.jsonl').read_text());m=json.loads((out/'www-1995-spring-poster.manifest.json').read_text())
self.assertIsNone(row['abstract']);self.assertEqual((m['missing_abstract_count'],m['status']),(1,'partial'))
sha,p,h=self.review()
with self.assertRaisesRegex(ValueError,'Missing abstract'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
self.assertFalse((self.root/'bad').exists())
def test_reviewed_names_must_be_in_byline_region_not_citations_or_contact_guess(self):
sha,p,h=self.review(authors=['Unrelated Person'],page='<h1>Canonical full title</h1><p>Jane Doe</p><h2>Abstract</h2><p>Unrelated Person cited here</p>')
with self.assertRaisesRegex(ValueError,'byline region'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
sha,p,h=self.review(author_source='author_index',authors=['Contact Person'])
with self.assertRaisesRegex(ValueError,'author-index'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
def test_review_hash_cache_binding_and_canonical_h1_are_all_required(self):
sha,p,h=self.review()
with self.assertRaisesRegex(ValueError,'checksum'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256='0'*64)
d=json.loads(p.read_text());d['source_cache_sha256']='0'*64;p.write_text(json.dumps(d));h=hashlib.sha256(p.read_bytes()).hexdigest()
with self.assertRaisesRegex(ValueError,'binding'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
d['source_cache_sha256']=sha;d['records'][0]['title']='Wrong title';p.write_text(json.dumps(d));h=hashlib.sha256(p.read_bytes()).hexdigest()
with self.assertRaisesRegex(ValueError,'canonical H1'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
def test_unlisted_review_entry_and_changed_primary_digest_fail_closed(self):
sha,p,h=self.review();d=json.loads(p.read_text());d['records'][0]['individual_sha256']='0'*64;p.write_text(json.dumps(d));h=hashlib.sha256(p.read_bytes()).hexdigest()
with self.assertRaisesRegex(ValueError,'page checksum'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
self.pages[SPRING+'posters/abstract.html']=self.pages[SPRING+'posters/abstract.html'].replace('01/paper.html','02/other.html')
self.pages[SPRING+'posters/author.html']=self.pages[SPRING+'posters/author.html'].replace('01/paper.html','02/other.html')
sha,p,h=self.review()
with self.assertRaisesRegex(ValueError,'outside abstract'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)