Compare commits
22 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| b5c0618be5 | |||
| 81864e48ae | |||
| 348ebe0695 | |||
| 02a408ba8f | |||
| 714f745bd5 | |||
| cc78e837d2 | |||
| c0b2146ae8 | |||
| ae88367fac | |||
| a305d00c75 | |||
| 57fcfc7693 | |||
| a07b1e42a8 | |||
| 7c6f4c9f1d | |||
| 3f0ac694a0 | |||
| 32c7edd0a5 | |||
| 658d799334 | |||
| 53fbe9c75c | |||
| dac6edbe1e | |||
| 7a674ac797 | |||
| eacc1e3292 | |||
| 2275bc7530 | |||
| 9040649320 | |||
| 64ced7af42 |
@@ -48,6 +48,9 @@ jobs:
|
||||
outside.mkdir()
|
||||
subprocess.run([str(venv/'bin/paper-atlas'),'--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','cvf','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','www-archive','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','aamas-proceedings','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','iclr-archive','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','postgres','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','edition-tracks','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','joint-editions','--help'],cwd=outside,check=True)
|
||||
|
||||
@@ -6,6 +6,8 @@ Paper Atlas 将 arXiv、AI 会议论文集及公开投稿决定整理到可追
|
||||
|
||||
项目仍在建设中。候选范围包含 43 个会议(含 ACM MM、MLSys),尚未完成全部历史数据和最终去重复核。实际进度以数据库、来源清单和覆盖报告为准,而不是仓库文件数量。
|
||||
|
||||
历史 ICLR 2014–2016 官方录用页由 `collect iclr-archive` 适配,保留主会/workshop及重复展示证据;运行和身份核验边界见 [历史档案说明](docs/coverage/iclr-archive-2014-2016.md)。
|
||||
|
||||
## 项目结构
|
||||
|
||||
```text
|
||||
@@ -60,6 +62,10 @@ paper-atlas collect colm --cache /data/paper-index/raw/http-cache \
|
||||
--out /data/paper-index/collected
|
||||
```
|
||||
|
||||
早期主系列的Workshop名称按[ICML/IROS精确届次证据](docs/coverage/historical-primary-series-2026-11-09.md)识别,保留原名与范围证据;代码支持不代表已补齐生产数据。
|
||||
|
||||
AAMAS 的[官方论文集采集器](docs/aamas-proceedings.md)目前核验了 2012 版式,可从缓存或已校验快照采集标题、作者及页面提供的摘要;发表形式单独保存,全文/扩展摘要中未明确的投稿 track 保持待明确,不根据会话主题推定主会轨道。
|
||||
|
||||
`paper-atlas collect --help` 查看来源;`paper-atlas import --help` 查看导入方式。完整批次步骤见[使用指南](docs/usage.md)。
|
||||
|
||||
共享实例名为 `foundry-postgres`,数据位于 `foundry_data` 库的 `research` schema;其他项目可按权限复用同一组表。库名通过 libpq service 配置、schema 通过 search_path 或 `--pg-schema` 指定,不写死在查询中。PostgreSQL 凭据放在服务器权限为 `0600` 的 libpq service 文件中;命令行只使用服务名称。`paper-atlas maintain postgres --help` 提供迁移入口:先完整核验 21 张业务表,再发布目标 schema。SQLite FTS5 内部表由 PostgreSQL GIN 索引替代。切换后 SQLite 为只读备份;`import records`、`import arxiv` 和 `maintain backfill` 使用 `--pg-service foundry-research` 写入主库,整个写入会话只允许一个协调写入者,会议文件失败时整体回滚并保留失败批次记录。
|
||||
@@ -88,6 +94,7 @@ CI 在已授权的远程 runner 上运行测试、构建 wheel,并从源码目
|
||||
|
||||
开发通过工作分支和 PR 交付,由用户手动合并。
|
||||
|
||||
早期WWW档案的可重放元数据采集、论文/海报分轨与待核查条目见 [WWW档案采集](docs/www-archive.md)。
|
||||
历史联合会议的既有届次归属可通过 [`maintain joint-editions`](docs/joint-editions.md) 先只读规划,再按计划哈希执行;保留论文与投稿决定来源,不重导入。
|
||||
同年多届的 `event_key`、显式迁移、旧文件刷新与查询边界见 [会议事件身份](docs/event-identity.md)。
|
||||
研究类型与自定义标签可通过 [`annotate`](docs/annotations.md) 显式初始化、添加和撤回;两者独立查询,保留证据及完整更正历史,不从会议轨道推断研究类型。
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
# AAMAS 官方论文集元数据
|
||||
|
||||
`paper_atlas.collectors.aamas` 是按需运行的采集适配器,通过统一入口 `collect aamas-proceedings` 使用。目前只支持已独立核验的 **AAMAS 2012** HTML 版式;其他年份需先核对官方目录和逐篇结构,再增加适配,不能仅替换年份。
|
||||
|
||||
## 来源与范围
|
||||
|
||||
[官方论文集介绍](https://www.ifaamas.org/Proceedings/aamas2012/)及其 Full Papers、Extended Abstracts、Demos、Doctorial Consortium Abstracts 四个栏目分别列出 136、146、28、24 条,共 334 条。旧 DBLP 候选卷的 310 条不能替代这四类官方目录,也不能全部标记为主会全文。栏目数量来自 2026-11-09 冻结的原始页面;页面结构或数量发生变化时,采集器停止,留待重新核验。
|
||||
|
||||
采集标题、逐篇作者、页面实际提供的摘要、官方条目 ID、目录锚点和 PDF 地址。**只保存 PDF 链接,不下载 PDF。** 全文页面的摘要藏在标题链接内部的 `abstract` 元素,作者链接里的 “has N papers” 是界面徽标,二者不会混入标题或姓名。未提供的摘要保存为 `null`,不会用标题、说明文字或其他论文填充。
|
||||
|
||||
官方介绍说明了 general、robotics、virtual agents、innovative applications 等投稿轨道,但逐篇目录的主题会话不能独立证明某篇论文的投稿轨道。因此全文和扩展摘要均使用 `proceedings_unspecified`;演示、博士栏目使用 `demo`、`doctoral`。发表形式保留在 `metadata.publication_format` 和 `aamas_proceedings:publication_format` 标签中,会话原文单独保存。它们不是研究类型标注,也不会写入自定义 annotation。
|
||||
|
||||
目录条目仅提供发表录用证据,不是原始决定信。`observed_at` 是页面采集时间,不能当作 2012 年的决定日期;官方介绍中的投稿、撤稿、录用汇总数不能识别逐篇拒稿或撤稿,采集器不会由总量相减生成这些决定。
|
||||
|
||||
## 运行
|
||||
|
||||
在已确认的远程运行环境中,使用 HTTP 缓存采集:
|
||||
|
||||
```sh
|
||||
paper-atlas collect aamas-proceedings --year 2012 \
|
||||
--cache /data/paper-index/cache/aamas \
|
||||
--out /data/paper-index/collected/aamas-2012
|
||||
```
|
||||
|
||||
也可使用已冻结的官方页面:
|
||||
|
||||
```sh
|
||||
paper-atlas collect aamas-proceedings --year 2012 \
|
||||
--snapshots /data/paper-index/collected/historical-next-source-evidence-v1 \
|
||||
--out /data/paper-index/collected/aamas-2012
|
||||
```
|
||||
|
||||
快照目录需含 `receipts.json`,其中每个响应包含原始 `url`、HTTP `status`、相对 `file`、`bytes`、`sha256` 和 `finished_at`。要求五个准确官方 URL 均唯一匹配且返回 200;验证全部字节数和哈希后才解析,不接受目录穿越或绝对文件路径。快照中其他来源的响应不会被当成 AAMAS 页面。
|
||||
|
||||
输出为按数据库 track 分组的三个 JSONL 文件和对应 manifest、五份原始 HTML 证据,以及最后写出的 `aamas-proceedings-2012.json` 采集报告。锁文件避免同一输出目录并发运行。manifest 始终为 `partial`,记录发表形式数量和缺失摘要数;**采集命令不连接数据库,也不导入。**
|
||||
|
||||
## 复核与恢复
|
||||
|
||||
每条记录携带对应官方页面的 URL、采集时间、字节数、SHA-256 与输出内证据路径。重新运行前先检查已有采集报告和文件哈希;采集中断不等于主库已导入,报告缺失或哈希不一致的目录不能直接作为导入队列。输出中的原始证据应随文件一起保存。
|
||||
|
||||
生产导入前仍需单独与主库核验稳定标识、标题、作者及已有投稿,按证据补充 DOI/arXiv 等标识;目录页上的作者计数徽标和会话 ID 不能作为这些标识。需要另建经过核验的增量清单、保护旧历史,完成独立验收后才能计入覆盖。跨来源同一论文不得仅因同题就新增或合并投稿。
|
||||
|
||||
本功能的代码、测试或采集成功均不代表 AAMAS 2012 已导入生产,也不证明拒稿全集完整。全库最终去重、摘要和来源覆盖验收属于独立后续阶段。
|
||||
@@ -190,3 +190,7 @@ WWW早期逐来源核查见 `www-early-proceedings-2026-11-08.json`:1995春季
|
||||
WWW事件标识生产迁移已独立验收(报告SHA `f5dbb8a8`):保留1399个原届次字段及IDs、59个Fall投稿和完整全库历史哈希。服务器已缓存46个官方HTML来源(约223KB),只作为后续维护式采集器的输入;尚未补导早期WWW论文,也未启用可选注解表。
|
||||
|
||||
IJCAI 联办届次的逐条后续核查见 [2018/2022 分区与目录差集](ijcai-sections-2026-11-08.md) 和 [稳定 ID 映射](ijcai-sections-2026-11-08.json)。已定位 345 条非 main 分类及 2022 官方 HTML 缺失但官方文件仍存在的一条记录;生产分类尚未修正,不能把整卷数量作为 main 覆盖率。
|
||||
|
||||
2026年11月9日的[当前历史年度缺口核对](historical-gaps-2026-11-09.md)单列135个历史会议—年份、14个2026年度及85个经范围修正的初步候选来源年度;它使用当前生产edition/别名快照,保留原固定快照不变。年度有记录不等于逐篇、逐track或公开拒稿历史完整。
|
||||
|
||||
[早期NLP来源范围复核](historical-source-scope-2026-11-09.md)保留3份官方网页的响应哈希,排除4个COLING卷对ACL缺口的误匹配,并区分会议举办记录、目录可获取性与论文级覆盖;尚未导入或关闭这些年度缺口。
|
||||
|
||||
@@ -0,0 +1,867 @@
|
||||
{
|
||||
"snapshot_at": "2026-11-08 16:34:40.741334+00:00",
|
||||
"scope": "Official first-pass conference-year census vs current populated editions and real aliases. Missing year means no imported attempt in that year, not missing paper count or proof all other years complete. Includes announced/nonarchival exceptions below.",
|
||||
"venues": [
|
||||
{
|
||||
"venue": "NeurIPS",
|
||||
"expected_years": 40,
|
||||
"observed_expected_years": 39,
|
||||
"missing_years": [
|
||||
2026
|
||||
],
|
||||
"missing_count": 1,
|
||||
"canonical_matches": [
|
||||
"NeurIPS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ICML",
|
||||
"expected_years": 43,
|
||||
"observed_expected_years": 14,
|
||||
"missing_years": [
|
||||
1980,
|
||||
1983,
|
||||
1985,
|
||||
1987,
|
||||
1988,
|
||||
1989,
|
||||
1990,
|
||||
1991,
|
||||
1992,
|
||||
1993,
|
||||
1994,
|
||||
1995,
|
||||
1996,
|
||||
1997,
|
||||
1998,
|
||||
1999,
|
||||
2000,
|
||||
2001,
|
||||
2002,
|
||||
2003,
|
||||
2004,
|
||||
2005,
|
||||
2006,
|
||||
2007,
|
||||
2008,
|
||||
2009,
|
||||
2010,
|
||||
2011,
|
||||
2012
|
||||
],
|
||||
"missing_count": 29,
|
||||
"canonical_matches": [
|
||||
"ICML"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ICLR",
|
||||
"expected_years": 14,
|
||||
"observed_expected_years": 11,
|
||||
"missing_years": [
|
||||
2014,
|
||||
2015,
|
||||
2016
|
||||
],
|
||||
"missing_count": 3,
|
||||
"canonical_matches": [
|
||||
"ICLR"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "COLM",
|
||||
"expected_years": 3,
|
||||
"observed_expected_years": 3,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"COLM"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "COLT",
|
||||
"expected_years": 39,
|
||||
"observed_expected_years": 39,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"COLT"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "AISTATS",
|
||||
"expected_years": 29,
|
||||
"observed_expected_years": 25,
|
||||
"missing_years": [
|
||||
1985,
|
||||
1989,
|
||||
1991,
|
||||
1993
|
||||
],
|
||||
"missing_count": 4,
|
||||
"canonical_matches": [
|
||||
"AISTATS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "UAI",
|
||||
"expected_years": 42,
|
||||
"observed_expected_years": 42,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"UAI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "CLeaR",
|
||||
"expected_years": 5,
|
||||
"observed_expected_years": 5,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"CLEAR"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ACL",
|
||||
"expected_years": 64,
|
||||
"observed_expected_years": 48,
|
||||
"missing_years": [
|
||||
1963,
|
||||
1964,
|
||||
1965,
|
||||
1966,
|
||||
1967,
|
||||
1968,
|
||||
1969,
|
||||
1970,
|
||||
1971,
|
||||
1972,
|
||||
1973,
|
||||
1974,
|
||||
1975,
|
||||
1976,
|
||||
1977,
|
||||
1978
|
||||
],
|
||||
"missing_count": 16,
|
||||
"canonical_matches": [
|
||||
"ACL",
|
||||
"ACL+COLING",
|
||||
"ACL+EACL"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "EMNLP",
|
||||
"expected_years": 31,
|
||||
"observed_expected_years": 31,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"CoNLL+EMNLP",
|
||||
"EMNLP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "NAACL",
|
||||
"expected_years": 18,
|
||||
"observed_expected_years": 18,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"NAACL"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "EACL",
|
||||
"expected_years": 19,
|
||||
"observed_expected_years": 19,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"ACL+EACL",
|
||||
"EACL"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "COLING",
|
||||
"expected_years": 31,
|
||||
"observed_expected_years": 28,
|
||||
"missing_years": [
|
||||
1971,
|
||||
1976,
|
||||
1978
|
||||
],
|
||||
"missing_count": 3,
|
||||
"canonical_matches": [
|
||||
"ACL+COLING",
|
||||
"COLING",
|
||||
"COLING+LREC"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "CoNLL",
|
||||
"expected_years": 30,
|
||||
"observed_expected_years": 30,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"CoNLL",
|
||||
"CoNLL+EMNLP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "CVPR",
|
||||
"expected_years": 39,
|
||||
"observed_expected_years": 14,
|
||||
"missing_years": [
|
||||
1983,
|
||||
1985,
|
||||
1986,
|
||||
1988,
|
||||
1989,
|
||||
1991,
|
||||
1992,
|
||||
1993,
|
||||
1994,
|
||||
1996,
|
||||
1997,
|
||||
1998,
|
||||
1999,
|
||||
2000,
|
||||
2001,
|
||||
2003,
|
||||
2004,
|
||||
2005,
|
||||
2006,
|
||||
2007,
|
||||
2008,
|
||||
2009,
|
||||
2010,
|
||||
2011,
|
||||
2012
|
||||
],
|
||||
"missing_count": 25,
|
||||
"canonical_matches": [
|
||||
"CVPR"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ICCV",
|
||||
"expected_years": 20,
|
||||
"observed_expected_years": 7,
|
||||
"missing_years": [
|
||||
1987,
|
||||
1988,
|
||||
1990,
|
||||
1993,
|
||||
1995,
|
||||
1998,
|
||||
1999,
|
||||
2001,
|
||||
2003,
|
||||
2005,
|
||||
2007,
|
||||
2009,
|
||||
2011
|
||||
],
|
||||
"missing_count": 13,
|
||||
"canonical_matches": [
|
||||
"ICCV"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ECCV",
|
||||
"expected_years": 19,
|
||||
"observed_expected_years": 4,
|
||||
"missing_years": [
|
||||
1990,
|
||||
1992,
|
||||
1994,
|
||||
1996,
|
||||
1998,
|
||||
2000,
|
||||
2002,
|
||||
2004,
|
||||
2006,
|
||||
2008,
|
||||
2010,
|
||||
2012,
|
||||
2014,
|
||||
2016,
|
||||
2026
|
||||
],
|
||||
"missing_count": 15,
|
||||
"canonical_matches": [
|
||||
"ECCV"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "AAAI",
|
||||
"expected_years": 40,
|
||||
"observed_expected_years": 40,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"AAAI",
|
||||
"AAAI+IAAI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "IJCAI",
|
||||
"expected_years": 35,
|
||||
"observed_expected_years": 35,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"ECAI+IJCAI",
|
||||
"IJCAI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ECAI",
|
||||
"expected_years": 29,
|
||||
"observed_expected_years": 27,
|
||||
"missing_years": [
|
||||
1974,
|
||||
1980
|
||||
],
|
||||
"missing_count": 2,
|
||||
"canonical_matches": [
|
||||
"ECAI",
|
||||
"ECAI+IJCAI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "KDD",
|
||||
"expected_years": 32,
|
||||
"observed_expected_years": 32,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"KDD"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "SIGIR",
|
||||
"expected_years": 50,
|
||||
"observed_expected_years": 50,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"SIGIR"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "WWW",
|
||||
"expected_years": 33,
|
||||
"observed_expected_years": 27,
|
||||
"missing_years": [
|
||||
1994,
|
||||
1996,
|
||||
1997,
|
||||
1998,
|
||||
1999,
|
||||
2000
|
||||
],
|
||||
"missing_count": 6,
|
||||
"canonical_matches": [
|
||||
"WWW"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "WSDM",
|
||||
"expected_years": 19,
|
||||
"observed_expected_years": 19,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"WSDM"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "RecSys",
|
||||
"expected_years": 20,
|
||||
"observed_expected_years": 19,
|
||||
"missing_years": [
|
||||
2026
|
||||
],
|
||||
"missing_count": 1,
|
||||
"canonical_matches": [
|
||||
"RecSys"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "CIKM",
|
||||
"expected_years": 35,
|
||||
"observed_expected_years": 34,
|
||||
"missing_years": [
|
||||
2026
|
||||
],
|
||||
"missing_count": 1,
|
||||
"canonical_matches": [
|
||||
"CIKM"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ICDM",
|
||||
"expected_years": 26,
|
||||
"observed_expected_years": 25,
|
||||
"missing_years": [
|
||||
2026
|
||||
],
|
||||
"missing_count": 1,
|
||||
"canonical_matches": [
|
||||
"ICDM"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "SDM",
|
||||
"expected_years": 26,
|
||||
"observed_expected_years": 25,
|
||||
"missing_years": [
|
||||
2026
|
||||
],
|
||||
"missing_count": 1,
|
||||
"canonical_matches": [
|
||||
"SDM"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ECML-PKDD",
|
||||
"expected_years": 38,
|
||||
"observed_expected_years": 33,
|
||||
"missing_years": [
|
||||
1986,
|
||||
1987,
|
||||
1988,
|
||||
1989,
|
||||
1991
|
||||
],
|
||||
"missing_count": 5,
|
||||
"canonical_matches": [
|
||||
"ECML",
|
||||
"ECML-PKDD",
|
||||
"PKDD"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ICRA",
|
||||
"expected_years": 43,
|
||||
"observed_expected_years": 43,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"ICRA"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "IROS",
|
||||
"expected_years": 39,
|
||||
"observed_expected_years": 35,
|
||||
"missing_years": [
|
||||
1988,
|
||||
1989,
|
||||
1990,
|
||||
2026
|
||||
],
|
||||
"missing_count": 4,
|
||||
"canonical_matches": [
|
||||
"IROS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "RSS",
|
||||
"expected_years": 22,
|
||||
"observed_expected_years": 20,
|
||||
"missing_years": [
|
||||
2025,
|
||||
2026
|
||||
],
|
||||
"missing_count": 2,
|
||||
"canonical_matches": [
|
||||
"RSS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "CoRL",
|
||||
"expected_years": 10,
|
||||
"observed_expected_years": 9,
|
||||
"missing_years": [
|
||||
2026
|
||||
],
|
||||
"missing_count": 1,
|
||||
"canonical_matches": [
|
||||
"CORL"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ICAPS",
|
||||
"expected_years": 37,
|
||||
"observed_expected_years": 28,
|
||||
"missing_years": [
|
||||
1990,
|
||||
1991,
|
||||
1992,
|
||||
1993,
|
||||
1995,
|
||||
1997,
|
||||
1999,
|
||||
2001,
|
||||
2026
|
||||
],
|
||||
"missing_count": 9,
|
||||
"canonical_matches": [
|
||||
"AIPS",
|
||||
"ICAPS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "KR",
|
||||
"expected_years": 23,
|
||||
"observed_expected_years": 23,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"KR"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "AAMAS",
|
||||
"expected_years": 25,
|
||||
"observed_expected_years": 22,
|
||||
"missing_years": [
|
||||
2012,
|
||||
2025,
|
||||
2026
|
||||
],
|
||||
"missing_count": 3,
|
||||
"canonical_matches": [
|
||||
"AAMAS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ICASSP",
|
||||
"expected_years": 51,
|
||||
"observed_expected_years": 49,
|
||||
"missing_years": [
|
||||
1977,
|
||||
2026
|
||||
],
|
||||
"missing_count": 2,
|
||||
"canonical_matches": [
|
||||
"ICASSP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "INTERSPEECH",
|
||||
"expected_years": 39,
|
||||
"observed_expected_years": 39,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"ECST",
|
||||
"EUROSPEECH",
|
||||
"ICSLP",
|
||||
"INTERSPEECH"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ISMIR",
|
||||
"expected_years": 27,
|
||||
"observed_expected_years": 26,
|
||||
"missing_years": [
|
||||
2026
|
||||
],
|
||||
"missing_count": 1,
|
||||
"canonical_matches": [
|
||||
"ISMIR"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "MICCAI",
|
||||
"expected_years": 29,
|
||||
"observed_expected_years": 29,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"MICCAI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "FAccT",
|
||||
"expected_years": 9,
|
||||
"observed_expected_years": 9,
|
||||
"missing_years": [],
|
||||
"missing_count": 0,
|
||||
"canonical_matches": [
|
||||
"FAccT"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "MLSys",
|
||||
"expected_years": 9,
|
||||
"observed_expected_years": 8,
|
||||
"missing_years": [
|
||||
2018
|
||||
],
|
||||
"missing_count": 1,
|
||||
"canonical_matches": [
|
||||
"MLSys"
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "MM",
|
||||
"expected_years": 34,
|
||||
"observed_expected_years": 33,
|
||||
"missing_years": [
|
||||
2026
|
||||
],
|
||||
"missing_count": 1,
|
||||
"canonical_matches": [
|
||||
"MM"
|
||||
]
|
||||
}
|
||||
],
|
||||
"expected_venue_years": 1266,
|
||||
"missing_venue_years": 150,
|
||||
"venues_with_missing_years": 25,
|
||||
"known_nonpublication_exceptions": [
|
||||
{
|
||||
"venue": "NeurIPS",
|
||||
"year": 2026,
|
||||
"reason": "announced;not in proceedings index at census"
|
||||
},
|
||||
{
|
||||
"venue": "CoRL",
|
||||
"year": 2026,
|
||||
"reason": "not in publisher proceedings index at census"
|
||||
},
|
||||
{
|
||||
"venue": "MLSys",
|
||||
"year": 2018,
|
||||
"reason": "first non-archival SysML meeting; not a missing formal publication volume"
|
||||
}
|
||||
],
|
||||
"historical_missing_years_excluding_nonarchival_2018": 135,
|
||||
"historical_venues_with_missing_years": 16,
|
||||
"historical_gaps": [
|
||||
{
|
||||
"venue": "ICML",
|
||||
"years": [
|
||||
1980,
|
||||
1983,
|
||||
1985,
|
||||
1987,
|
||||
1988,
|
||||
1989,
|
||||
1990,
|
||||
1991,
|
||||
1992,
|
||||
1993,
|
||||
1994,
|
||||
1995,
|
||||
1996,
|
||||
1997,
|
||||
1998,
|
||||
1999,
|
||||
2000,
|
||||
2001,
|
||||
2002,
|
||||
2003,
|
||||
2004,
|
||||
2005,
|
||||
2006,
|
||||
2007,
|
||||
2008,
|
||||
2009,
|
||||
2010,
|
||||
2011,
|
||||
2012
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ICLR",
|
||||
"years": [
|
||||
2014,
|
||||
2015,
|
||||
2016
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "AISTATS",
|
||||
"years": [
|
||||
1985,
|
||||
1989,
|
||||
1991,
|
||||
1993
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ACL",
|
||||
"years": [
|
||||
1963,
|
||||
1964,
|
||||
1965,
|
||||
1966,
|
||||
1967,
|
||||
1968,
|
||||
1969,
|
||||
1970,
|
||||
1971,
|
||||
1972,
|
||||
1973,
|
||||
1974,
|
||||
1975,
|
||||
1976,
|
||||
1977,
|
||||
1978
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "COLING",
|
||||
"years": [
|
||||
1971,
|
||||
1976,
|
||||
1978
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "CVPR",
|
||||
"years": [
|
||||
1983,
|
||||
1985,
|
||||
1986,
|
||||
1988,
|
||||
1989,
|
||||
1991,
|
||||
1992,
|
||||
1993,
|
||||
1994,
|
||||
1996,
|
||||
1997,
|
||||
1998,
|
||||
1999,
|
||||
2000,
|
||||
2001,
|
||||
2003,
|
||||
2004,
|
||||
2005,
|
||||
2006,
|
||||
2007,
|
||||
2008,
|
||||
2009,
|
||||
2010,
|
||||
2011,
|
||||
2012
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ICCV",
|
||||
"years": [
|
||||
1987,
|
||||
1988,
|
||||
1990,
|
||||
1993,
|
||||
1995,
|
||||
1998,
|
||||
1999,
|
||||
2001,
|
||||
2003,
|
||||
2005,
|
||||
2007,
|
||||
2009,
|
||||
2011
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ECCV",
|
||||
"years": [
|
||||
1990,
|
||||
1992,
|
||||
1994,
|
||||
1996,
|
||||
1998,
|
||||
2000,
|
||||
2002,
|
||||
2004,
|
||||
2006,
|
||||
2008,
|
||||
2010,
|
||||
2012,
|
||||
2014,
|
||||
2016
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ECAI",
|
||||
"years": [
|
||||
1974,
|
||||
1980
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "WWW",
|
||||
"years": [
|
||||
1994,
|
||||
1996,
|
||||
1997,
|
||||
1998,
|
||||
1999,
|
||||
2000
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ECML-PKDD",
|
||||
"years": [
|
||||
1986,
|
||||
1987,
|
||||
1988,
|
||||
1989,
|
||||
1991
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "IROS",
|
||||
"years": [
|
||||
1988,
|
||||
1989,
|
||||
1990
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "RSS",
|
||||
"years": [
|
||||
2025
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ICAPS",
|
||||
"years": [
|
||||
1990,
|
||||
1991,
|
||||
1992,
|
||||
1993,
|
||||
1995,
|
||||
1997,
|
||||
1999,
|
||||
2001
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "AAMAS",
|
||||
"years": [
|
||||
2012,
|
||||
2025
|
||||
]
|
||||
},
|
||||
{
|
||||
"venue": "ICASSP",
|
||||
"years": [
|
||||
1977
|
||||
]
|
||||
}
|
||||
],
|
||||
"current_2026_gaps": 14
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
# 历史年度缺口的当前核对
|
||||
|
||||
生产只读快照:2026-11-08 16:34:40.741334 UTC(北京时间11月9日00:34),foundry_data/research。对比已核实的43份首轮官方会议年度清单,使用当前有投稿记录的edition及真实venue_aliases,不沿用旧固定快照的missing_years。
|
||||
|
||||
- 清单共1266个会议—年份位置,当前有150个位置尚无投稿记录,涉及25个目标会议。
|
||||
- 其中2025年及以前136个;排除已明确非存档的SysML2018首届,剩**135个历史年度待补采或核实,涉及16个会议**。
|
||||
- 另有14个2026年度待核实。已知NeurIPS2026和CoRL2026在首轮核查时尚未进入论文集索引,不能一律当已公开论文遗漏;其他年度也需逐个核实发布时间。
|
||||
|
||||
这些是“整年没有任何投稿记录”的数量,不是缺论文篇数,也不是完整性百分比。一个年份只有少数记录、只有公开拒稿或缺了某个track,仍会被计作“已有年度”;这些更细缺口及公开拒稿的逐年可获得性尚须继续核验。会议别名共享的联合届次不能当成两批独立论文,未见录用也不能推断拒稿。
|
||||
|
||||
| 会议 | 历史缺口年份数 |
|
||||
| --- | ---: |
|
||||
| ICML | 29 |
|
||||
| CVPR | 25 |
|
||||
| ACL | 16 |
|
||||
| ECCV | 14 |
|
||||
| ICCV | 13 |
|
||||
| ICAPS | 8 |
|
||||
| WWW | 6 |
|
||||
| ECML-PKDD | 5 |
|
||||
| AISTATS | 4 |
|
||||
| ICLR、COLING、IROS | 各3 |
|
||||
| ECAI、AAMAS | 各2 |
|
||||
| RSS、ICASSP | 各1 |
|
||||
|
||||
[完整逐年结果](historical-gaps-2026-11-09.json) 保留43目标的官方预期、现有年份数、缺失年份与规范会议匹配。原始只读生产导出保存在 `/data/paper-index/collected/historical-gap-current-20261109-v1.json`。 导出SHA256 `580cdea6dd9869adafe9bc28fb75cf3854bb290ad65aa5a600d9bc368a528b93`。年度对照清单为 `docs/coverage/inventory-2026-11-08.json`;候选来源文件绑定的输入哈希与本目录缺口JSON逐字节一致。来源清单的年份和哈希沿用相应 `*-editions-2026-11-08.json`,不是本轮重新访问所有会议网站;2026发布状态需要后续更新。
|
||||
|
||||
## 候选来源与下一步
|
||||
|
||||
另一次只读目录核查在现有、已校验的DBLP快照中,为135个历史年度中的**85个**匹配到初步候选论文集,另**50个**尚未按本轮规则匹配到,需要继续识别目录键或寻找官方来源。未匹配不等于会议没举办、未出版或没有公开来源。
|
||||
|
||||
[候选卷清单](historical-source-candidates-2026-11-09.json) 保留卷key、标题、出版年份、原始bibliographic counts和出版者链接。初筛使用目标family映射内的四位卷key年份或卷标题唯一显式年份,再按具体会议身份复核;两位键、出版年份与会议年份差异、前身系列及其他目录命名可能漏匹配。原始计数可能含workshop、前言、索引或多个卷,**不能当可直接导入的录用论文数量**。
|
||||
|
||||
后续[来源范围复核](historical-source-scope-2026-11-09.md)排除了4个误归ACL的COLING卷,因此候选年度由89修正为85、未匹配年度由46增至50。原卷及排除理由保留在JSON中;135个历史年度缺口不因此改变。ACL1963–1978与COLING1971/1976/1978在当前Anthology目录未列出,继续记为来源待核实,不能一律称作已公开但漏采的论文集。
|
||||
|
||||
优先核查已有候选卷较集中的CVPR22年、ICCV12年、ECCV14年和ICML25年。其中ICML早期workshop名称可能属于官方主系列前身,也可能是附属workshop,须对照官方届次证据逐卷区分,不能靠标题包含workshop一刀切。所有候选须经过范围、稳定标识、重复/冲突、原始字段和覆盖预检;这一步没有导入候选卷。
|
||||
|
||||
## 复核方法
|
||||
|
||||
只读获取当前有记录的edition,不连接papers大文本或标签表:
|
||||
|
||||
```sql
|
||||
WITH counts AS (
|
||||
SELECT edition_id,count(*) AS submissions
|
||||
FROM submissions GROUP BY edition_id
|
||||
)
|
||||
SELECT e.id,e.venue_id,v.acronym,e.year,e.track,e.event_key,counts.submissions
|
||||
FROM editions e JOIN venues v ON v.id=e.venue_id
|
||||
JOIN counts ON counts.edition_id=e.id ORDER BY e.id;
|
||||
|
||||
SELECT venue_id,alias FROM venue_aliases ORDER BY venue_id,alias;
|
||||
```
|
||||
|
||||
逐目标取规范会议名或别名相等的edition年份集合,与该目标edition_census.expected_years做集合差;再把2026与明确非存档的SysML2018单列。保留原始导出与校验哈希,不把每个matched year误当逐篇/逐track完整。
|
||||
|
||||
与本清单同一阶段的WWW补采只补1995年现有poster轨道,不新增缺失年度,所以即使其15条增量已验收,135这个年度计数也不会随之下降。早期1994年的50条正式展示候选、307条混合节目候选共357条仍需核实;其中并非每条都是论文。
|
||||
|
||||
本清单用于明确第14步剩余范围,不代表第14步完成,也不替代所有导入结束后的第16步全库去重、摘要、约束和实际查询验收。
|
||||
@@ -0,0 +1,33 @@
|
||||
# ICML 与 IROS 早期主系列的来源范围
|
||||
|
||||
本轮修正采集阶段的届次识别,不修改任何现有生产投稿轨道或历史。正式主系列曾以Workshop命名,不等于附属workshop;也不能反过来把所有旧workshop升为主会。
|
||||
|
||||
## 已核实的精确范围
|
||||
|
||||
| 会议 | 论文集键 | 依据与行为 |
|
||||
| --- | --- | --- |
|
||||
| ICML | `conf/icml/1988`、`1989`、`1990`、`1991`、`1992` | [官方历史](https://icml.cc/Conferences/2008/past_icmls.shtml.html)将其列为第5–9届,年份与举办地对应固定目录中的完整标题。1989/1991/1992的Workshop名称不再导致主系列被默认排除;1988/1990保留main并增加来源范围证据 |
|
||||
| IROS | `conf/iros/1988`、`1989`、`1990` | [IEEE RAS官方历届目录](https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/)列出三届与论文集链接,地点及日期对应固定目录标题;这些是IROS主系列 |
|
||||
|
||||
规则同时要求family、booktitle基础别名(ML或IROS)、空轨道后缀、完整精确论文集键、匹配的事件年份及固定完整标题。仍执行原有书目年份核查:与事件年不同且标题无明确事件年时拒绝推断。ML91原始标题没有四位年份,使用精确1991卷键、完整标题、逐篇书目年和官方第8届记录,不从两位标题代码猜世纪。
|
||||
|
||||
`1991w`/`1988w`等卷键、booktitle中的Workshops后缀、不同届次/地点/完整标题、未列入的年份均不会获得上述主系列豁免。AISTATS/COLT既有精确规则保留;真正的附属workshop仍受默认不采集workshop的边界约束。
|
||||
|
||||
## 原始字段、决定与后续导入
|
||||
|
||||
采集器仍保存原始proceedings标题(包括Workshop)、书目年、原始作者名、完整来源书目字段及稳定标识;新增 `track_scope_note` 与官方 `track_scope_evidence_url`。接受状态仍只源自该论文的inproceedings书目证据;不从历史目录生成占位论文或拒稿,也不重新导入已经存在的记录。
|
||||
|
||||
新规则的输出必须重新做原始来源/标题/作者/稳定标识与现有库冲突预检,再按增量导入。此前被隔离的ICML1988/1990记录不能仅凭代码规则而直接解除全部冲突。当前正在运行的历史批次使用原固定版本和文件,不能在运行中替换其源码或输入。
|
||||
|
||||
## 验证与证据
|
||||
|
||||
新增回归覆盖:ICML三届Workshop名称、IROS最初三届、ICML原Conference名称的两届、错误卷键/后缀/标题/未知年份,以及归一化记录经过原生文件导入后保留轨道证据。构建与测试仅在已授权的ces-control远程runner执行,本机仅做语法和文本/JSON一致性检查;结果以相应commit的CI回执为准。
|
||||
|
||||
官方HTML在工作证据目录 `paper-index/historical-primary-series-evidence-v1/`,附响应状态、字节数、时间与SHA256;仅取元数据网页,没有下载论文全文。
|
||||
|
||||
| 来源 | UTC采样时间 | HTTP | 字节数 | SHA256 |
|
||||
| --- | --- | ---: | ---: | --- |
|
||||
| [icml-history](https://icml.cc/Conferences/2008/past_icmls.shtml.html) | 2026-11-08T18:36:43.043135+00:00 | 200 | 11993 | `80cce886fe985551339a52b13c4077acf827e8fcc01555f854dc24077bdebdf0` |
|
||||
| [iros-history](https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/) | 2026-11-08T18:36:45.687171+00:00 | 200 | 466059 | `fdb4594621e68567e1099f5c88aae53de835b7b8146025148c3b1968df990697` |
|
||||
|
||||
本次不关闭历史年度缺口,也不替代全部导入后的全库去重、来源/年度/track覆盖、摘要和实际查询验收。
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,32 @@
|
||||
# 早期 NLP 来源范围复核
|
||||
|
||||
核查时间:2026-11-08 18:27 UTC(北京时间11月9日)。本轮只修正来源候选和记录可获得性,没有操作生产数据库。
|
||||
|
||||
## 跨会议候选修正
|
||||
|
||||
先前候选清单将 `conf/coling/1965`、`1967`、`1969`、`1973` 匹配到 ACL 年会缺口。这四卷标题明确属于 COLING;[COLING 官方 Anthology 目录](https://aclanthology.org/venues/coling/)将它们列在 COLING 下。出版者写作 ACL 或目标映射包含 COLING,不足以证明某一年是 ACL 联办届次。
|
||||
|
||||
从 ACL 候选中排除上述四卷,原始卷字段和排除理由保留在[候选JSON](historical-source-candidates-2026-11-09.json)的 `excluded_cross_venue_candidates` 中。135个历史会议—年份位置仍未关闭;有初步候选来源的年度由89修正为85,尚未匹配的由46增至50。这些数字是目录初筛结果,不是已验证可导入的论文数。
|
||||
|
||||
联合届次仍按具体年度的明确会议身份处理,例如有官方联办证据的卷可以属于多个会议。不能一律排除跨family卷,也不能把联合卷重复导入为两批论文。
|
||||
|
||||
## 来源待核实的早期年份
|
||||
|
||||
| 范围 | 本轮官方来源观察 | 处理 |
|
||||
| --- | --- | --- |
|
||||
| ACL 1963–1978,共16年 | [ACL 官方 Anthology 目录](https://aclanthology.org/venues/acl/)目前从1979年列起 | 保留历史举办清单;逐年核实是否有可索引发表材料,不能据会议举办就断言漏采正式论文集,也不据目录缺失断言未出版 |
|
||||
| COLING 1971、1976、1978 | [主办委员会历史](https://ufal.mff.cuni.cz/iccl-old)列出这三届;当前Anthology COLING目录未列出 | 来源缺口继续开放,需要主办方或出版者逐篇索引;不生成占位论文或录用/拒稿记录 |
|
||||
|
||||
这19个位置不能用其他会议的卷补齐,也不能因本轮未找到目录而删除。公开拒稿可获得性是独立核查,不从论文集或录用清单缺失推断。
|
||||
|
||||
## 证据与复核
|
||||
|
||||
三份原始HTML及 `receipts.json` 保存在工作证据目录 `paper-index/nlp-history-source-review-v1/`;结构化响应URL、时间、状态、字节数和完整SHA256收录于候选JSON的 `scope_review.evidence`。三次响应均为HTTP200;未获取论文全文。
|
||||
|
||||
| 官方来源 | 响应SHA256 |
|
||||
| --- | --- |
|
||||
| [acl-venue](https://aclanthology.org/venues/acl/) | `d04c188c0769e065f18ca9c0246f70086e2f7027ce3efe5ea0f2cdc641796b72` |
|
||||
| [coling-venue](https://aclanthology.org/venues/coling/) | `8e7ec0a2915bfbccaa26b7a7558a9cf977c4aadc119794516ecebb826f61623f` |
|
||||
| [iccl-history](https://ufal.mff.cuni.cz/iccl-old) | `c5286dc51236c90685d22543ff1f7b209adcb3eedb82a7b1ac2115c8dd0169fc` |
|
||||
|
||||
复核时先校验保存的原始响应哈希,再读取目录中的会议身份和年度。实时网页若已变化,应另存新响应和时间,不能把新内容套用旧哈希。本次不改最初会议年度清单或生产年度快照,不代表第14步覆盖验收或第16步最终全库审计完成。
|
||||
@@ -0,0 +1,44 @@
|
||||
# ICLR 2014–2016 官方历史录用档案
|
||||
|
||||
此适配器是长期维护工具的一部分,按需运行。只识别已经独立核对的三种页面布局;后续页面发生变化必须重新核查,不能放松计数或轨道边界以使采集成功。
|
||||
|
||||
| 官方页面 | 页面链接次数 | 整理后的不同记录 |
|
||||
|---|---:|---:|
|
||||
| [2014 Conference Proceedings](https://iclr.cc/archive/2014/conference-proceedings/) | 35 | main 35 |
|
||||
| [2015 accepted-main](https://iclr.cc/archive/www/doku.php%3Fid=iclr2015:accepted-main.html) | 116 | main 31、workshop 74 |
|
||||
| [2016 accepted-main](https://iclr.cc/archive/www/doku.php%3Fid=iclr2016:accepted-main.html) | 80 | main 80 |
|
||||
|
||||
2015 的 11 篇 oral 全部再次出现在 poster 段落中。相同年份、轨道及 arXiv 标识的重复展示只生成一条来源记录,但每次展示的标题、作者显示字符串、链接、版本和段落均保留在 `source_variants`。两处标题或解析作者冲突时停止采集、等待人工核查。不同年份或不同轨道不合并投稿。
|
||||
|
||||
2014 的作者以分号列出,一条有末尾分号,一条标题和作者斜体块不在普通段落中,部分作者段落拆成多个格式片段;必须读取完整作者段落或独立斜体块,不能只读首个格式片段。2016 的代码链接不属于作者。保留原文,解析结果不替代来源证据。oral、poster 作为 `iclr_archive:presentation` 标签,不改成独立投稿轨道。arXiv 版本留在来源变体中,稳定标识使用去版本号的编号。
|
||||
|
||||
## 来源快照与运行
|
||||
|
||||
2026-11-09 已冻结的官方响应 SHA256:
|
||||
|
||||
- 2014:`a33858176fbffae0066ed3c10310f7dd98a66d12a36334b6339ef1f915ef07af`
|
||||
- 2015:`131491663742cb084662a4804009b8318e0fa7bc184d4a9044d16dd28f05b888`
|
||||
- 2016:`7320b8652431793b6ea0743f871ee446bf08b6361ccd4e421b6b53449e71f4f6`
|
||||
|
||||
在服务器运行,输出新目录,不下载论文 PDF:
|
||||
|
||||
```sh
|
||||
paper-atlas collect iclr-archive --year 2015 \
|
||||
--snapshots /data/paper-index/collected/historical-next-source-evidence-v1 \
|
||||
--out /data/paper-index/collected/iclr-archive-source-v1/2015
|
||||
|
||||
# 或重新下载官方 HTML,并保留 HTTP 缓存与收据
|
||||
paper-atlas collect iclr-archive --year 2016 \
|
||||
--cache /data/paper-index/cache/iclr-archive \
|
||||
--out /data/paper-index/collected/iclr-archive-source-v1/2016
|
||||
```
|
||||
|
||||
冻结输入目录必须包含 `receipts.json`,每项有精确 `url`、HTTP `status: 200`、相对 `file`、`bytes`、`sha256` 和带时区的 `finished_at`。读取时重新核对尺寸、哈希和路径;拒绝越界及指向目录外的软链接。已完成输出不可覆盖,应检查报告后另建明确版本目录。解析或计数失败时不生成完成报告,也不导入数据库。
|
||||
|
||||
输出分轨道 JSONL、manifest、原始 HTML 证据和年度采集报告。仅支持这三年已审查的页面。2014 workshop、其他未列轨道、逐篇摘要及公开拒稿范围尚未通过这些页面核验,不能据此声称完整。缺席不代表 rejected、withdrawn 或从未被录用。
|
||||
|
||||
## 导入前后的边界
|
||||
|
||||
这三页合计 220 条不同来源记录,不等于新增 220 篇论文;同一工作可能已存在于 arXiv 或其他会议。先核验来源字段与原始页面,再执行 PostgreSQL 只读预检,比较稳定标识、标题、作者和原始来源。标题/作者差异、版本关联及冲突必须保留证据并明确评估;不能仅靠标题猜测身份,也不能覆盖原有摘要。
|
||||
|
||||
采集器不连接数据库。正式导入使用既有 `import records`,必须有新的生产保护基线和独立验收;全部导入结束后仍需全库去重和历史覆盖审计。来自官方录用清单的 accepted 是清单观察,不是原始审稿决定签名或决定日期。
|
||||
@@ -0,0 +1,201 @@
|
||||
{
|
||||
"format": 1,
|
||||
"source_cache_sha256": "8a65562f3490497cada2919ec77827134beabd5ec9c3690dec3625e2318aa54e",
|
||||
"records": [
|
||||
{
|
||||
"entry_id": "45",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/45/index.html",
|
||||
"individual_sha256": "83bede6afc9a7039fae2dd1f28274a394418b34f5d5632590b913617580bda1a",
|
||||
"title": "WISE - World Wide Information System for R&D; Efforts",
|
||||
"authors": [
|
||||
"Fritz Loseries",
|
||||
"Veronika Samara",
|
||||
"Bodo Urban"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "16",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/16/index.html",
|
||||
"individual_sha256": "cf76da903eaeaa27d19857fdaa13220f6821559bf4ac22844fae10abbaa6f3f3",
|
||||
"title": "The World Wide Web as a Tool for Collaborative Inter-Agency Projects: Creation of an Environmental Metadata Database for the Analysis of the Venice Lagoon Ecosystem.",
|
||||
"authors": [
|
||||
"Mirko Labbri"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "17",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/17/index.html",
|
||||
"individual_sha256": "0578fec9c82e8f2f98c4de0f1bcc44011ebb84c53353fdcc423a69cf70673bc2",
|
||||
"title": "NetRepreneur",
|
||||
"authors": [
|
||||
"Kari-Pekka Turtiainen",
|
||||
"Antti Auer"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "29",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/29/index.html",
|
||||
"individual_sha256": "624ad3f685de1163411e33e90488975d848a63243fd7ffbed0862c6fb58c4140",
|
||||
"title": "Mosaic TueV 2.4.3: Multilingual GUI and Improvements towards User Friendliness",
|
||||
"authors": [
|
||||
"Mike Bretz",
|
||||
"Uwe Koch",
|
||||
"Paul de Bra"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "13",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/13/index.html",
|
||||
"individual_sha256": "02409321a3403a94d8362a9862f4960386ea9c344fb47f4bcdf9eae128e8b46e",
|
||||
"title": "Using a WWW Browser as an Alternative User Interface for Interactive Applications",
|
||||
"authors": [
|
||||
"Frank Lonczewski"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "36",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/36/index.html",
|
||||
"individual_sha256": "ed9dada4ea5903cb8f0a4e75f2958416a6e32647fbf3530e58edd9abf26070ce",
|
||||
"title": "HyperSource: A Hypermedia Program Development And Documentation System",
|
||||
"authors": [
|
||||
"Jan Oliver Borchers"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "57",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/57/index.html",
|
||||
"individual_sha256": "3ad4206ad28a19823172cd2845f636fe7a461c251940e9a93deb8f6484b8a984",
|
||||
"title": "Progressive Image Transmission Capability (PROTRAC) on the World Wide Web",
|
||||
"authors": [
|
||||
"Vojislav Lalich-Petrich",
|
||||
"Gaurav Bhatia",
|
||||
"Larry Davis"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "38",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/38/index.html",
|
||||
"individual_sha256": "940370acc5e6a954b197053fe451e466a904d1f0ac82c8fe4e6ca455faf2c927",
|
||||
"title": "Measuring the Web with Lycos",
|
||||
"authors": [
|
||||
"Michael L. Mauldin"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": "Official abstract listing has no abstract. Canonical HTML has body sections but no explicit abstract; body is not substituted as a summary."
|
||||
},
|
||||
{
|
||||
"entry_id": "05",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/05/index.html",
|
||||
"individual_sha256": "1858be87d6e7f47476e1cb8fb4793c0fce3b8a6ff653bcfd4a35089e0520f177",
|
||||
"title": "Virtual Places - Adding people to the web",
|
||||
"authors": [
|
||||
"Yosi Mass"
|
||||
],
|
||||
"author_source": "author_index",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": "Official abstract listing and canonical HTML have no abstract. Canonical page identifies a contact and links PostScript; no full-text download. Author identity comes from author index, not inferred from contact role."
|
||||
},
|
||||
{
|
||||
"entry_id": "37",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/37/index.html",
|
||||
"individual_sha256": "b700a0a51b7a82a5f88373ae0283bb369d1cf6f455acff7d91cd7313fa8b3adf",
|
||||
"title": "The Internet Public Library: Serving a Global Community",
|
||||
"authors": [
|
||||
"Susan Schweitzer"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "41",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/41/index.html",
|
||||
"individual_sha256": "397b474109f9c1240a77d0f490f2b02dd35fae93febbb7397d1b2a31759a8ec1",
|
||||
"title": "Graphical Browsing in the WorldWideWeb and the MultiMedia Forum",
|
||||
"authors": [
|
||||
"Rainer Pagé",
|
||||
"Klaus Süllow"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "47",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/47/index.html",
|
||||
"individual_sha256": "52eb4bfdd658730a222c98170b72dabf9cbcc5ffd2ada1e900b826baf2b02833",
|
||||
"title": "Interoperating Advanced Hypermedia Systems with WWW",
|
||||
"authors": [
|
||||
"Jian Zhang",
|
||||
"Martin Fruehauf",
|
||||
"Thomas Kirste"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "50",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/50/index.html",
|
||||
"individual_sha256": "8dc8b0561bd8b1b106c5b209a973c695c3cc9df5288450d4a01fe32541476b7e",
|
||||
"title": "Web Enabled Agile Manufacturing",
|
||||
"authors": [
|
||||
"S. V. Iyer",
|
||||
"T. Singh",
|
||||
"U. Ravaioli",
|
||||
"R. E. DeVor",
|
||||
"S. G. Kapoor",
|
||||
"G. Veknatasubramian"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "55",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/55/index.html",
|
||||
"individual_sha256": "21780cd4047f513e8c9aeced5247002e27cf71719f08de85c6531b57a9e5058d",
|
||||
"title": "Weblets: Fundamental Building Blocks for WWW Tools",
|
||||
"authors": [
|
||||
"Thomas J. Watt Jr."
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
},
|
||||
{
|
||||
"entry_id": "59",
|
||||
"individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/59/index.html",
|
||||
"individual_sha256": "fed4cc9dcaee350a12c5e4b6fa9c99be9bead25a81632512d31566400d3d0fad",
|
||||
"title": "Usenet and the World-Wide Web",
|
||||
"authors": [
|
||||
"M. L. Grant"
|
||||
],
|
||||
"author_source": "canonical_byline",
|
||||
"reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
|
||||
"missing_abstract_reason": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
# Early WWW archive collector
|
||||
|
||||
`paper-atlas collect www-archive` is a maintained metadata adapter. It reads official
|
||||
IW3C2 archive HTML, keeps exact source hashes, and separates importable rows from
|
||||
program entries that still need review. It fetches no PDF, follows no paper-body or
|
||||
reference link, and makes no database connection.
|
||||
|
||||
For a fresh source snapshot on the server:
|
||||
|
||||
```sh
|
||||
paper-atlas collect www-archive \
|
||||
--http-cache /data/paper-index/cache/www-archive \
|
||||
--source-cache /data/paper-index/collected/www-archive-source-v2 \
|
||||
--out /data/paper-index/collected/www-archive-normalized-v2
|
||||
```
|
||||
|
||||
For reproducible offline replay of the already verified snapshot:
|
||||
|
||||
```sh
|
||||
paper-atlas collect www-archive \
|
||||
--source-cache /data/paper-index/collected/www-early-archive-source-v1 \
|
||||
--source-cache-sha256 c9388ccef44486e0684c9ee9e8620a465cdc4574e6bc9ea9391d743e01a3312c \
|
||||
--out /data/paper-index/collected/www-archive-normalized-v1
|
||||
```
|
||||
|
||||
Use a new output directory for another batch. Existing snapshots and normalized
|
||||
batches are never overwritten. The manifest hash binds every source URL, byte count,
|
||||
response hash and cache filename. Plain files must remain directly within the cache;
|
||||
symlinks, escaping paths, duplicate URLs, non-200 responses and altered bytes fail.
|
||||
The command uses UTF-8 where valid and explicitly records an ISO-8859-1 fallback for
|
||||
older pages; original bytes remain authoritative for encoding corrections.
|
||||
|
||||
## Evidence and outputs
|
||||
|
||||
- WWW3 Spring 1995 papers use `event_key=www-1995-spring`, `track=main`. The author,
|
||||
session and abstract indexes must have the same nonempty entry-ID set. The ID is
|
||||
local to this exact event and proceedings section, never a DOI or OpenReview ID.
|
||||
- Posters remain `track=poster`. An abstract without a linked author-index entry
|
||||
stays in review even when another index has a placeholder `#` link for it.
|
||||
- Paper URLs must agree between author and session indexes. A malformed whitespace
|
||||
URL in the abstract list may use the exact event/entry-ID match only when both
|
||||
independent indexes corroborate the same clean URL. The original href is retained.
|
||||
- Canonical names come from the explicitly labeled author index. The source lists
|
||||
them alphabetically; this is **not** original byline order. Both the raw index names
|
||||
and the full abstract-page byline/affiliation text remain in metadata. No person ID
|
||||
or cross-source identity is inferred from a name. Conflicting titles or missing
|
||||
abstracts stay in review. Topic keywords are not treated as an abstract.
|
||||
- WWW1 1994 formal presentations and WWW2 1994 mixed categories remain review
|
||||
candidates. The first list contains uncertain business-track annotations and a
|
||||
count discrepancy; the second contains panels, tutorials, products and other
|
||||
activities. Neither presentation listing nor a missing full-text link establishes
|
||||
a regular-paper acceptance or a rejection. Raw category/title/name claims are kept.
|
||||
|
||||
The main and poster JSONL files have independent manifests with explicit event key,
|
||||
track, count and SHA256. `source_listing_count` records the full recognized abstract
|
||||
listing, while `held_entry_count` records excluded entries; neither is a ready-row
|
||||
count. Their status stays `partial`: a verified linked subset does
|
||||
not prove complete event coverage. `review-candidates.jsonl` contains nested
|
||||
candidates and reasons, with no importable decision status. It is excluded from
|
||||
`import-queue.json`. `report.json` accounts for ready/held records and every source
|
||||
used. Newlines delimit JSONL; Unicode U+2028/U+2029 inside metadata are not line breaks.
|
||||
The output is published only after all required source and scope checks pass.
|
||||
|
||||
## Before importing
|
||||
|
||||
This independent collector branch starts from main; it does not include or silently
|
||||
install the event-aware writer. Its output requires the reviewed integrated runtime
|
||||
from PR31 (or a main revision containing that implementation). The production event
|
||||
schema migration has been independently accepted, preserving existing edition1212
|
||||
and its 59 Fall submissions; that is not an import of the Spring proceedings.
|
||||
|
||||
Inspect held entries and original author names, preflight every ready row against
|
||||
current PostgreSQL stable identifiers and candidate identities, and verify the
|
||||
source/year/track/event coverage before running `import records`. When a ready subset
|
||||
excludes held entries, preserve the independently known listing total in the existing
|
||||
incremental-coverage protocol: bind a freshly checked parent count and expected count,
|
||||
and set the full expected count from this manifest. A new source uses previous count
|
||||
0 and previous expected count null; an existing source requires its actual current
|
||||
values. These live parent values are not guessed by the offline collector. Do not automatically
|
||||
run this queue against an older writer. Preserve extended versions and independent
|
||||
submission/decision evidence, and keep unresolved identities held. A normalization
|
||||
report or passing tests alone do not establish production import, complete historical
|
||||
coverage or final global deduplication.
|
||||
|
||||
Initial fixed-cache replay (64ced7a, CI1057): 38 main papers and38 posters ready;
|
||||
372 entries held (50 first-conference fields,307 mixed program entries,15 posters).
|
||||
Independent report44dfd070 verified source hashes,counts and author evidence, finding
|
||||
one existing exact-title candidate. That publication is explicitly linked to
|
||||
arXiv1811.12161 by its WWW1995/journal-reference declaration; arXiv1810.07232 is a
|
||||
different1997 invited chapter and remains separate despite the overlap note. This
|
||||
is a preparation observation, not evidence of completed production import or final
|
||||
global deduplication.
|
||||
|
||||
## 以单篇官方页面核实待处理 poster
|
||||
|
||||
作者索引漏链接、摘要目录使用短标题或标题存在拼写差异时,不放宽全局标题匹配。可以先取得单篇官方 HTML,人工核对 H1、署名区与原目录同一 entry URL,再提供带 SHA256 的逐条核查文件:
|
||||
|
||||
```sh
|
||||
paper-atlas collect www-archive \
|
||||
--source-cache /data/paper-index/collected/www-poster-reviewed-source-v2 \
|
||||
--source-cache-sha256 8a65562f3490497cada2919ec77827134beabd5ec9c3690dec3625e2318aa54e \
|
||||
--reviewed-posters docs/coverage/www-1995-posters-reviewed-2026-11-09.json \
|
||||
--reviewed-posters-sha256 <核查文件的SHA256> \
|
||||
--out <新的输出目录>
|
||||
```
|
||||
|
||||
[1995 年15条poster的核查输入](coverage/www-1995-posters-reviewed-2026-11-09.json) 绑定61页来源快照:原46页和补取的15页单篇HTML。程序验证输入文件及快照哈希、单篇页面哈希、完全一致的 canonical H1、原摘要目录中的同一URL/entry ID及作者证据,不通过相似标题猜身份。规范标题采用单篇H1,摘要目录标题及作者索引标题均在 metadata 中保留。
|
||||
|
||||
`author_source=canonical_byline` 表示名字出现在H1之后、摘要/关键词或下一节标题之前的署名区域,由逐条核查明确选出,不从全文引用或机构猜作者;保留原始缩写、顺序、重音和拼写。只有 `author_source=author_index` 时使用原作者索引顺序并逐项核对;05的单篇页面写的是联系人,作者身份使用原作者索引证据,不由联系人角色推断。核查输入是审阅后的数据声明,程序的字符串/哈希检查不代替人工识别署名语义。
|
||||
|
||||
38和05的来源没有可明确引用的摘要。保留 `abstract=null` 和逐条缺失原因,manifest准确报告 missing_abstract_count,继续标partial;正文、联系方式及PostScript链接不当作摘要,也不下载正文。缺摘要不等于拒稿或未发表。本批输出仍包含原先已验收记录,正式导入前必须只取新增external ID,检查实时覆盖父记录并按incremental模式准备,不能直接重导入全部输出。
|
||||
@@ -13,6 +13,7 @@ COMMANDS = {
|
||||
('download',): ('paper_atlas.io.snapshots', {}, '下载并校验固定版本快照'),
|
||||
('collect','pmlr'): ('paper_atlas.collectors.pmlr', {}, 'PMLR 官方元数据'),
|
||||
('collect','acl'): ('paper_atlas.collectors.acl', {}, 'ACL Anthology XML'),
|
||||
('collect','iclr-archive'): ('paper_atlas.collectors.iclr_archive', {}, '核对ICLR 2014–2016官方历史录用档案'),
|
||||
('collect','iclr-program'): ('paper_atlas.collectors.iclr', {}, 'ICLR 官方公开程序'),
|
||||
('collect','iclr-snapshot'): ('paper_atlas.collectors.openreview', {}, 'ICLR 公开投稿与决定快照'),
|
||||
('collect','public-openreview'): ('paper_atlas.collectors.public_openreview', {}, '核验其他会议的公开拒稿及撤稿标签'),
|
||||
@@ -23,6 +24,8 @@ COMMANDS = {
|
||||
('collect','mlsys'): ('paper_atlas.collectors.proceedings', {'source':'mlsys'}, 'MLSys 官方历年标题与摘要'),
|
||||
('collect','cvf'): ('paper_atlas.collectors.vision', {'source':'cvf'}, 'CVPR / ICCV 官方开放目录'),
|
||||
('collect','ecva'): ('paper_atlas.collectors.vision', {'source':'ecva'}, 'ECCV 官方开放目录'),
|
||||
('collect','aamas-proceedings'): ('paper_atlas.collectors.aamas', {}, '按官方目录区分 AAMAS 发表形式和摘要'),
|
||||
('collect','www-archive'): ('paper_atlas.collectors.www_archive', {}, '核验早期WWW官方档案并分开待核查条目'),
|
||||
('collect','dblp'): ('paper_atlas.collectors.dblp_records', {}, '核对并整理 DBLP 历史发表记录'),
|
||||
('catalogue','dblp'): ('paper_atlas.collectors.dblp', {}, '盘点 DBLP 历史会议目录'),
|
||||
('import','arxiv'): ('paper_atlas.ingestion.arxiv', {}, '导入 arXiv 元数据 ZIP'),
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
"""Official AAMAS proceedings metadata, with publication format separate from track."""
|
||||
import argparse
|
||||
from collections import Counter, defaultdict
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import tempfile
|
||||
from urllib.parse import urljoin, urlsplit
|
||||
|
||||
from paper_atlas.db.submissions import timestamp
|
||||
from paper_atlas.io.html import Document, Node, clean
|
||||
from paper_atlas.io.http import fetch
|
||||
|
||||
SOURCE = 'aamas_proceedings'
|
||||
ROOT = 'https://www.ifaamas.org/Proceedings/aamas2012/'
|
||||
# Counts independently checked against the frozen publisher sections. This
|
||||
# adapter deliberately refuses other editions/layouts until they are reviewed.
|
||||
SECTIONS = {
|
||||
'fullpapers': ('Full Papers', 'full_paper', 'proceedings_unspecified', 136),
|
||||
'extabstracts': ('Extended Abstracts', 'extended_abstract', 'proceedings_unspecified', 146),
|
||||
'demos': ('Demos', 'demo', 'demo', 28),
|
||||
'doctorals': ('Doctorial Consortium Abstracts', 'doctoral_abstract', 'doctoral', 24),
|
||||
}
|
||||
|
||||
|
||||
def classes(node):
|
||||
return set(node.attrs.get('class', '').split())
|
||||
|
||||
|
||||
def text_without(node, omitted):
|
||||
if node.tag in ('script', 'style') or classes(node) & omitted:
|
||||
return ''
|
||||
parts = [text_without(child, omitted) if isinstance(child, Node) else child
|
||||
for child in node.children]
|
||||
value = ''.join(parts)
|
||||
return ' '+value+' ' if node.tag in ('p', 'div', 'section', 'li', 'br') else value
|
||||
|
||||
|
||||
def one(nodes, description):
|
||||
if len(nodes) != 1:
|
||||
raise ValueError('Expected one '+description)
|
||||
return nodes[0]
|
||||
|
||||
|
||||
def parse_section(html, section, observed_at, evidence):
|
||||
heading, publication_format, track, _ = SECTIONS[section]
|
||||
document = Document(html)
|
||||
titles = [clean(n.text()) for n in document.root.walk() if n.tag == 'title']
|
||||
if titles != [heading+' - AAMAS 2012']:
|
||||
raise ValueError('Unexpected proceedings edition or section title')
|
||||
headings = [clean(n.text()) for n in document.root.walk() if n.tag == 'h1']
|
||||
if headings != [heading]:
|
||||
raise ValueError('Unexpected proceedings section heading')
|
||||
url = ROOT+'resources/'+section+'.html'
|
||||
records = []
|
||||
seen = set()
|
||||
session = None
|
||||
for node in document.root.walk():
|
||||
if node.tag == 'h2' and 'section' in classes(node):
|
||||
session = {'id': node.attrs.get('id'), 'title': clean(node.text())}
|
||||
if node.tag != 'div' or 'paper' not in classes(node):
|
||||
continue
|
||||
paper_id = node.attrs.get('id', '')
|
||||
if not re.fullmatch(r'[A-Za-z0-9]+_[0-9]+', paper_id) or paper_id in seen:
|
||||
raise ValueError('Missing or repeated publisher paper ID')
|
||||
seen.add(paper_id)
|
||||
title_node = one([n for n in node.walk() if n.tag == 'span' and 'title' in classes(n)], 'paper title')
|
||||
link = one([n for n in title_node.children if isinstance(n, Node) and n.tag == 'a'], 'paper link')
|
||||
paper_url = urljoin(url, link.attrs.get('href', ''))
|
||||
if paper_url != ROOT+'papers/'+paper_id+'.pdf':
|
||||
raise ValueError('Paper link differs from publisher ID or official host')
|
||||
title = clean(text_without(link, {'abstract'}))
|
||||
abstract_nodes = [n for n in link.walk() if n.tag == 'span' and 'abstract' in classes(n)]
|
||||
if len(abstract_nodes) > 1:
|
||||
raise ValueError('Multiple abstracts for one paper')
|
||||
abstract = (clean(abstract_nodes[0].text()) or None) if abstract_nodes else None
|
||||
author_node = one([n for n in node.walk() if n.tag == 'span' and 'authors' in classes(n)], 'author list')
|
||||
author_display = clean(text_without(author_node, {'amount'}))
|
||||
authors = [clean(name) for name in author_display.split(',')]
|
||||
if not title or not author_display or any(not name for name in authors):
|
||||
raise ValueError('Empty paper title or author')
|
||||
tags = [{'namespace': 'aamas_proceedings:publication_format', 'value': publication_format}]
|
||||
if session:
|
||||
tags.append({'namespace': 'aamas_proceedings:session', 'value': session['title']})
|
||||
records.append({
|
||||
'source': SOURCE, 'external_id': '2012/'+paper_id,
|
||||
'url': url+'#'+paper_id, 'title': title, 'abstract': abstract,
|
||||
'authors': authors, 'identifiers': {}, 'venue': 'AAMAS', 'venue_aliases': [],
|
||||
'year': 2012, 'track': track, 'status': 'accepted',
|
||||
'observed_at': observed_at,
|
||||
'raw_decision': 'listed in official AAMAS 2012 proceedings '+heading,
|
||||
'metadata': {'publisher_paper_id': paper_id, 'paper_url': paper_url,
|
||||
'section_url': url, 'section_heading': heading,
|
||||
'publication_format': publication_format, 'session': session,
|
||||
'authors_display': author_display, 'source_evidence': evidence,
|
||||
'publication_evidence': 'official published proceedings entry; not an original decision note',
|
||||
'track_scope_note': ('Full papers and extended abstracts include general and special submission tracks; '
|
||||
'session topics do not establish an individual submission track.'
|
||||
if track == 'proceedings_unspecified' else 'Explicit publisher section.'),
|
||||
'decision_date_precision': 'not_supplied'},
|
||||
'tags': tags})
|
||||
if not records:
|
||||
raise ValueError('No publisher paper entries found')
|
||||
return records
|
||||
|
||||
|
||||
def read_pages(cache=None, snapshots=None):
|
||||
if bool(cache) == bool(snapshots):
|
||||
raise ValueError('Choose exactly one HTTP cache or frozen snapshot directory')
|
||||
urls = {'introduction': ROOT, **{key: ROOT+'resources/'+key+'.html' for key in SECTIONS}}
|
||||
pages = {}
|
||||
receipts = []
|
||||
frozen = json.loads((Path(snapshots)/'receipts.json').read_text())['receipts'] if snapshots else []
|
||||
for key, url in urls.items():
|
||||
if snapshots:
|
||||
receipt = one([r for r in frozen if r['url'] == url], 'frozen response receipt')
|
||||
filename = Path(receipt['file'])
|
||||
if filename.is_absolute() or '..' in filename.parts or receipt.get('status') != 200:
|
||||
raise ValueError('Unsafe or unsuccessful frozen response')
|
||||
data = (Path(snapshots)/filename).read_bytes()
|
||||
if len(data) > 8*1024**2 or hashlib.sha256(data).hexdigest() != receipt['sha256'] or len(data) != receipt['bytes']:
|
||||
raise ValueError('Frozen publisher response checksum or size mismatch')
|
||||
observed = receipt['finished_at']
|
||||
else:
|
||||
data = fetch(url, cache, 8*1024**2, delay=1)
|
||||
receipt = json.loads((Path(cache)/hashlib.sha256(url.encode()).hexdigest()).with_suffix('.json').read_text())
|
||||
observed = receipt['retrieved_at']
|
||||
evidence = {'url': url, 'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data),
|
||||
'file': 'evidence/'+hashlib.sha256(data).hexdigest()+'.html', 'observed_at': observed,
|
||||
'http_status': 200, 'response_receipt': receipt}
|
||||
pages[key] = (data, evidence)
|
||||
receipts.append(evidence)
|
||||
return pages, receipts
|
||||
|
||||
|
||||
def _collect(year, out, *, cache=None, snapshots=None):
|
||||
if year != 2012:
|
||||
raise ValueError('Only the independently reviewed AAMAS 2012 layout is supported')
|
||||
pages, receipts = read_pages(cache, snapshots)
|
||||
intro = Document(pages['introduction'][0].decode('utf-8'))
|
||||
if [clean(n.text()) for n in intro.root.walk() if n.tag == 'title'] != ['AAMAS 2012']:
|
||||
raise ValueError('Unexpected proceedings introduction')
|
||||
records = []
|
||||
for section, (_, _, _, expected) in SECTIONS.items():
|
||||
data, evidence = pages[section]
|
||||
group = parse_section(data.decode('utf-8'), section, evidence['observed_at'], evidence)
|
||||
if len(group) != expected:
|
||||
raise ValueError('Publisher section count differs from reviewed inventory: '+section)
|
||||
records.extend(group)
|
||||
if len({r['external_id'] for r in records}) != len(records):
|
||||
raise ValueError('Paper IDs collide across publisher sections')
|
||||
grouped = defaultdict(list)
|
||||
for record in records:
|
||||
grouped[record['track']].append(record)
|
||||
out = Path(out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
files = []
|
||||
with tempfile.TemporaryDirectory(prefix='.aamas-', dir=out) as temporary:
|
||||
stage = Path(temporary)
|
||||
(stage/'evidence').mkdir()
|
||||
for data, evidence in pages.values():
|
||||
(stage/evidence['file']).write_bytes(data)
|
||||
for track, group in sorted(grouped.items()):
|
||||
path = stage/f'aamas-proceedings-2012-{track}.jsonl'
|
||||
path.write_text(''.join(json.dumps(r, ensure_ascii=False)+'\n' for r in group))
|
||||
manifest = {'source': SOURCE, 'parser_version': 1, 'venue': 'AAMAS', 'year': 2012, 'track': track,
|
||||
'count': len(group), 'file': str(out/path.name), 'file_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
'list_url': ROOT, 'status': 'partial', 'missing_abstract_count': sum(not r['abstract'] for r in group),
|
||||
'publication_format_counts': dict(Counter(r['metadata']['publication_format'] for r in group)),
|
||||
'note': 'Publisher sections verified; individual full/extended submission track, decision history and independent identities still need review.'}
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps(manifest, ensure_ascii=False, indent=2)+'\n')
|
||||
files.append(manifest)
|
||||
result = {'source': SOURCE, 'year': year, 'finished_at': timestamp(), 'status': 'collected_pending_identity_review',
|
||||
'records': len(records), 'files': files, 'source_receipts': receipts,
|
||||
'publication_format_counts': dict(Counter(r['metadata']['publication_format'] for r in records)),
|
||||
'database_written': False, 'pdfs_downloaded': False,
|
||||
'note': 'Aggregate submission/withdrawal totals do not identify individual rejected or withdrawn papers.'}
|
||||
(stage/'aamas-proceedings-2012.json').write_text(json.dumps(result, ensure_ascii=False, indent=2)+'\n')
|
||||
# Validated files first; write the completed collection report last.
|
||||
(out/'evidence').mkdir(exist_ok=True)
|
||||
for path in (stage/'evidence').iterdir():
|
||||
path.replace(out/'evidence'/path.name)
|
||||
for path in stage.iterdir():
|
||||
if path.is_file() and path.name != 'aamas-proceedings-2012.json':
|
||||
path.replace(out/path.name)
|
||||
(stage/'aamas-proceedings-2012.json').replace(out/'aamas-proceedings-2012.json')
|
||||
return result
|
||||
|
||||
|
||||
def collect(year, out, *, cache=None, snapshots=None):
|
||||
if year != 2012:
|
||||
raise ValueError('Only the independently reviewed AAMAS 2012 layout is supported')
|
||||
out = Path(out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
with (out/'aamas.collect.lock').open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
return _collect(year, out, cache=cache, snapshots=snapshots)
|
||||
|
||||
|
||||
def main(argv=None, *, prog=None):
|
||||
parser = argparse.ArgumentParser(prog=prog)
|
||||
parser.add_argument('--year', type=int, choices=[2012], required=True)
|
||||
parser.add_argument('--out', required=True)
|
||||
source = parser.add_mutually_exclusive_group(required=True)
|
||||
source.add_argument('--cache')
|
||||
source.add_argument('--snapshots')
|
||||
args = parser.parse_args(argv)
|
||||
print(json.dumps(collect(args.year, args.out, cache=args.cache, snapshots=args.snapshots), ensure_ascii=False))
|
||||
@@ -54,7 +54,34 @@ HISTORICAL_WORKSHOP_TITLES = {
|
||||
}
|
||||
|
||||
|
||||
# Exact pinned volume titles corroborated by official series histories. The
|
||||
# original title (including Workshop) remains in source metadata. In particular
|
||||
# ML91 has no four-digit year in its title; its exact key/title and bibliographic
|
||||
# year are checked independently, rather than guessing a title year.
|
||||
HISTORICAL_PRIMARY_EVIDENCE = {
|
||||
'icml': ('ML', 'https://icml.cc/Conferences/2008/past_icmls.shtml.html'),
|
||||
'iros': ('IROS', 'https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/'),
|
||||
}
|
||||
HISTORICAL_PRIMARY_TITLES = {
|
||||
('icml', 1988): 'Machine Learning, Proceedings of the Fifth International Conference on Machine Learning, Ann Arbor, Michigan, USA, June 12-14, 1988',
|
||||
('icml', 1989): 'Proceedings of the Sixth International Workshop on Machine Learning (ML 1989), Cornell University, Ithaca, New York, USA, June 26-27, 1989',
|
||||
('icml', 1990): 'Machine Learning, Proceedings of the Seventh International Conference on Machine Learning, Austin, Texas, USA, June 21-23, 1990',
|
||||
('icml', 1991): 'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA',
|
||||
('icml', 1992): 'Proceedings of the Ninth International Workshop on Machine Learning (ML 1992), Aberdeen, Scotland, UK, July 1-3, 1992',
|
||||
('iros', 1988): "IEEE International Workshop on Intelligent Robots and Systems '88, Proceedings. IROS 1988, Tokyo, Japan, October 31 - November 2, 1988",
|
||||
('iros', 1989): "IEEE International Workshop on Intelligent Robots and Systems '89, Proceedings. IROS 1989, September 4-6, 1989, Tsukuba, Japan",
|
||||
('iros', 1990): "IEEE International Workshop on Intelligent Robots and Systems '90, Towards a New Frontier of Applications, Proceedings. IROS 1990, July 3-6, 1990, Ibaraki, Japan",
|
||||
}
|
||||
|
||||
|
||||
def historical_track(family, base, suffix, volume_key, year, title):
|
||||
primary = HISTORICAL_PRIMARY_EVIDENCE.get(family)
|
||||
if (primary and not suffix and base.casefold() == primary[0].casefold()
|
||||
and volume_key == f'conf/{family}/{year}'
|
||||
and title == HISTORICAL_PRIMARY_TITLES.get((family, year))):
|
||||
return ('main',
|
||||
'Historical primary series; exact proceedings key, complete title and bibliographic/event year corroborated by official series history. Workshop in this original title is not an affiliated track.',
|
||||
primary[1])
|
||||
if suffix or base.casefold() != family or volume_key != f'conf/{family}/{year}':
|
||||
return None
|
||||
prefix = HISTORICAL_WORKSHOP_TITLES.get((family, year))
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
"""Frozen official ICLR 2014–2016 lists; presentation evidence is not a track."""
|
||||
import argparse
|
||||
from collections import Counter, defaultdict
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import tempfile
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from paper_atlas.db.submissions import timestamp
|
||||
from paper_atlas.io.html import Document, Node, clean
|
||||
from paper_atlas.io.http import fetch
|
||||
|
||||
SOURCE = 'iclr_archive'
|
||||
PAGES = {
|
||||
2014: ('https://iclr.cc/archive/2014/conference-proceedings/', {'main': 35}),
|
||||
2015: ('https://iclr.cc/archive/www/doku.php%3Fid=iclr2015:accepted-main.html', {'main': 31, 'workshop': 74}),
|
||||
2016: ('https://iclr.cc/archive/www/doku.php%3Fid=iclr2016:accepted-main.html', {'main': 80}),
|
||||
}
|
||||
SECTIONS = {
|
||||
2015: {'Main Conference - Oral Presentations': ('main', 'oral'),
|
||||
'Main Conference - Poster Presentations': ('main', 'poster'),
|
||||
'Workshop Papers': ('workshop', 'workshop')},
|
||||
2016: {'Oral Presentations': ('main', 'oral'), 'Poster Presentations': ('main', 'poster')},
|
||||
}
|
||||
|
||||
|
||||
def arxiv_link(url):
|
||||
parsed = urlsplit(url)
|
||||
if parsed.hostname != 'arxiv.org':
|
||||
return None
|
||||
match = re.fullmatch(r'/(?:abs|pdf)/(\d{4}\.\d{4,5})(v[1-9]\d*)?(?:\.pdf)?', parsed.path)
|
||||
if (parsed.scheme not in ('http', 'https') or parsed.netloc != 'arxiv.org'
|
||||
or parsed.query or parsed.fragment or not match):
|
||||
raise ValueError('Unexpected official arXiv link')
|
||||
return match[1], match[2] or None
|
||||
|
||||
|
||||
def without_links(node):
|
||||
if node.tag in ('a', 'script', 'style'):
|
||||
return ''
|
||||
value = ''.join(without_links(n) if isinstance(n, Node) else n for n in node.children)
|
||||
return ' '+value+' ' if node.tag in ('div', 'li', 'p', 'br') else value
|
||||
|
||||
|
||||
def authors(display, year):
|
||||
parts = display.rstrip(';').split(';') if year == 2014 else re.split(r',\s*(?:and\s+)?|\s+and\s+', display)
|
||||
names = [clean(p) for p in parts]
|
||||
if not names or any(not name for name in names):
|
||||
raise ValueError('Missing author or unsupported author delimiter')
|
||||
return names
|
||||
|
||||
|
||||
def parse_page(html, year, evidence):
|
||||
if year not in PAGES:
|
||||
raise ValueError('Unreviewed ICLR archive year')
|
||||
document = Document(html)
|
||||
nodes = list(document.root.walk())
|
||||
headings = [clean(n.text()) for n in nodes if n.tag in ('h1', 'h2', 'h3')]
|
||||
if year == 2014:
|
||||
if 'Conference Proceedings' not in headings or 'conference papers accepted' not in clean(document.root.text()):
|
||||
raise ValueError('Unexpected ICLR 2014 accepted list')
|
||||
elif (set(SECTIONS[year]) - set(headings)
|
||||
or (year == 2016 and 'Accepted Papers (Conference Track)' not in headings)):
|
||||
raise ValueError('Unexpected ICLR accepted sections')
|
||||
entries = []
|
||||
section = None
|
||||
for index, node in enumerate(nodes):
|
||||
if node.tag in ('h1', 'h2', 'h3'):
|
||||
section = clean(node.text())
|
||||
if year == 2014:
|
||||
if node.tag != 'a' or not arxiv_link(node.attrs.get('href', '')):
|
||||
continue
|
||||
link = node
|
||||
# This archived Google Sites page has an entry outside a paragraph.
|
||||
# Authors can span several differently formatted italic spans.
|
||||
# Read their whole paragraph, never only the first styled fragment.
|
||||
display = None
|
||||
for following in nodes[index+sum(1 for _ in link.walk()):]:
|
||||
if following.tag == 'a' and arxiv_link(following.attrs.get('href', '')):
|
||||
break
|
||||
if following.tag in ('i', 'em') or (following.tag == 'p' and any(
|
||||
n.tag in ('i', 'em') or 'font-style:italic' in n.attrs.get('style', '').replace(' ', '')
|
||||
for n in following.walk())):
|
||||
text = clean(following.text())
|
||||
if text:
|
||||
display = text
|
||||
break
|
||||
if not display:
|
||||
raise ValueError('Missing explicit ICLR 2014 author span')
|
||||
track, presentation = 'main', 'conference'
|
||||
else:
|
||||
if node.tag != 'li':
|
||||
continue
|
||||
links = [n for n in node.walk() if n.tag == 'a' and arxiv_link(n.attrs.get('href', ''))]
|
||||
if not links:
|
||||
continue
|
||||
if len(links) != 1 or section not in SECTIONS[year]:
|
||||
raise ValueError('Ambiguous paper link or unknown accepted section')
|
||||
link, = links
|
||||
display = clean(without_links(node)).lstrip(',').strip()
|
||||
track, presentation = SECTIONS[year][section]
|
||||
arxiv, version = arxiv_link(link.attrs['href'])
|
||||
title = clean(link.text())
|
||||
if not title:
|
||||
raise ValueError('Empty official title')
|
||||
entries.append({'arxiv': arxiv, 'version': version, 'title': title,
|
||||
'authors': authors(display, year), 'authors_display': display,
|
||||
'paper_url': link.attrs['href'], 'track': track,
|
||||
'section': section, 'presentation': presentation})
|
||||
# Check every arXiv occurrence, including links outside the expected layout.
|
||||
occurrences = sum(n.tag == 'a' and bool(arxiv_link(n.attrs.get('href', ''))) for n in nodes)
|
||||
if len(entries) != occurrences or not entries:
|
||||
raise ValueError('Some official paper links were not parsed')
|
||||
grouped = defaultdict(list)
|
||||
for entry in entries:
|
||||
grouped[(entry['track'], entry['arxiv'])].append(entry)
|
||||
result = []
|
||||
for (track, arxiv), variants in grouped.items():
|
||||
first = variants[0]
|
||||
if any((v['title'], v['authors']) != (first['title'], first['authors']) for v in variants):
|
||||
raise ValueError('Repeated presentation has conflicting title or authors; review required')
|
||||
presentations = sorted({v['presentation'] for v in variants})
|
||||
result.append({'source': SOURCE, 'external_id': f'{year}/{track}/{arxiv}',
|
||||
'url': PAGES[year][0], 'title': first['title'], 'abstract': None,
|
||||
'authors': first['authors'], 'identifiers': {'arxiv': arxiv},
|
||||
'venue': 'ICLR', 'venue_aliases': [], 'year': year, 'track': track,
|
||||
'status': 'accepted', 'observed_at': evidence['observed_at'],
|
||||
'raw_decision': 'listed in official ICLR accepted '+track+' list',
|
||||
'metadata': {'source_evidence': evidence, 'source_variants': variants,
|
||||
'presentations': presentations, 'decision_date_precision': 'not_supplied',
|
||||
'identity_note': 'Official arXiv link, not title matching; validate canonical identity before import.',
|
||||
'decision_evidence': 'Archived accepted list, not an original decision note.'},
|
||||
'tags': [{'namespace': SOURCE+':presentation', 'value': value} for value in presentations]})
|
||||
return result
|
||||
|
||||
|
||||
def read_page(year, *, cache=None, snapshots=None):
|
||||
if bool(cache) == bool(snapshots):
|
||||
raise ValueError('Choose exactly one cache or frozen snapshot directory')
|
||||
url = PAGES[year][0]
|
||||
if snapshots:
|
||||
root = Path(snapshots).resolve()
|
||||
receipts = json.loads((root/'receipts.json').read_bytes())['receipts']
|
||||
matches = [r for r in receipts if r['url'] == url]
|
||||
if len(matches) != 1:
|
||||
raise ValueError('Expected one exact official response receipt')
|
||||
receipt, = matches
|
||||
filename = Path(receipt['file'])
|
||||
path = root/filename
|
||||
if (filename.is_absolute() or '..' in filename.parts or not path.resolve().is_relative_to(root)
|
||||
or receipt['status'] != 200):
|
||||
raise ValueError('Unsafe or unsuccessful archive snapshot')
|
||||
data = path.read_bytes()
|
||||
if (len(data) > 8*1024**2 or len(data) != receipt['bytes']
|
||||
or hashlib.sha256(data).hexdigest() != receipt['sha256']):
|
||||
raise ValueError('Archive snapshot checksum or size differs')
|
||||
observed = timestamp(receipt['finished_at'])
|
||||
else:
|
||||
data = fetch(url, cache, 8*1024**2, delay=1)
|
||||
receipt = json.loads((Path(cache)/hashlib.sha256(url.encode()).hexdigest()).with_suffix('.json').read_bytes())
|
||||
observed = timestamp(receipt['retrieved_at'])
|
||||
evidence = {'url': url, 'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data),
|
||||
'file': 'evidence/'+hashlib.sha256(data).hexdigest()+'.html',
|
||||
'observed_at': observed, 'http_status': 200, 'response_receipt': receipt}
|
||||
return data, evidence
|
||||
|
||||
|
||||
def collect(year, out, *, cache=None, snapshots=None):
|
||||
if year not in PAGES:
|
||||
raise ValueError('Only independently reviewed ICLR archive years are supported')
|
||||
out = Path(out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
with (out/'iclr-archive.collect.lock').open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
report = out/f'iclr-archive-{year}.json'
|
||||
if report.exists():
|
||||
raise FileExistsError('Existing completed archive output must be inspected, not overwritten')
|
||||
data, evidence = read_page(year, cache=cache, snapshots=snapshots)
|
||||
records = parse_page(data.decode('utf-8'), year, evidence)
|
||||
counts = Counter(r['track'] for r in records)
|
||||
if dict(counts) != PAGES[year][1]:
|
||||
raise ValueError('Distinct archive entries differ from reviewed inventory')
|
||||
files = []
|
||||
with tempfile.TemporaryDirectory(prefix='.iclr-archive-', dir=out) as temporary:
|
||||
stage = Path(temporary)
|
||||
(stage/'evidence').mkdir()
|
||||
(stage/evidence['file']).write_bytes(data)
|
||||
for track in sorted(counts):
|
||||
group = [r for r in records if r['track'] == track]
|
||||
path = stage/f'iclr-archive-{year}-{track}.jsonl'
|
||||
path.write_text(''.join(json.dumps(r, ensure_ascii=False)+'\n' for r in group))
|
||||
manifest = {'source': SOURCE, 'parser_version': 1, 'venue': 'ICLR', 'year': year,
|
||||
'track': track, 'count': len(group), 'file': str(out/path.name),
|
||||
'file_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
'list_url': PAGES[year][0], 'status': 'partial', 'missing_abstract_count': len(group)}
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps(manifest, ensure_ascii=False, indent=2)+'\n')
|
||||
files.append(manifest)
|
||||
result = {'source': SOURCE, 'year': year, 'records': len(records), 'files': files,
|
||||
'source_evidence': evidence, 'database_written': False, 'pdfs_downloaded': False,
|
||||
'status': 'collected_pending_identity_review',
|
||||
'note': 'Accepted list only. Oral/poster are presentation evidence; absent papers are not rejected. '
|
||||
'ICLR2014 workshop and other unlisted tracks remain outside these reviewed pages.'}
|
||||
(stage/report.name).write_text(json.dumps(result, ensure_ascii=False, indent=2)+'\n')
|
||||
(out/'evidence').mkdir(exist_ok=True)
|
||||
for path in (stage/'evidence').iterdir():
|
||||
path.replace(out/'evidence'/path.name)
|
||||
for path in stage.iterdir():
|
||||
if path.is_file() and path.name != report.name:
|
||||
path.replace(out/path.name)
|
||||
(stage/report.name).replace(report)
|
||||
return result
|
||||
|
||||
|
||||
def main(argv=None, *, prog=None):
|
||||
parser = argparse.ArgumentParser(prog=prog)
|
||||
parser.add_argument('--year', type=int, choices=sorted(PAGES), required=True)
|
||||
parser.add_argument('--out', required=True)
|
||||
source = parser.add_mutually_exclusive_group(required=True)
|
||||
source.add_argument('--cache')
|
||||
source.add_argument('--snapshots')
|
||||
args = parser.parse_args(argv)
|
||||
print(json.dumps(collect(args.year, args.out, cache=args.cache, snapshots=args.snapshots), ensure_ascii=False))
|
||||
@@ -0,0 +1,340 @@
|
||||
"""Reproducible early WWW metadata; unresolved program entries stay outside imports."""
|
||||
import argparse
|
||||
from collections import defaultdict
|
||||
import hashlib
|
||||
from html.parser import HTMLParser
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import tempfile
|
||||
from urllib.parse import urljoin, urlsplit
|
||||
|
||||
from paper_atlas.io.html import Document, clean
|
||||
from paper_atlas.io.http import fetch
|
||||
|
||||
ROOT = 'https://archives.iw3c2.org/'
|
||||
SPRING = ROOT+'www3/proceedings/'
|
||||
FALL = ROOT+'www2/Proceedings/WWW2_Proceedings.html'
|
||||
SEEDS = [ROOT+'www1/PrelimProcs.html', ROOT+'www1/Proceedings.html', FALL,
|
||||
SPRING+'proceedings.html', *[SPRING+'papers/'+x+'.html' for x in ('author','session','abstract')],
|
||||
*[SPRING+'posters/'+x+'.html' for x in ('author','abstract')]]
|
||||
|
||||
|
||||
def decode(raw):
|
||||
try:
|
||||
return raw.decode('utf-8-sig'), 'utf-8'
|
||||
except UnicodeDecodeError:
|
||||
return raw.decode('iso-8859-1'), 'iso-8859-1 fallback; original bytes retained'
|
||||
|
||||
|
||||
def text(html):
|
||||
return re.sub(r'[ \t\r\n\f\v]+',' ',Document(html).root.text()).strip(' \t\r\n\f\v')
|
||||
|
||||
|
||||
def anchors(html):
|
||||
return [(n.attrs.get('href',''),clean(n.text())) for n in Document(html).root.walk() if n.tag=='a']
|
||||
|
||||
|
||||
def source_url(url):
|
||||
p=urlsplit(url)
|
||||
if p.scheme!='https' or p.netloc!='archives.iw3c2.org' or p.query or p.fragment or not p.path.endswith('.html'):
|
||||
raise ValueError('Only exact official archive HTML source URLs are allowed')
|
||||
return url
|
||||
|
||||
|
||||
class Sources:
|
||||
def __init__(self, directory, expected_sha256):
|
||||
self.root=Path(directory).resolve();manifest=self.root/'manifest.json';raw=manifest.read_bytes()
|
||||
if hashlib.sha256(raw).hexdigest()!=expected_sha256:
|
||||
raise ValueError('Source cache manifest checksum mismatch')
|
||||
self.rows={};self.used=set()
|
||||
for row in json.loads(raw)['sources']:
|
||||
url=source_url(row['url'])
|
||||
if url in self.rows or row.get('status')!=200:
|
||||
raise ValueError('Duplicate or incomplete source response')
|
||||
p=self.root/row['file']
|
||||
if p.is_symlink() or p.parent!=self.root or not p.is_file():
|
||||
raise ValueError('Source cache paths must be plain files directly within the cache')
|
||||
data=p.read_bytes()
|
||||
if len(data)>2*1024**2 or len(data)!=row['bytes'] or hashlib.sha256(data).hexdigest()!=row['sha256']:
|
||||
raise ValueError('Source response checksum or size mismatch')
|
||||
self.rows[url]={'raw':data,'receipt':row}
|
||||
|
||||
def read(self,url):
|
||||
source_url(url)
|
||||
if url not in self.rows:
|
||||
raise ValueError('Required source missing from cache: '+url)
|
||||
self.used.add(url)
|
||||
return decode(self.rows[url]['raw'])[0]
|
||||
|
||||
def proof(self,urls):
|
||||
return [{'url':url,'sha256':self.rows[url]['receipt']['sha256'],
|
||||
'encoding':decode(self.rows[url]['raw'])[1]} for url in sorted(set(urls))]
|
||||
|
||||
|
||||
def download_sources(http_cache, directory):
|
||||
directory=Path(directory)
|
||||
if directory.exists():
|
||||
raise ValueError('Refusing to overwrite a source snapshot')
|
||||
directory.mkdir(parents=True)
|
||||
responses={url:fetch(url,http_cache,2*1024**2) for url in SEEDS}
|
||||
html=decode(responses[FALL])[0]
|
||||
for href,label in anchors(html):
|
||||
if href!='Author_Inst.html' and href.endswith('.html'):
|
||||
url=source_url(urljoin(FALL,href))
|
||||
if not url.startswith(ROOT+'www2/Proceedings/'):
|
||||
raise ValueError('Unexpected fall category source')
|
||||
responses[url]=fetch(url,http_cache,2*1024**2)
|
||||
receipts=[]
|
||||
for i,(url,raw) in enumerate(sorted(responses.items())):
|
||||
name='source-'+str(i).zfill(2)+'.html';(directory/name).write_bytes(raw)
|
||||
receipts.append({'url':url,'status':200,'file':name,'bytes':len(raw),'sha256':hashlib.sha256(raw).hexdigest()})
|
||||
manifest=directory/'manifest.json';manifest.write_text(json.dumps({'sources':receipts},indent=2)+'\n')
|
||||
return hashlib.sha256(manifest.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def identity(href,track):
|
||||
url=urljoin(SPRING+track+'/',href);p=urlsplit(url)
|
||||
match=re.fullmatch(r'/www3/proceedings/'+track+r'/(\d+)/[^?#]+',p.path)
|
||||
if p.scheme=='https' and p.netloc=='archives.iw3c2.org' and not p.query and not p.fragment and match:
|
||||
return match[1],url
|
||||
return None
|
||||
|
||||
|
||||
class AuthorIndex(HTMLParser):
|
||||
def __init__(self,html,track):
|
||||
super().__init__(convert_charrefs=True);self.track=track;self.name='';self.bold=None;self.link=None;self.link_text=[]
|
||||
self.entries=defaultdict(lambda:{'authors':[],'urls':set(),'titles':set()});self.feed(html)
|
||||
|
||||
def handle_starttag(self,tag,attrs):
|
||||
if tag=='b':self.bold=[]
|
||||
if tag=='a':self.link=dict(attrs).get('href');self.link_text=[]
|
||||
|
||||
def handle_data(self,data):
|
||||
if self.bold is not None:self.bold.append(data)
|
||||
if self.link is not None:self.link_text.append(data)
|
||||
|
||||
def handle_endtag(self,tag):
|
||||
if tag=='b' and self.bold is not None:
|
||||
self.name=clean(''.join(self.bold));self.bold=None
|
||||
if tag=='a' and self.link is not None:
|
||||
own=identity(self.link,self.track)
|
||||
if own:
|
||||
key,url=own;entry=self.entries[key];entry['urls'].add(url);entry['titles'].add(clean(''.join(self.link_text)))
|
||||
if not self.name:raise ValueError('Paper link has no archive author context')
|
||||
if self.name not in entry['authors']:entry['authors'].append(self.name)
|
||||
self.link=None
|
||||
|
||||
|
||||
def sections(html):
|
||||
"""Use explicit h3 boundaries; old HTML omits many p/li closing tags."""
|
||||
result=[]
|
||||
for chunk in re.split(r'<h3\b[^>]*>',html,flags=re.I)[1:]:
|
||||
parts=re.split(r'</h3\s*>',chunk,maxsplit=1,flags=re.I)
|
||||
if len(parts)!=2:raise ValueError('Unclosed archive entry heading')
|
||||
heading,body=parts
|
||||
body=re.split(r'<(?:hr|h2|address)\b',body,maxsplit=1,flags=re.I)[0]
|
||||
author=re.search(r'<i\b[^>]*>(.*?)</i\s*>',body,re.I|re.S)
|
||||
paragraphs=[]
|
||||
for paragraph in re.findall(r'<p\b[^>]*>(.*?)(?=</p\s*>|<p\b|$)',body,re.I|re.S):
|
||||
value=text(paragraph)
|
||||
if value and not value.casefold().startswith('keywords:'):paragraphs.append(value)
|
||||
result.append({'heading':heading,'title':text(heading),'abstract':'\n\n'.join(paragraphs)or None,
|
||||
'author_byline_raw':text(author[1])if author else None})
|
||||
if not result:raise ValueError('No explicit archive entry headings')
|
||||
return result
|
||||
|
||||
|
||||
def reviewed_posters(sources, path, expected_sha256, cache_sha256):
|
||||
if path is None:
|
||||
if expected_sha256:raise ValueError('Reviewed poster path required')
|
||||
return {},None
|
||||
raw=Path(path).read_bytes()
|
||||
if hashlib.sha256(raw).hexdigest()!=expected_sha256:raise ValueError('Reviewed poster checksum mismatch')
|
||||
review=json.loads(raw)
|
||||
if set(review)!={'format','source_cache_sha256','records'} or review['format']!=1 or review['source_cache_sha256']!=cache_sha256:
|
||||
raise ValueError('Reviewed poster snapshot binding differs')
|
||||
if not isinstance(review['records'],list) or not 0<len(review['records'])<=100:
|
||||
raise ValueError('Bounded nonempty poster review required')
|
||||
result={}
|
||||
for row in review['records']:
|
||||
if set(row)!={'entry_id','individual_url','individual_sha256','title','authors','author_source','reason','missing_abstract_reason'}:
|
||||
raise ValueError('Unexpected reviewed poster fields')
|
||||
key=row['entry_id'];own=identity(row['individual_url'],'posters')
|
||||
if not isinstance(key,str) or not own or own[0]!=key or key in result:
|
||||
raise ValueError('Reviewed poster identity duplicated or outside canonical entry')
|
||||
if row['author_source'] not in ('canonical_byline','author_index') or not isinstance(row['authors'],list) or not row['authors'] or any(not isinstance(a,str) or not a.strip()for a in row['authors']):
|
||||
raise ValueError('Explicit author evidence required')
|
||||
if len(set(row['authors']))!=len(row['authors']) or not isinstance(row['reason'],str) or not row['reason'].strip():
|
||||
raise ValueError('Reviewed author order/reason invalid')
|
||||
if sources.rows.get(own[1],{}).get('receipt',{}).get('sha256')!=row['individual_sha256']:
|
||||
raise ValueError('Individual poster page checksum differs')
|
||||
page=sources.read(own[1]);headings=re.findall(r'<h1\b[^>]*>(.*?)</h1\s*>',page,re.I|re.S)
|
||||
if len(headings)!=1 or text(headings[0])!=row['title']:
|
||||
raise ValueError('Reviewed title differs from exact canonical H1')
|
||||
header=re.split(r'</h1\s*>',page,maxsplit=1,flags=re.I)[1]
|
||||
header=re.split(r'<h[23]\b',header,maxsplit=1,flags=re.I)[0]
|
||||
header=re.split(r'\b(?:keywords|abstract)\s*:',text(header),maxsplit=1,flags=re.I)[0]
|
||||
if row['author_source']=='canonical_byline' and any(a.casefold() not in header.casefold()for a in row['authors']):
|
||||
raise ValueError('Reviewed author absent from canonical byline region')
|
||||
result[key]={**row,'canonical_byline_region_raw':header}
|
||||
return result,hashlib.sha256(raw).hexdigest()
|
||||
|
||||
|
||||
def reviewed_row(sources,key,url,item,authors,review):
|
||||
if url!=review['individual_url']:
|
||||
raise ValueError('Reviewed canonical page differs from abstract-entry URL')
|
||||
if authors and authors['urls']!={url}:
|
||||
raise ValueError('Reviewed poster author-index URL disagrees')
|
||||
if review['author_source']=='author_index':
|
||||
names=[]
|
||||
for name in authors['authors']if authors else []:
|
||||
parts=name.split(',');names.append(clean(parts[1]+' '+parts[0])if len(parts)==2 else name)
|
||||
if names!=review['authors']:raise ValueError('Reviewed author-index evidence differs')
|
||||
missing=review['missing_abstract_reason']
|
||||
if not isinstance(missing,str) or bool(missing.strip())==bool(item['abstract']):
|
||||
raise ValueError('Missing abstract requires explicit reason; never invent a summary')
|
||||
urls=[SPRING+'posters/author.html',SPRING+'posters/abstract.html',url]
|
||||
return {'source':'www_archive','external_id':'www3:posters:'+key,'url':url,'title':review['title'],
|
||||
'abstract':item['abstract'],'authors':review['authors'],'identifiers':{},'venue':'WWW','year':1995,
|
||||
'track':'poster','event_key':'www-1995-spring','status':'accepted','raw_decision':'listed in official posters proceedings',
|
||||
'metadata':{'entry_id':key,'source_evidence':sources.proof(urls),'abstract_heading_title_raw':item['title'],
|
||||
'author_byline_raw':item['author_byline_raw'],'author_index_raw':authors['authors']if authors else [],
|
||||
'author_index_titles_raw':sorted(authors['titles'])if authors else [],'abstract_href_raw':url,
|
||||
'author_order':review['author_source'],'reviewed_poster_identity':review,
|
||||
'missing_abstract_reason':missing or None},
|
||||
'tags':[{'namespace':'www_archive:proceedings_section','value':'posters'}]}
|
||||
|
||||
|
||||
def spring_rows(sources,track,reviews=None):
|
||||
base=SPRING+track+'/';author_url=base+'author.html';abstract_url=base+'abstract.html'
|
||||
authors=AuthorIndex(sources.read(author_url),track).entries
|
||||
listing=sections(sources.read(abstract_url));by_id={};held=[];session={}
|
||||
if track=='papers':
|
||||
session_url=base+'session.html'
|
||||
for href,title in anchors(sources.read(session_url)):
|
||||
own=identity(href,track)
|
||||
if own:
|
||||
key,url=own
|
||||
if key in session:raise ValueError('Duplicate main session entry ID')
|
||||
session[key]=url
|
||||
for item in listing:
|
||||
links=[identity(h,track)for h,t in anchors(item['heading'])if identity(h,track)]
|
||||
if len(links)!=1:
|
||||
held.append({'reason':'unverified_heading_identity','evidence':abstract_url,'candidate':item});continue
|
||||
key,url=links[0]
|
||||
if key in by_id:raise ValueError('Duplicate abstract entry ID')
|
||||
by_id[key]=(url,item)
|
||||
if not by_id:raise ValueError('No stable abstract entry IDs')
|
||||
if track=='papers' and (set(authors)!=set(by_id) or set(session)!=set(by_id)):
|
||||
raise ValueError('Main author/session/abstract entry ID sets disagree')
|
||||
if set(authors)-set(by_id):raise ValueError('Author index has entries outside the abstract listing')
|
||||
reviews=reviews or {}
|
||||
if set(reviews)-set(by_id):raise ValueError('Reviewed poster outside abstract listing')
|
||||
rows=[]
|
||||
for key,(url,item) in by_id.items():
|
||||
if key in reviews:
|
||||
if track!='posters':raise ValueError('Reviews only apply to poster entries')
|
||||
rows.append(reviewed_row(sources,key,url,item,authors.get(key),reviews[key]));continue
|
||||
if key not in authors:
|
||||
held.append({'reason':'abstract_entry_missing_linked_author_index','entry_id':key,'evidence':abstract_url,'candidate':item});continue
|
||||
alternatives=authors[key]['urls']
|
||||
if len(alternatives)!=1:raise ValueError('Ambiguous author-index paper URL')
|
||||
canonical=next(iter(alternatives))
|
||||
if track=='papers' and session[key]!=canonical:raise ValueError('Main author/session paper URLs disagree')
|
||||
if url!=canonical and (track!='papers' or not re.search(r'\s',url)):
|
||||
raise ValueError('Uncorroborated archive URL discrepancy')
|
||||
if not item['abstract']:
|
||||
held.append({'reason':'missing_abstract','entry_id':key,'evidence':abstract_url,'candidate':item});continue
|
||||
if item['title'].casefold() not in {t.casefold() for t in authors[key]['titles']}:
|
||||
held.append({'reason':'abstract_and_author_index_titles_disagree','entry_id':key,'evidence':abstract_url,'candidate':item,'author_index_titles':sorted(authors[key]['titles'])});continue
|
||||
names=[]
|
||||
for name in authors[key]['authors']:
|
||||
parts=name.split(',')
|
||||
names.append(clean(parts[1]+' '+parts[0])if len(parts)==2 else name)
|
||||
proof_urls=[author_url,abstract_url]+([session_url]if track=='papers'else[])
|
||||
rows.append({'source':'www_archive','external_id':'www3:'+track+':'+key,'url':canonical,
|
||||
'title':item['title'],'abstract':item['abstract'],'authors':names,'identifiers':{},
|
||||
'venue':'WWW','year':1995,'track':'main'if track=='papers'else'poster','event_key':'www-1995-spring',
|
||||
'status':'accepted','raw_decision':'listed in official '+track+' proceedings',
|
||||
'metadata':{'entry_id':key,'source_evidence':sources.proof(proof_urls),
|
||||
'author_index_raw':authors[key]['authors'],'author_index_titles_raw':sorted(authors[key]['titles']),'author_order':'archive author-index order, not original byline rank',
|
||||
'author_byline_raw':item['author_byline_raw'],'abstract_href_raw':url,
|
||||
'url_corroborated_by_author_and_session':url!=canonical},
|
||||
'tags':[{'namespace':'www_archive:proceedings_section','value':track}]})
|
||||
return rows,held,{'abstract_entries':len(by_id),'linked_author_entries':len(authors),'ready':len(rows),'held':len(held)}
|
||||
|
||||
|
||||
def early_review(sources):
|
||||
held=[];url=ROOT+'www1/PrelimProcs.html';html=sources.read(url)
|
||||
for ordinal,chunk in enumerate(re.split(r'<li\b[^>]*>',html,flags=re.I)[1:],1):
|
||||
title=re.search(r'<em>Title:</em>\s*<b>(.*?)</b>',chunk,re.I|re.S)
|
||||
if not title:raise ValueError('Unclassified first-conference entry')
|
||||
names=re.findall(r'<em>Author:</em>(.*?)(?=<br\b)',chunk,re.I|re.S)
|
||||
held.append({'reason':'formal_presentation_scope_and_count_need_review','event_key':'www-1994-spring',
|
||||
'ordinal':ordinal,'evidence':url,'candidate':{'title':text(title[1]),
|
||||
'authors_raw':[re.sub(r'\s*<[^>]+>','',text(n)).strip()for n in names],
|
||||
'keywords_raw':text(re.search(r'<em>Keywords:</em>(.*?)(?=<br\b)',chunk,re.I|re.S)[1]) if re.search(r'<em>Keywords:</em>(.*?)(?=<br\b)',chunk,re.I|re.S)else None},
|
||||
'source_evidence':sources.proof([url])})
|
||||
if not held:raise ValueError('First-conference presentation listing empty')
|
||||
category_urls={}
|
||||
for href,label in anchors(sources.read(FALL)):
|
||||
if href!='Author_Inst.html' and href.endswith('.html'):
|
||||
category_url=source_url(urljoin(FALL,href))
|
||||
if not category_url.startswith(ROOT+'www2/Proceedings/'):raise ValueError('Unexpected fall category')
|
||||
category_urls[category_url]=label
|
||||
if not category_urls:raise ValueError('Fall archive category listing empty')
|
||||
for url,label in sorted(category_urls.items()):
|
||||
html=sources.read(url)
|
||||
for ordinal,chunk in enumerate(re.split(r'<dt\b[^>]*>',html,flags=re.I)[1:],1):
|
||||
fields=re.split(r'<dd\b[^>]*>',chunk,maxsplit=1,flags=re.I)
|
||||
title=text(fields[0]);raw_byline=text(re.split(r'</dl\b',fields[1],maxsplit=1,flags=re.I)[0]) if len(fields)==2 else None
|
||||
if title:
|
||||
held.append({'reason':'mixed_program_category_not_verified_paper_acceptance','event_key':'www-1994-fall',
|
||||
'category':label,'ordinal':ordinal,'evidence':url,'candidate':{'title':title,'authors_raw':raw_byline},
|
||||
'source_evidence':sources.proof([FALL,url])})
|
||||
return held,{'first_conference_fields':sum(r['event_key']=='www-1994-spring'for r in held),'fall_categories':len(category_urls),
|
||||
'fall_program_entries':sum(r['event_key']=='www-1994-fall'for r in held)}
|
||||
|
||||
|
||||
def collect(source_cache,cache_sha256,out,*,reviewed=None,reviewed_sha256=None):
|
||||
out=Path(out)
|
||||
if out.exists():raise ValueError('Refusing to overwrite an existing normalized batch')
|
||||
sources=Sources(source_cache,cache_sha256);groups={};held=[];counts={}
|
||||
reviews,review_sha=reviewed_posters(sources,reviewed,reviewed_sha256,cache_sha256)
|
||||
for track in ('papers','posters'):
|
||||
rows,review,counts[track]=spring_rows(sources,track,reviews if track=='posters'else None);groups[track]=rows;held.extend(review)
|
||||
review,counts['earlier']=early_review(sources);held.extend(review)
|
||||
for item in held:
|
||||
if 'source_evidence' not in item:item['source_evidence']=sources.proof([item['evidence']])
|
||||
out.parent.mkdir(parents=True,exist_ok=True)
|
||||
with tempfile.TemporaryDirectory(prefix='.www-archive-',dir=out.parent)as tmp:
|
||||
stage=Path(tmp);queue=[]
|
||||
for track,rows in groups.items():
|
||||
if not rows:continue
|
||||
filename='www-1995-spring-'+('main'if track=='papers'else'poster')+'.jsonl';raw=''.join(json.dumps(r,ensure_ascii=False)+'\n'for r in rows).encode();(stage/filename).write_bytes(raw)
|
||||
manifest={'source':'www_archive','parser_version':2 if review_sha else 1,'venue':'WWW','year':1995,'track':rows[0]['track'],'event_key':'www-1995-spring',
|
||||
'list_url':SPRING+track+'/abstract.html','count':len(rows),'status':'partial','file_sha256':hashlib.sha256(raw).hexdigest(),
|
||||
'missing_abstract_count':sum(not bool((r['abstract']or '').strip())for r in rows),'source_listing_count':counts[track]['abstract_entries'],'held_entry_count':counts[track]['held'],'source_cache_sha256':cache_sha256,'scope':'Verified linked archive entries; held entries and other event tracks excluded, not complete event coverage.'}
|
||||
(stage/filename).with_suffix('.manifest.json').write_text(json.dumps(manifest,indent=2)+'\n');queue.append(str(out/filename))
|
||||
(stage/'review-candidates.jsonl').write_text(''.join(json.dumps(r,ensure_ascii=False)+'\n'for r in held))
|
||||
report={'status':'normalized_not_imported','source_cache_sha256':cache_sha256,'counts':counts,'ready':sum(len(r)for r in groups.values()),'held':len(held),
|
||||
'source_evidence':sources.proof(sources.used),'files':queue,'scope':'Early WWW archive only; no PDF fetching, no rejected/withdrawn inference, no database access. Event-aware importer required.'}
|
||||
if review_sha:report['reviewed_posters_sha256']=review_sha
|
||||
(stage/'report.json').write_text(json.dumps(report,ensure_ascii=False,indent=2)+'\n')
|
||||
(stage/'import-queue.json').write_text(json.dumps({'files':queue},indent=2)+'\n')
|
||||
stage.rename(out)
|
||||
return report
|
||||
|
||||
|
||||
def main(argv=None,*,prog=None):
|
||||
p=argparse.ArgumentParser(prog=prog);p.add_argument('--source-cache',required=True);p.add_argument('--source-cache-sha256');p.add_argument('--http-cache');p.add_argument('--out',required=True)
|
||||
p.add_argument('--reviewed-posters');p.add_argument('--reviewed-posters-sha256')
|
||||
args=p.parse_args(argv)
|
||||
if bool(args.http_cache)==bool(args.source_cache_sha256):p.error('Choose a fresh --http-cache download or an existing --source-cache-sha256 snapshot')
|
||||
sha=download_sources(args.http_cache,args.source_cache)if args.http_cache else args.source_cache_sha256
|
||||
if bool(args.reviewed_posters)!=bool(args.reviewed_posters_sha256):p.error('Reviewed poster file and SHA256 must be supplied together')
|
||||
print(json.dumps(collect(args.source_cache,sha,args.out,reviewed=args.reviewed_posters,reviewed_sha256=args.reviewed_posters_sha256),ensure_ascii=False),flush=True)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -4,7 +4,7 @@ import re
|
||||
from paper_atlas.io.publication_ids import mlsys_id
|
||||
|
||||
|
||||
PUBLICATIONS={'pmlr','acl_anthology','colm','neurips','mlsys','cvf','ecva','dblp'}
|
||||
PUBLICATIONS={'pmlr','acl_anthology','colm','neurips','mlsys','cvf','ecva','dblp','aamas_proceedings'}
|
||||
|
||||
|
||||
def timestamp(value=None):
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Publisher evidence, section boundaries and native-ingestion regression checks."""
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from paper_atlas.collectors import aamas
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
|
||||
def paper(pid, abstract=True):
|
||||
summary = '<span class="abstract">Research with <i>evidence</i>.<br>More detail.</span>' if abstract else ''
|
||||
return f'''<div class="paper" id="{pid}"><span class="id">{pid}</span>
|
||||
<p class="paper"><span class="title"><a href="../papers/{pid}.pdf">{summary}
|
||||
A structured & auditable paper {pid}</a></span><span class="authors">
|
||||
<a href="authors.html#Jane">Jane Doe<span class="amount">has 7 papers</span></a>, Bob Smith
|
||||
</span></p></div>'''
|
||||
|
||||
|
||||
def page(section, pid, abstract=True):
|
||||
heading = aamas.SECTIONS[section][0]
|
||||
return (f'<html><head><title>{heading} - AAMAS 2012</title></head><body>'
|
||||
f'<h1>{heading}</h1><h2 class="section" id="s-1A">Session 1A – Robotics I</h2>'
|
||||
+paper(pid, abstract)+'</body></html>')
|
||||
|
||||
|
||||
class AamasProceedingsTests(unittest.TestCase):
|
||||
def snapshots(self, root):
|
||||
root.mkdir()
|
||||
receipts = []
|
||||
items = {'introduction': ('<title>AAMAS 2012</title><p>Aggregate withdrawals do not identify papers.</p>', aamas.ROOT)}
|
||||
for index, section in enumerate(aamas.SECTIONS):
|
||||
items[section] = (page(section, f'P{index}_1', section != 'extabstracts'), aamas.ROOT+'resources/'+section+'.html')
|
||||
for key, (html, url) in items.items():
|
||||
data = html.encode()
|
||||
(root/(key+'.html')).write_bytes(data)
|
||||
receipts.append({'url': url, 'status': 200, 'file': key+'.html', 'bytes': len(data),
|
||||
'sha256': hashlib.sha256(data).hexdigest(), 'finished_at': '2026-11-09T00:00:00+00:00'})
|
||||
(root/'receipts.json').write_text(json.dumps({'receipts': receipts}))
|
||||
return root
|
||||
|
||||
def test_hidden_abstract_and_author_badges_are_not_title_or_authors(self):
|
||||
rows = aamas.parse_section(page('fullpapers', '1A_1'), 'fullpapers', '2026-11-09T00:00:00Z', {'sha256': 'proof'})
|
||||
row, = rows
|
||||
self.assertEqual(row['title'], 'A structured & auditable paper 1A_1')
|
||||
self.assertEqual(row['abstract'], 'Research with evidence. More detail.')
|
||||
self.assertEqual(row['authors'], ['Jane Doe', 'Bob Smith'])
|
||||
self.assertEqual(row['metadata']['authors_display'], 'Jane Doe, Bob Smith')
|
||||
self.assertEqual(row['url'], aamas.ROOT+'resources/fullpapers.html#1A_1')
|
||||
self.assertEqual(row['metadata']['paper_url'], aamas.ROOT+'papers/1A_1.pdf')
|
||||
self.assertEqual(row['metadata']['source_evidence'], {'sha256': 'proof'})
|
||||
|
||||
def test_publication_formats_and_session_topics_do_not_invent_submission_tracks(self):
|
||||
for section, (_, fmt, track, _) in aamas.SECTIONS.items():
|
||||
row, = aamas.parse_section(page(section, 'P_1', False), section, '2026-11-09T00:00:00Z', {})
|
||||
self.assertEqual((row['track'], row['metadata']['publication_format']), (track, fmt))
|
||||
self.assertIsNone(row['abstract'])
|
||||
self.assertEqual(row['status'], 'accepted')
|
||||
self.assertEqual(row['metadata']['decision_date_precision'], 'not_supplied')
|
||||
self.assertEqual(row['metadata']['session']['title'], 'Session 1A – Robotics I')
|
||||
self.assertIn({'namespace': 'aamas_proceedings:publication_format', 'value': fmt}, row['tags'])
|
||||
self.assertEqual(aamas.SECTIONS['fullpapers'][2], 'proceedings_unspecified')
|
||||
self.assertEqual(aamas.SECTIONS['extabstracts'][2], 'proceedings_unspecified')
|
||||
|
||||
def test_changed_edition_section_duplicate_id_or_pdf_link_is_rejected(self):
|
||||
html = page('fullpapers', '1A_1')
|
||||
for changed in [html.replace('AAMAS 2012', 'AAMAS 2025'),
|
||||
html.replace('<h1>Full Papers</h1>', '<h1>Rejected Papers</h1>'),
|
||||
html.replace('</body>', paper('1A_1')+'</body>'),
|
||||
html.replace('../papers/1A_1.pdf', 'https://example.org/1A_1.pdf'),
|
||||
html.replace('../papers/1A_1.pdf', '../papers/1A_2.pdf')]:
|
||||
with self.subTest(changed=changed):
|
||||
with self.assertRaises(ValueError):
|
||||
aamas.parse_section(changed, 'fullpapers', '2026-11-09T00:00:00Z', {})
|
||||
|
||||
def test_frozen_responses_require_checksums_success_and_safe_paths(self):
|
||||
for failure in ['checksum', 'status', 'path']:
|
||||
with self.subTest(failure=failure), tempfile.TemporaryDirectory() as temporary:
|
||||
root = self.snapshots(Path(temporary)/'source')
|
||||
path = root/'receipts.json'
|
||||
receipts = json.loads(path.read_text())
|
||||
item = receipts['receipts'][0]
|
||||
if failure == 'checksum': item['sha256'] = '0'*64
|
||||
elif failure == 'status': item['status'] = 404
|
||||
else: item['file'] = '../outside.html'
|
||||
path.write_text(json.dumps(receipts))
|
||||
with self.assertRaises(ValueError):
|
||||
aamas.read_pages(snapshots=root)
|
||||
|
||||
def test_reviewed_section_counts_and_year_are_required(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
source = self.snapshots(root/'source')
|
||||
with self.assertRaises(ValueError):
|
||||
aamas.collect(2012, root/'out', snapshots=source)
|
||||
self.assertFalse((root/'out/aamas-proceedings-2012.json').exists())
|
||||
with self.assertRaises(ValueError):
|
||||
aamas.collect(2025, root/'future', snapshots=source)
|
||||
self.assertFalse((root/'future').exists())
|
||||
|
||||
def test_ids_colliding_across_sections_do_not_produce_a_completed_collection(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
source = self.snapshots(root/'source')
|
||||
receipts_path = source/'receipts.json'
|
||||
receipts = json.loads(receipts_path.read_text())
|
||||
for item in receipts['receipts'][1:]:
|
||||
path = source/item['file']
|
||||
data = re.sub(r'P[0-9]_1', 'SAME_1', path.read_text()).encode()
|
||||
path.write_bytes(data)
|
||||
item.update(bytes=len(data), sha256=hashlib.sha256(data).hexdigest())
|
||||
receipts_path.write_text(json.dumps(receipts))
|
||||
small = {key: (*value[:3], 1) for key, value in aamas.SECTIONS.items()}
|
||||
with patch.dict(aamas.SECTIONS, small, clear=True), self.assertRaises(ValueError):
|
||||
aamas.collect(2012, root/'out', snapshots=source)
|
||||
self.assertFalse((root/'out/aamas-proceedings-2012.json').exists())
|
||||
|
||||
def test_collection_and_native_ingestion_preserve_formats_evidence_and_missing_abstracts(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
source = self.snapshots(root/'source')
|
||||
small = {key: (*value[:3], 1) for key, value in aamas.SECTIONS.items()}
|
||||
with patch.dict(aamas.SECTIONS, small, clear=True):
|
||||
result = aamas.collect(2012, root/'out', snapshots=source)
|
||||
self.assertEqual(result['records'], 4)
|
||||
self.assertEqual(len(result['files']), 3)
|
||||
self.assertFalse(result['database_written'])
|
||||
self.assertFalse(result['pdfs_downloaded'])
|
||||
db = root/'sample.sqlite'
|
||||
for manifest in result['files']:
|
||||
path = Path(manifest['file'])
|
||||
self.assertEqual(hashlib.sha256(path.read_bytes()).hexdigest(), manifest['file_sha256'])
|
||||
for raw in path.read_text().splitlines():
|
||||
record = json.loads(raw)
|
||||
evidence = record['metadata']['source_evidence']
|
||||
self.assertEqual(hashlib.sha256((root/'out'/evidence['file']).read_bytes()).hexdigest(), evidence['sha256'])
|
||||
ingest_file(db, path)
|
||||
with sqlite3.connect(db) as connection:
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM papers').fetchone()[0], 4)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM submissions').fetchone()[0], 4)
|
||||
self.assertEqual(connection.execute("SELECT count(*) FROM decisions WHERE status='accepted'").fetchone()[0], 4)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM papers WHERE abstract IS NULL').fetchone()[0], 1)
|
||||
metadata = [json.loads(r[0]) for r in connection.execute('SELECT metadata_json FROM source_records')]
|
||||
self.assertEqual({r['publication_format'] for r in metadata}, {'full_paper','extended_abstract','demo','doctoral_abstract'})
|
||||
namespaces = {r[0] for r in connection.execute('SELECT DISTINCT namespace FROM tags')}
|
||||
self.assertEqual(namespaces, {'aamas_proceedings:publication_format', 'aamas_proceedings:session'})
|
||||
self.assertEqual(connection.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Cross-source publisher identities must preserve attempts and decision history."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from paper_atlas.db.submissions import identities_for_row
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
|
||||
DOI = '10.1234/aamas.identity.example'
|
||||
|
||||
|
||||
def row(source, external_id, *, venue='AAMAS', year=2012, track='demo', status='accepted', identifiers=None):
|
||||
return {'source': source, 'external_id': external_id,
|
||||
'url': 'https://example.org/'+source+'/'+external_id,
|
||||
'title': 'A shared publication with independently recorded evidence',
|
||||
'abstract': 'An independently recorded source abstract.',
|
||||
'authors': ['Jane Doe', 'Bob Smith'],
|
||||
'identifiers': identifiers if identifiers is not None else {'doi': DOI},
|
||||
'venue': venue, 'year': year, 'track': track, 'status': status,
|
||||
'observed_at': '2026-11-09T00:00:00Z',
|
||||
'raw_decision': source+' observed '+status,
|
||||
'metadata': {'original_source': source}, 'tags': []}
|
||||
|
||||
|
||||
def ingest(db, root, record):
|
||||
path = root/(record['source']+'-'+record['external_id'].replace('/', '-')+'.jsonl')
|
||||
path.write_text(json.dumps(record)+'\n')
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps({
|
||||
'source': record['source'], 'parser_version': 1, 'venue': record['venue'],
|
||||
'year': record['year'], 'track': record['track'], 'count': 1,
|
||||
'list_url': 'https://example.org/'+record['source'], 'status': 'partial',
|
||||
'file_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
'missing_abstract_count': 0}))
|
||||
ingest_file(db, path)
|
||||
|
||||
|
||||
class AamasPublicationIdentityTests(unittest.TestCase):
|
||||
def test_only_accepted_known_publisher_rows_supply_publication_doi_identity(self):
|
||||
record = row('aamas_proceedings', '2012/D1_1', identifiers={'doi': 'https://doi.org/'+DOI.upper()})
|
||||
self.assertEqual(identities_for_row(record), [('publication_doi', DOI)])
|
||||
for status in ['rejected', 'desk_rejected', 'withdrawn', 'under_review', 'unknown']:
|
||||
with self.subTest(status=status):
|
||||
self.assertEqual(identities_for_row({**record, 'status': status}), [])
|
||||
self.assertEqual(identities_for_row({**record, 'source': 'unverified_source'}), [])
|
||||
|
||||
def test_same_edition_doi_joins_sources_in_both_orders_without_duplicate_attempts(self):
|
||||
publisher = row('aamas_proceedings', '2012/D1_1')
|
||||
bibliography = row('dblp', 'conf/atal/Example12')
|
||||
for order in [(publisher, bibliography), (bibliography, publisher)]:
|
||||
with self.subTest(first=order[0]['source']), tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
db = root/'sample.sqlite'
|
||||
for record in order:
|
||||
ingest(db, root, record)
|
||||
ingest(db, root, publisher)
|
||||
with sqlite3.connect(db) as connection:
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM papers').fetchone()[0], 1)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM source_records').fetchone()[0], 2)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM submissions').fetchone()[0], 1)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM submission_sources').fetchone()[0], 2)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM decision_sources').fetchone()[0], 2)
|
||||
self.assertEqual(connection.execute('SELECT scheme,value FROM submission_identifiers').fetchall(), [('publication_doi', DOI)])
|
||||
metadata = [json.loads(r[0]) for r in connection.execute('SELECT metadata_json FROM source_records')]
|
||||
self.assertEqual({r['original_source'] for r in metadata}, {'aamas_proceedings', 'dblp'})
|
||||
self.assertEqual(connection.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
|
||||
def test_cross_venue_rejection_and_distinct_tracks_remain_separate_attempts(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
db = root/'sample.sqlite'
|
||||
rejected = row('openreview_snapshot', 'original-forum', venue='AAAI', year=2011,
|
||||
track='main', status='rejected', identifiers={'arxiv': '1101.00001', 'openreview': 'original-forum'})
|
||||
accepted = row('aamas_proceedings', '2012/D1_1', identifiers={'arxiv': '1101.00001', 'doi': DOI})
|
||||
another_track = row('aamas_proceedings', '2012/F1_1', track='proceedings_unspecified', identifiers={'doi': DOI})
|
||||
for record in [rejected, accepted, another_track]:
|
||||
ingest(db, root, record)
|
||||
with sqlite3.connect(db) as connection:
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM papers').fetchone()[0], 1)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM submissions').fetchone()[0], 3)
|
||||
history = connection.execute('''SELECT v.acronym,e.year,e.track,d.status FROM decisions d
|
||||
JOIN submissions s ON s.id=d.submission_id JOIN editions e ON e.id=s.edition_id
|
||||
JOIN venues v ON v.id=e.venue_id ORDER BY v.acronym,e.track''').fetchall()
|
||||
self.assertEqual(set(history), {('AAAI',2011,'main','rejected'),('AAMAS',2012,'demo','accepted'),('AAMAS',2012,'proceedings_unspecified','accepted')})
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM decision_sources').fetchone()[0], 3)
|
||||
self.assertEqual(connection.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Regression coverage for primary series once named Workshop, with strict scope boundaries."""
|
||||
from contextlib import closing
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from paper_atlas.collectors.dblp_records import normalize_row
|
||||
from paper_atlas.collectors.dblp_scope import scope
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
ICML_HISTORY = 'https://icml.cc/Conferences/2008/past_icmls.shtml.html'
|
||||
IROS_HISTORY = 'https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/'
|
||||
|
||||
|
||||
class HistoricalPrimarySeriesTests(unittest.TestCase):
|
||||
def example(self, family, year, title, book=None, key=None):
|
||||
key = key or f'conf/{family}/{year}'
|
||||
book = book or ('ML' if family == 'icml' else 'IROS')
|
||||
row = {'record_type':'inproceedings', 'key':f'conf/{family}/Paper{year}', 'publtype':None,
|
||||
'fields':{'title':['An independently identified historical research paper'],
|
||||
'author':['Jane Doe'], 'booktitle':[book], 'year':[str(year)],
|
||||
'crossref':[key], 'ee':[f'https://doi.org/10.1234/{family}.{year}']}}
|
||||
volume = {'record_type':'proceedings', 'key':key,
|
||||
'fields':{'title':[title], 'year':[str(year)]}}
|
||||
return row, volume
|
||||
|
||||
def test_icml_workshop_primary_editions_are_not_silently_filtered(self):
|
||||
examples = {
|
||||
1989:'Proceedings of the Sixth International Workshop on Machine Learning (ML 1989), Cornell University, Ithaca, New York, USA, June 26-27, 1989',
|
||||
1991:'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA',
|
||||
1992:'Proceedings of the Ninth International Workshop on Machine Learning (ML 1992), Aberdeen, Scotland, UK, July 1-3, 1992',
|
||||
}
|
||||
for year, title in examples.items():
|
||||
with self.subTest(year=year):
|
||||
row, volume = self.example('icml', year, title)
|
||||
record, reason = normalize_row(row, {volume['key']:volume}, {'ICML'})
|
||||
self.assertIsNone(reason)
|
||||
self.assertEqual((record['venue'],record['year'],record['track'],record['status']), ('ICML',year,'main','accepted'))
|
||||
self.assertEqual(record['metadata']['proceedings_title'], title)
|
||||
self.assertEqual(record['metadata']['track_scope_evidence_url'], ICML_HISTORY)
|
||||
self.assertEqual(record['metadata']['bibliographic_year'], year)
|
||||
|
||||
def test_iros_first_three_editions_are_primary_series(self):
|
||||
examples = {
|
||||
1988:"IEEE International Workshop on Intelligent Robots and Systems '88, Proceedings. IROS 1988, Tokyo, Japan, October 31 - November 2, 1988",
|
||||
1989:"IEEE International Workshop on Intelligent Robots and Systems '89, Proceedings. IROS 1989, September 4-6, 1989, Tsukuba, Japan",
|
||||
1990:"IEEE International Workshop on Intelligent Robots and Systems '90, Towards a New Frontier of Applications, Proceedings. IROS 1990, July 3-6, 1990, Ibaraki, Japan",
|
||||
}
|
||||
for year, title in examples.items():
|
||||
with self.subTest(year=year):
|
||||
row, volume = self.example('iros', year, title)
|
||||
record, reason = normalize_row(row, {volume['key']:volume}, {'IROS'})
|
||||
self.assertIsNone(reason)
|
||||
self.assertEqual((record['venue'],record['year'],record['track']), ('IROS',year,'main'))
|
||||
self.assertEqual(record['metadata']['proceedings_title'], title)
|
||||
self.assertEqual(record['metadata']['track_scope_evidence_url'], IROS_HISTORY)
|
||||
|
||||
def test_main_named_icml_predecessors_also_carry_primary_scope_provenance(self):
|
||||
examples = {
|
||||
1988:'Machine Learning, Proceedings of the Fifth International Conference on Machine Learning, Ann Arbor, Michigan, USA, June 12-14, 1988',
|
||||
1990:'Machine Learning, Proceedings of the Seventh International Conference on Machine Learning, Austin, Texas, USA, June 21-23, 1990',
|
||||
}
|
||||
for year, title in examples.items():
|
||||
row, volume = self.example('icml', year, title)
|
||||
edition, reason = scope(row, volume)
|
||||
self.assertIsNone(reason)
|
||||
self.assertEqual(edition['track'], 'main')
|
||||
self.assertEqual(edition['track_scope_evidence_url'], ICML_HISTORY)
|
||||
|
||||
def test_wrong_keys_titles_suffixes_and_unknown_years_keep_workshop_boundary(self):
|
||||
title = 'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA'
|
||||
for year, changed, book, key in [
|
||||
(1991,title,'ML','conf/icml/1991w'),
|
||||
(1991,title,'ML (Workshops)','conf/icml/1991'),
|
||||
(1991,title.replace('Eighth','Ninth'),'ML','conf/icml/1991'),
|
||||
(1991,title + ', Affiliated Workshop','ML','conf/icml/1991'),
|
||||
(1987,'Machine Learning Workshop 1987, Irvine','ML','conf/icml/1987'),
|
||||
]:
|
||||
with self.subTest(year=year, book=book, key=key, title=changed):
|
||||
row, volume = self.example('icml',year,changed,book,key)
|
||||
record, reason = normalize_row(row,{volume['key']:volume},{'ICML'})
|
||||
self.assertIsNone(record)
|
||||
self.assertEqual(reason,'separate_workshop')
|
||||
self.assertNotIn('track_scope_evidence_url',scope(row,volume)[0])
|
||||
title = "IEEE International Workshop on Intelligent Robots and Systems '88, Proceedings. IROS 1988, Tokyo, Japan, October 31 - November 2, 1988"
|
||||
for changed, book, key in [
|
||||
(title,'IROS','conf/iros/1988w'),
|
||||
(title,'IROS (Workshops)','conf/iros/1988'),
|
||||
(title.replace('Tokyo','Another city'),'IROS','conf/iros/1988'),
|
||||
]:
|
||||
row, volume = self.example('iros',1988,changed,book,key)
|
||||
self.assertEqual(normalize_row(row,{volume['key']:volume},{'IROS'})[1],'separate_workshop')
|
||||
|
||||
def test_primary_scope_proof_survives_native_file_ingestion(self):
|
||||
title = 'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA'
|
||||
row, volume = self.example('icml',1991,title)
|
||||
record, reason = normalize_row(row,{volume['key']:volume},{'ICML'})
|
||||
self.assertIsNone(reason)
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root=Path(temporary);path=root/'records.jsonl';path.write_text(json.dumps(record)+'\n')
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps({'source':'dblp','count':1,'list_url':ICML_HISTORY,'file_sha256':hashlib.sha256(path.read_bytes()).hexdigest()}))
|
||||
ingest_file(root/'test.sqlite',path)
|
||||
with closing(sqlite3.connect(root/'test.sqlite')) as connection:
|
||||
self.assertEqual(connection.execute('SELECT track FROM editions').fetchone()[0],'main')
|
||||
self.assertEqual(connection.execute('SELECT status FROM decisions').fetchone()[0],'accepted')
|
||||
metadata=json.loads(connection.execute('SELECT metadata_json FROM source_records').fetchone()[0])
|
||||
self.assertEqual(metadata['proceedings_title'],title)
|
||||
self.assertEqual(metadata['track_scope_evidence_url'],ICML_HISTORY)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,148 @@
|
||||
"""Historical accepted lists, repeated presentations and preserved work histories."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from paper_atlas.collectors import iclr_archive as archive
|
||||
from paper_atlas.db.store import Store
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
OBSERVED = '2026-11-09T00:00:00Z'
|
||||
|
||||
|
||||
def entry(identifier, title='A shared research paper', display='Jane Doe and Bob Smith'):
|
||||
return f'<li><div><a href="http://arxiv.org/abs/{identifier}">{title}</a>, {display}</div></li>'
|
||||
|
||||
|
||||
def page2015():
|
||||
return ('<h3>Main Conference - Oral Presentations</h3><ol>'+entry('1412.0001v2')+'</ol>'
|
||||
'<h3>Main Conference - Poster Presentations</h3><ol>'+entry('1412.0001v2')+'</ol>'
|
||||
'<h3>Workshop Papers</h3><ol>'+entry('1412.0002', 'A workshop paper')+'</ol>')
|
||||
|
||||
|
||||
class IclrArchiveTests(unittest.TestCase):
|
||||
def frozen(self, root, year, html):
|
||||
root.mkdir()
|
||||
data = html.encode()
|
||||
(root/'page.html').write_bytes(data)
|
||||
receipt = {'url': archive.PAGES[year][0], 'file': 'page.html', 'status': 200,
|
||||
'bytes': len(data), 'sha256': hashlib.sha256(data).hexdigest(), 'finished_at': OBSERVED}
|
||||
(root/'receipts.json').write_text(json.dumps({'receipts': [receipt]}))
|
||||
return root
|
||||
|
||||
def test_oral_poster_repeat_is_one_attempt_with_all_source_variants(self):
|
||||
rows = archive.parse_page(page2015(), 2015, {'observed_at': OBSERVED})
|
||||
main, workshop = rows
|
||||
self.assertEqual((main['track'], workshop['track']), ('main', 'workshop'))
|
||||
self.assertEqual(main['identifiers'], {'arxiv': '1412.0001'})
|
||||
self.assertEqual(main['external_id'], '2015/main/1412.0001')
|
||||
self.assertEqual(main['metadata']['presentations'], ['oral', 'poster'])
|
||||
self.assertEqual(len(main['metadata']['source_variants']), 2)
|
||||
self.assertEqual(main['metadata']['source_variants'][0]['version'], 'v2')
|
||||
self.assertEqual(main['authors'], ['Jane Doe', 'Bob Smith'])
|
||||
self.assertTrue(all(r['status'] == 'accepted' and r['abstract'] is None for r in rows))
|
||||
|
||||
def test_2014_author_span_outside_paragraph_and_trailing_separator(self):
|
||||
html = ('<h3>Conference Proceedings</h3><p>conference papers accepted</p>'
|
||||
'<div><b><a href="http://arxiv.org/abs/1312.0001">A <i>research</i> title</a></b></div>'
|
||||
'<p><span style="font-style: italic">Jane Doe; </span>'
|
||||
'<span style="font-style:italic">Bob Smith; Carol Wu;</span></p>'
|
||||
'<div><a href="http://arxiv.org/abs/1312.0002">Another title</a>'
|
||||
'<div><i><span>First Author; </span><span>Second Author</span></i></div></div>')
|
||||
row, unwrapped = archive.parse_page(html, 2014, {'observed_at': OBSERVED})
|
||||
self.assertEqual(row['title'], 'A research title')
|
||||
self.assertEqual(row['authors'], ['Jane Doe', 'Bob Smith', 'Carol Wu'])
|
||||
self.assertEqual(row['metadata']['source_variants'][0]['authors_display'], 'Jane Doe; Bob Smith; Carol Wu;')
|
||||
self.assertEqual(row['track'], 'main')
|
||||
self.assertEqual(unwrapped['authors'], ['First Author', 'Second Author'])
|
||||
|
||||
def test_same_arxiv_in_different_tracks_keeps_separate_source_attempts(self):
|
||||
html = page2015().replace('1412.0002', '1412.0001v3')
|
||||
rows = archive.parse_page(html, 2015, {'observed_at': OBSERVED})
|
||||
self.assertEqual(len(rows), 2)
|
||||
self.assertEqual({r['external_id'] for r in rows}, {'2015/main/1412.0001', '2015/workshop/1412.0001'})
|
||||
self.assertEqual(rows[1]['metadata']['source_variants'][0]['version'], 'v3')
|
||||
|
||||
def test_2016_code_links_and_breaks_do_not_become_authors(self):
|
||||
html = ('<h1>Accepted Papers (Conference Track)</h1><h3>Oral Presentations</h3><ol>'
|
||||
'<li><a href="http://arxiv.org/abs/1511.00001">A title</a>'
|
||||
'<a href="https://example.org/code">[code]</a><br>Jane Doe, Bob Smith</li></ol>'
|
||||
'<h3>Poster Presentations</h3>')
|
||||
row, = archive.parse_page(html, 2016, {'observed_at': OBSERVED})
|
||||
self.assertEqual(row['authors'], ['Jane Doe', 'Bob Smith'])
|
||||
self.assertEqual(row['metadata']['presentations'], ['oral'])
|
||||
|
||||
def test_conflicting_repeat_unknown_sections_and_unparsed_links_fail(self):
|
||||
variants = [page2015().replace('Poster Presentations', 'Other Presentations'),
|
||||
page2015().replace('</ol>', '</ol><a href="http://arxiv.org/abs/1412.0003">stray</a>', 1),
|
||||
page2015().replace('Jane Doe and Bob Smith', 'Different Author', 1)]
|
||||
for html in variants:
|
||||
with self.subTest(html=html), self.assertRaises(ValueError):
|
||||
archive.parse_page(html, 2015, {'observed_at': OBSERVED})
|
||||
|
||||
def test_arxiv_versions_are_preserved_and_unsafe_links_rejected(self):
|
||||
self.assertEqual(archive.arxiv_link('https://arxiv.org/pdf/1511.00001v3.pdf'), ('1511.00001', 'v3'))
|
||||
self.assertIsNone(archive.arxiv_link('https://arxiv.org.example.org/abs/1511.00001'))
|
||||
for url in ['https://arxiv.org/abs/1511.00001?other=1', 'ftp://arxiv.org/abs/1511.00001',
|
||||
'https://user@arxiv.org/abs/1511.00001', 'https://arxiv.org/abs/not-a-paper']:
|
||||
with self.subTest(url=url), self.assertRaises(ValueError):
|
||||
archive.arxiv_link(url)
|
||||
|
||||
def test_receipt_hash_status_path_and_symlink_cannot_escape_snapshot(self):
|
||||
for failure in ['hash', 'status', 'path', 'symlink']:
|
||||
with self.subTest(failure=failure), tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
source = self.frozen(root/'source', 2015, page2015())
|
||||
file = source/'receipts.json'
|
||||
data = json.loads(file.read_text())
|
||||
receipt = data['receipts'][0]
|
||||
if failure == 'hash': receipt['sha256'] = '0'*64
|
||||
elif failure == 'status': receipt['status'] = 404
|
||||
elif failure == 'path': receipt['file'] = '../outside.html'
|
||||
else:
|
||||
(root/'outside.html').write_text(page2015())
|
||||
(source/'page.html').unlink()
|
||||
(source/'page.html').symlink_to(root/'outside.html')
|
||||
file.write_text(json.dumps(data))
|
||||
with self.assertRaises(ValueError):
|
||||
archive.read_page(2015, snapshots=source)
|
||||
|
||||
def test_changed_inventory_leaves_no_completed_output(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
source = self.frozen(root/'source', 2015, page2015())
|
||||
with self.assertRaises(ValueError):
|
||||
archive.collect(2015, root/'out', snapshots=source)
|
||||
self.assertFalse((root/'out/iclr-archive-2015.json').exists())
|
||||
|
||||
def test_native_import_keeps_rejected_to_accepted_history_and_abstract(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
source = self.frozen(root/'source', 2015, page2015())
|
||||
with patch.dict(archive.PAGES, {2015: (archive.PAGES[2015][0], {'main': 1, 'workshop': 1})}):
|
||||
result = archive.collect(2015, root/'out', snapshots=source)
|
||||
with self.assertRaises(FileExistsError):
|
||||
archive.collect(2015, root/'out', snapshots=source)
|
||||
db = root/'sample.sqlite'
|
||||
store = Store(db)
|
||||
pid, rid = store.record('arxiv', '1412.0001', 'https://arxiv.org/abs/1412.0001',
|
||||
'A shared research paper', 'Keep this canonical abstract.',
|
||||
['Jane Doe', 'Bob Smith'], {'arxiv': '1412.0001'}, {})
|
||||
store.decision(pid, rid, 'Earlier Conference', 2014, 'main', 'original-attempt',
|
||||
'https://example.org/rejected', 'rejected', 'explicit earlier decision', observed_at=OBSERVED)
|
||||
store.conn.commit()
|
||||
store.conn.close()
|
||||
for manifest in result['files']:
|
||||
ingest_file(db, manifest['file'])
|
||||
ingest_file(db, manifest['file'])
|
||||
with sqlite3.connect(db) as c:
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0], 2)
|
||||
self.assertEqual(c.execute('SELECT abstract FROM papers WHERE id=?', (pid,)).fetchone()[0], 'Keep this canonical abstract.')
|
||||
history = c.execute('SELECT v.acronym,e.year,d.status FROM decisions d JOIN submissions s ON s.id=d.submission_id JOIN editions e ON e.id=s.edition_id JOIN venues v ON v.id=e.venue_id WHERE s.paper_id=? ORDER BY e.year', (pid,)).fetchall()
|
||||
self.assertEqual(history, [('Earlier Conference', 2014, 'rejected'), ('ICLR', 2015, 'accepted')])
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM submissions').fetchone()[0], 3)
|
||||
self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
@@ -0,0 +1,155 @@
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from paper_atlas.collectors.www_archive import (ROOT, SPRING, FALL, Sources, collect, decode,
|
||||
early_review, spring_rows)
|
||||
|
||||
|
||||
class ArchiveTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
t=tempfile.TemporaryDirectory();self.addCleanup(t.cleanup);self.root=Path(t.name);self.cache=self.root/'cache';self.cache.mkdir()
|
||||
self.pages={ROOT+'www1/PrelimProcs.html':'''<h1>Formal presentations</h1><ol><li><em>Keywords:</em> BUSINESS TRACK?<br><em>Title:</em><b> First contribution</b><br><em>Author:</em>Jane Doe <jane@example.org><br><li><em>Title:</em><b>Unlinked contribution</b><br><em>Author:</em>John Roe<br></ol>''',
|
||||
FALL:'<h1>Conference Tracks</h1><ol><li><a href="Agents.html">Agents</a><li><a href="Tutor.html">Tutorials</a></ol>',
|
||||
ROOT+'www2/Proceedings/Agents.html':'<h3>Panel</h3><dl><dt><a href="a.html">Linked talk</a><dd>Jane Doe<dt>Unlinked talk</a><dd>John Roe</dl>',
|
||||
ROOT+'www2/Proceedings/Tutor.html':'<h1>Tutorials</h1><dl><dt>Training session<dd>Teacher</dl>'}
|
||||
for track in ['papers','posters']:
|
||||
self.pages[SPRING+track+'/author.html']='<b>Doe, Jane</b><ul><li><a href="01/paper.html">Same contribution</a></li></ul><b>Roe, John</b><a href="01/paper.html">Same contribution</a>'
|
||||
self.pages[SPRING+track+'/abstract.html']='''<h1>Abstracts</h1><a href="#one">Same contribution</a><h3><a href="01/paper.html">Same contribution</a></h3><i>Author: Jane Doe, John Roe<br>Institute</i><p>Actual abstract with <i>inline emphasis</i> and a separator\u2028 preserved.</p><hr><h2>Another session</h2>'''
|
||||
self.pages[SPRING+'papers/session.html']='<a href="01/paper.html">Same contribution</a>'
|
||||
|
||||
def sources(self):
|
||||
rows=[]
|
||||
for i,(url,html) in enumerate(sorted(self.pages.items())):
|
||||
raw=html.encode()if isinstance(html,str)else html;name='source-'+str(i)+'.html';(self.cache/name).write_bytes(raw)
|
||||
rows.append(dict(url=url,status=200,file=name,bytes=len(raw),sha256=hashlib.sha256(raw).hexdigest()))
|
||||
p=self.cache/'manifest.json';p.write_text(json.dumps({'sources':rows}));sha=hashlib.sha256(p.read_bytes()).hexdigest()
|
||||
return Sources(self.cache,sha),sha
|
||||
|
||||
def test_offline_batch_separates_tracks_review_and_importable_rows(self):
|
||||
_,sha=self.sources();out=self.root/'batch';report=collect(self.cache,sha,out)
|
||||
self.assertEqual(report['ready'],2);self.assertEqual(report['held'],5)
|
||||
queue=json.loads((out/'import-queue.json').read_text())['files'];self.assertEqual(len(queue),2)
|
||||
for name in queue:
|
||||
p=Path(name);rows=[json.loads(l)for l in p.read_text().split('\n') if l];self.assertEqual(len(rows),1)
|
||||
row=rows[0];self.assertEqual(row['event_key'],'www-1995-spring');self.assertEqual(row['authors'],['Jane Doe','John Roe'])
|
||||
self.assertIn(row['track'],['main','poster']);self.assertIn('inline emphasis',row['abstract']);self.assertNotIn('Another session',row['abstract'])
|
||||
self.assertIn('\u2028',row['abstract']);manifest=json.loads(p.with_suffix('.manifest.json').read_text());self.assertEqual(manifest['file_sha256'],hashlib.sha256(p.read_bytes()).hexdigest());self.assertEqual(manifest['status'],'partial')
|
||||
held=[json.loads(l)for l in(out/'review-candidates.jsonl').read_text().split('\n') if l]
|
||||
self.assertTrue(all('status'not in r and 'source_evidence'in r for r in held))
|
||||
self.assertFalse(any('review-candidates'in p for p in queue))
|
||||
|
||||
def test_partial_manifest_preserves_full_listing_size_separately_from_ready_count(self):
|
||||
self.pages[SPRING+'posters/abstract.html']+='<h3><a href="02/other.html">Unlinked poster</a></h3><i>Author: Other Person</i><p>Actual abstract</p>'
|
||||
_,sha=self.sources();out=self.root/'batch';collect(self.cache,sha,out)
|
||||
manifest=json.loads((out/'www-1995-spring-poster.manifest.json').read_text())
|
||||
self.assertEqual((manifest['count'],manifest['source_listing_count'],manifest['held_entry_count']),(1,2,1))
|
||||
self.assertNotIn('coverage_previous_count',manifest) # Live parent must be independently preflighted.
|
||||
|
||||
def test_malformed_href_requires_exact_id_and_two_clean_index_anchors(self):
|
||||
self.pages[SPRING+'papers/abstract.html']=self.pages[SPRING+'papers/abstract.html'].replace('01/paper.html','01/ paper.html')
|
||||
sources,_=self.sources();rows,held,_=spring_rows(sources,'papers');self.assertEqual(held,[])
|
||||
self.assertEqual(rows[0]['url'],SPRING+'papers/01/paper.html');self.assertTrue(rows[0]['metadata']['url_corroborated_by_author_and_session'])
|
||||
self.assertIn(' ',rows[0]['metadata']['abstract_href_raw'])
|
||||
|
||||
def test_main_index_disagreement_fails_without_publishing_batch(self):
|
||||
self.pages[SPRING+'papers/session.html']='<a href="02/other.html">Other work</a>';_,sha=self.sources();out=self.root/'batch'
|
||||
with self.assertRaisesRegex(ValueError,'ID sets disagree'):collect(self.cache,sha,out)
|
||||
self.assertFalse(out.exists())
|
||||
|
||||
def test_same_entry_id_with_conflicting_clean_url_is_rejected(self):
|
||||
self.pages[SPRING+'papers/session.html']='<a href="01/other.html">Same contribution</a>';sources,_=self.sources()
|
||||
with self.assertRaisesRegex(ValueError,'URLs disagree'):spring_rows(sources,'papers')
|
||||
|
||||
def test_poster_without_linked_author_entry_is_held_even_with_abstract(self):
|
||||
self.pages[SPRING+'posters/author.html']='<b>Doe, Jane</b><a href="#">Same contribution</a>';sources,_=self.sources();rows,held,counts=spring_rows(sources,'posters')
|
||||
self.assertEqual(rows,[]);self.assertEqual(counts['abstract_entries'],1);self.assertEqual(held[0]['reason'],'abstract_entry_missing_linked_author_index')
|
||||
|
||||
def test_unlinked_and_panel_entries_never_become_acceptance_or_rejection(self):
|
||||
sources,_=self.sources();held,counts=early_review(sources);self.assertEqual(counts,{'first_conference_fields':2,'fall_categories':2,'fall_program_entries':3})
|
||||
self.assertTrue(any(r['candidate']['title']=='Unlinked talk'for r in held));self.assertTrue(all('status'not in r for r in held))
|
||||
self.assertEqual(held[0]['candidate']['authors_raw'],['Jane Doe'])
|
||||
|
||||
def test_title_disagreement_goes_to_review_with_source_claims(self):
|
||||
self.pages[SPRING+'papers/abstract.html']=self.pages[SPRING+'papers/abstract.html'].replace('Same contribution','Different contribution');sources,_=self.sources();rows,held,_=spring_rows(sources,'papers')
|
||||
self.assertEqual(rows,[]);self.assertEqual(held[0]['reason'],'abstract_and_author_index_titles_disagree')
|
||||
|
||||
def test_cache_tampering_and_manifest_hash_mismatch_fail_closed(self):
|
||||
_,sha=self.sources()
|
||||
with self.assertRaisesRegex(ValueError,'manifest checksum'):Sources(self.cache,'0'*64)
|
||||
p=next(self.cache.glob('source-*.html'));p.write_bytes(p.read_bytes()+b'x')
|
||||
with self.assertRaisesRegex(ValueError,'checksum or size'):Sources(self.cache,sha)
|
||||
|
||||
def test_cache_cannot_read_paths_outside_snapshot(self):
|
||||
_,sha=self.sources();p=self.cache/'manifest.json';d=json.loads(p.read_text());d['sources'][0]['file']='../outside.html';p.write_text(json.dumps(d))
|
||||
with self.assertRaisesRegex(ValueError,'plain files'):Sources(self.cache,hashlib.sha256(p.read_bytes()).hexdigest())
|
||||
|
||||
def test_keywords_only_is_missing_abstract_not_invented_summary(self):
|
||||
self.pages[SPRING+'papers/abstract.html']='<h3><a href="01/paper.html">Same contribution</a></h3><i>Author: Jane Doe</i><p>Keywords: networking, hypertext</p>';sources,_=self.sources();rows,held,_=spring_rows(sources,'papers')
|
||||
self.assertEqual(rows,[]);self.assertEqual(held[0]['reason'],'missing_abstract')
|
||||
|
||||
def test_old_encoding_and_duplicate_abstract_identity(self):
|
||||
self.assertEqual(decode(b'M\xfcller')[0],'Müller')
|
||||
self.pages[SPRING+'papers/abstract.html']*=2;sources,_=self.sources()
|
||||
with self.assertRaisesRegex(ValueError,'Duplicate abstract'):spring_rows(sources,'papers')
|
||||
|
||||
def test_existing_output_is_never_overwritten(self):
|
||||
_,sha=self.sources();out=self.root/'batch';out.mkdir();(out/'keep').write_text('original')
|
||||
with self.assertRaisesRegex(ValueError,'overwrite'):collect(self.cache,sha,out)
|
||||
self.assertEqual((out/'keep').read_text(),'original')
|
||||
|
||||
def review(self,*,authors=None,author_source='canonical_byline',missing='',page=None):
|
||||
url=SPRING+'posters/01/paper.html'
|
||||
self.pages[url]=page or '<h1>Canonical full title</h1><p>Jane Doe, John Roe</p><p>Keywords: web</p><h2>Abstract</h2><p>Body</p>'
|
||||
_,sha=self.sources();raw=self.pages[url].encode()if isinstance(self.pages[url],str)else self.pages[url]
|
||||
row=dict(entry_id='01',individual_url=url,individual_sha256=hashlib.sha256(raw).hexdigest(),
|
||||
title='Canonical full title',authors=authors or ['Jane Doe','John Roe'],author_source=author_source,
|
||||
reason='Reviewed exact canonical entry and its explicit authorship; retain original title variants',missing_abstract_reason=missing)
|
||||
p=self.root/'review.json';p.write_text(json.dumps({'format':1,'source_cache_sha256':sha,'records':[row]}));return sha,p,hashlib.sha256(p.read_bytes()).hexdigest()
|
||||
|
||||
def test_reviewed_variant_keeps_primary_title_and_all_old_title_claims(self):
|
||||
sha,p,h=self.review();out=self.root/'reviewed';r=collect(self.cache,sha,out,reviewed=p,reviewed_sha256=h)
|
||||
row=json.loads((out/'www-1995-spring-poster.jsonl').read_text());self.assertEqual(row['title'],'Canonical full title')
|
||||
self.assertEqual(row['metadata']['abstract_heading_title_raw'],'Same contribution');self.assertEqual(row['metadata']['author_index_titles_raw'],['Same contribution'])
|
||||
self.assertEqual(len(row['metadata']['source_evidence']),3);self.assertEqual(r['reviewed_posters_sha256'],h)
|
||||
self.assertEqual(row['external_id'],'www3:posters:01');self.assertEqual(row['status'],'accepted')
|
||||
|
||||
def test_canonical_byline_can_resolve_unlinked_author_index_without_guessing(self):
|
||||
self.pages[SPRING+'posters/author.html']='<b>Doe, Jane</b><a href="#">Broken index link</a>'
|
||||
sha,p,h=self.review();out=self.root/'reviewed';collect(self.cache,sha,out,reviewed=p,reviewed_sha256=h)
|
||||
row=json.loads((out/'www-1995-spring-poster.jsonl').read_text());self.assertEqual(row['authors'],['Jane Doe','John Roe'])
|
||||
self.assertEqual(row['metadata']['author_index_raw'],[]);self.assertEqual(row['metadata']['author_order'],'canonical_byline')
|
||||
|
||||
def test_reviewed_missing_abstract_is_explicit_partial_not_body_summary(self):
|
||||
self.pages[SPRING+'posters/abstract.html']='<h3><a href="01/paper.html">Same contribution</a></h3><i>Author: Jane Doe</i><p>Keywords: web</p>'
|
||||
sha,p,h=self.review(missing='Neither listing nor canonical page supplies an explicit abstract')
|
||||
out=self.root/'reviewed';collect(self.cache,sha,out,reviewed=p,reviewed_sha256=h)
|
||||
row=json.loads((out/'www-1995-spring-poster.jsonl').read_text());m=json.loads((out/'www-1995-spring-poster.manifest.json').read_text())
|
||||
self.assertIsNone(row['abstract']);self.assertEqual((m['missing_abstract_count'],m['status']),(1,'partial'))
|
||||
sha,p,h=self.review()
|
||||
with self.assertRaisesRegex(ValueError,'Missing abstract'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
|
||||
self.assertFalse((self.root/'bad').exists())
|
||||
|
||||
def test_reviewed_names_must_be_in_byline_region_not_citations_or_contact_guess(self):
|
||||
sha,p,h=self.review(authors=['Unrelated Person'],page='<h1>Canonical full title</h1><p>Jane Doe</p><h2>Abstract</h2><p>Unrelated Person cited here</p>')
|
||||
with self.assertRaisesRegex(ValueError,'byline region'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
|
||||
sha,p,h=self.review(author_source='author_index',authors=['Contact Person'])
|
||||
with self.assertRaisesRegex(ValueError,'author-index'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
|
||||
|
||||
def test_review_hash_cache_binding_and_canonical_h1_are_all_required(self):
|
||||
sha,p,h=self.review()
|
||||
with self.assertRaisesRegex(ValueError,'checksum'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256='0'*64)
|
||||
d=json.loads(p.read_text());d['source_cache_sha256']='0'*64;p.write_text(json.dumps(d));h=hashlib.sha256(p.read_bytes()).hexdigest()
|
||||
with self.assertRaisesRegex(ValueError,'binding'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
|
||||
d['source_cache_sha256']=sha;d['records'][0]['title']='Wrong title';p.write_text(json.dumps(d));h=hashlib.sha256(p.read_bytes()).hexdigest()
|
||||
with self.assertRaisesRegex(ValueError,'canonical H1'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
|
||||
|
||||
def test_unlisted_review_entry_and_changed_primary_digest_fail_closed(self):
|
||||
sha,p,h=self.review();d=json.loads(p.read_text());d['records'][0]['individual_sha256']='0'*64;p.write_text(json.dumps(d));h=hashlib.sha256(p.read_bytes()).hexdigest()
|
||||
with self.assertRaisesRegex(ValueError,'page checksum'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
|
||||
self.pages[SPRING+'posters/abstract.html']=self.pages[SPRING+'posters/abstract.html'].replace('01/paper.html','02/other.html')
|
||||
self.pages[SPRING+'posters/author.html']=self.pages[SPRING+'posters/author.html'].replace('01/paper.html','02/other.html')
|
||||
sha,p,h=self.review()
|
||||
with self.assertRaisesRegex(ValueError,'outside abstract'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
|
||||
Reference in New Issue
Block a user