Compare commits
5 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 25ed4559ca | |||
| b226b3b98b | |||
| 05ada8daf8 | |||
| 13408869c2 | |||
| 443a79dce2 |
@@ -50,7 +50,9 @@ jobs:
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','cvf','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','www-archive','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','aamas-proceedings','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','aamas-proceedings','--year','2025','--snapshots',str(src/'tests/fixtures/aamas2025'),'--out',str(root/'installed-aamas2025')],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','iclr-archive','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','acl','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','postgres','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','edition-tracks','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','joint-editions','--help'],cwd=outside,check=True)
|
||||
|
||||
@@ -6,10 +6,14 @@ Paper Atlas 将 arXiv、AI 会议论文集及公开投稿决定整理到可追
|
||||
|
||||
项目仍在建设中。候选范围包含 43 个会议(含 ACM MM、MLSys),尚未完成全部历史数据和最终去重复核。实际进度以数据库、来源清单和覆盖报告为准,而不是仓库文件数量。
|
||||
|
||||
当前主动补漏从 **2020 年**起;2000 年起已发现的名单、缺口与待办保存在[补漏恢复清单](docs/coverage/backfill-handoff-2026-11-10.md),下次从这里接续。已有旧数据保留,范围排除不算补齐。
|
||||
|
||||
历史 ICLR 2014–2016 官方录用页由 `collect iclr-archive` 适配,保留主会/workshop及重复展示证据;运行和身份核验边界见 [历史档案说明](docs/coverage/iclr-archive-2014-2016.md)。
|
||||
|
||||
## 项目结构
|
||||
|
||||
ACL 1978 的年会与 TINLAP-2 来源关系按精确卷适配,保留原系列和不明逐篇角色;1965 的替代关系不转成全局别名。来源、排除条目和恢复边界见 [ACL 早期来源核查](docs/coverage/acl-early-sources-2026-11-09.md)。
|
||||
|
||||
```text
|
||||
paper_atlas/
|
||||
cli.py 统一命令入口
|
||||
@@ -66,7 +70,7 @@ paper-atlas collect colm --cache /data/paper-index/raw/http-cache \
|
||||
|
||||
EWSL、EPS、ECP、EWSP 的[具体前身卷与完整快照要求](docs/coverage/historical-predecessors-2026-11-09.md)单独核验,保留原始系列及事件/出版年份;旧候选文件缺少 ECP/EWSP 时会明确提示,避免静默遗漏。
|
||||
|
||||
AAMAS 的[官方论文集采集器](docs/aamas-proceedings.md)目前核验了 2012 版式,可从缓存或已校验快照采集标题、作者及页面提供的摘要;发表形式单独保存,全文/扩展摘要中未明确的投稿 track 保持待明确,不根据会话主题推定主会轨道。
|
||||
AAMAS 的[官方论文集采集器](docs/aamas-proceedings.md)支持已核验的 2012、2025 版式,可从缓存或已校验快照采集元数据。2025 年目录含 476 条非 keynote 出版物,保留完整作者和机构,HTML 未提供摘要时留空;发表形式与投稿 track 分开,不根据主题推定主会轨道或研究类型。
|
||||
|
||||
`paper-atlas collect --help` 查看来源;`paper-atlas import --help` 查看导入方式。完整批次步骤见[使用指南](docs/usage.md)。
|
||||
|
||||
|
||||
@@ -1,6 +1,14 @@
|
||||
# AAMAS 官方论文集元数据
|
||||
|
||||
`paper_atlas.collectors.aamas` 是按需运行的采集适配器,通过统一入口 `collect aamas-proceedings` 使用。目前只支持已独立核验的 **AAMAS 2012** HTML 版式;其他年份需先核对官方目录和逐篇结构,再增加适配,不能仅替换年份。
|
||||
`paper_atlas.collectors.aamas` 是按需运行的采集适配器,通过统一入口 `collect aamas-proceedings` 使用。支持已独立核验的 **AAMAS 2012、2025** HTML 版式;其他年份需先核对官方目录和逐篇结构,再增加适配,不能仅替换年份。2025 解析单独封装在 `paper_atlas.collectors.aamas2025`。
|
||||
|
||||
2025 版式使用[官方首页](https://www.ifaamas.org/Proceedings/aamas2025/)和[出版目录](https://www.ifaamas.org/Proceedings/aamas2025/forms/contents.htm)。原始目录共 479 条:3 keynote、260 research paper、148 extended abstract、12 Blue Sky、25 doctoral、22 Demo 栏目、9 JAAMAS。排除 keynote,输出 476 条;Demo 栏目中的 p3000 是明确标题的 COMP24 竞赛报告,发表年份仍为 2025,格式记 `competition_report`、投稿 track 留作 `proceedings_unspecified`,保留原 Demo 栏目证据。其余输出按 track 分组为 409 unspecified、12 blue_sky、25 doctoral、21 demo、9 jaamas。
|
||||
|
||||
作者按逐篇 BR 行解析,斜体机构原文另存,保留 Unicode、顺序及完整名单。HTML 没有提供摘要或逐篇 DOI,故 `abstract=null`、标识字典为空;只保存 PDF 链接,不下载研究 PDF。不能把 Research Paper Track 的所有条目推成 main,不能把目录缺席推为 rejected/withdrawn,也不能把出版格式标签当研究类型。
|
||||
|
||||
2025 用法:`paper-atlas collect aamas-proceedings --year 2025 --snapshots /data/review/aamas2025 --out /data/paper-index/collected/aamas-2025`;在线模式改用 `--cache /data/paper-index/cache/aamas2025`,只请求上述两个 HTML 页面。快照目录包含原始 `index.html`、`contents.html` 和 `receipts.json`,凭据必须列出各原始 URL、文件、字节数、SHA256、`http_status=200` 和带时区 `at`。已恢复的凭据需保留 `at_precision=filesystem_write_time_recovered` 等说明,不能伪造 HTTP 头或请求时间。2026-11-09 保存的两个页面通过哈希校验;独立录用清单 URL 返回 403,失败保留,不绕过或依赖其完成采集。
|
||||
|
||||
2025 命令同样使用目录锁、临时目录、最后写入采集报告。五个 JSONL/manifest 均保持 `partial`,记录摘要缺失数量;它不连接数据库。首次采集需独立核验全部标题/作者/栏目和 source IDs,随后做 PostgreSQL 身份预检、必要的逐条版本审核、原生导入与独立验收。若失败且没有完成报告,应先检查原始响应和错误,再使用新输出目录重跑,不重放已提交批次。代码验证、原始目录盘点和采集成功均不代表已导入生产或最终去重完成。
|
||||
|
||||
## 来源与范围
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# 会议覆盖与公开决定核查
|
||||
|
||||
**当前工作范围:2020 年起主动补漏。** 2000 年起已发现资料保留,恢复工作从 [补漏范围与恢复清单(2026-11-10)](backfill-handoff-2026-11-10.md) 开始;下文旧日期快照不改写,较早年份排除不算补齐。
|
||||
|
||||
这是 2026-11-08 的首份核查基线,**完整逐年政策普查和最终去重仍未完成**。收录 43 个目标,不能把“有记录”当作“完整覆盖”。结构化清单见 [inventory-2026-11-08.json](inventory-2026-11-08.json)。清单中的 `policy_checks` 均限定所列年份和轨道;空数组表示尚未核实,绝不表示没有公开拒稿。
|
||||
|
||||
## 快照与复核
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# ACL 早期年份:会议存在、出版目录与逐篇资料分开核验
|
||||
|
||||
固定普查中 ACL 1963–1978 仍有年份缺口。不能因为会议年表连续,就假设每年都有独立的正式论文集;也不能因为当前目录始于 1979,就断言此前没有摘要、通知或其他公开资料。本次不修改固定的 1,266 个目标年份基准。
|
||||
|
||||
## 1978 年:限定 T78-1 的来源关系
|
||||
|
||||
[同期年会通知](https://aclanthology.org/J77-4001.pdf)的 PDF 第 9–10 页(印刷页码 9–10)明确说明,1978 年 ACL 年会采用第二届 Theoretical Issues in Natural Language Processing 的形式,地点为 Urbana/Champaign,日期为 7 月 25–27 日。通知中的 ACM 联合赞助确认仍处于待确认状态,不能将该预告改写为已完成的行政决定。
|
||||
|
||||
[1982 年 Donald E. Walker 的历史回顾](https://aclanthology.org/www.mt-archive.info/70/ACL-1982-Walker-1.pdf)第 3 页(印刷页码 91)的年会表及缩写说明,独立对应了 1978 年的时间、地点与 TNLP/TINLAP-2 关系。两份扫描件相关页已渲染并逐页目视核对,不仅依靠 OCR。
|
||||
|
||||
[官方卷目录](https://aclanthology.org/volumes/T78-1/)与固定 XML `T78.xml` 保留原系列名 TINLAP、完整卷名、1978 年及编辑 David L. Waltz。适配仅接受精确的 collection `T78`、volume `1` 和上述字段组合;关键元数据变化会拒绝处理,等待重新核查。
|
||||
|
||||
- 规范事件名使用 `ACL+TINLAP`,原系列名 `TINLAP`、原始 XML 的 `venue_codes: [tinlap]` 及证据均保留。
|
||||
- ACL/TINLAP 别名只挂在这一规范事件系列上,不把所有 TINLAP 年份都算成 ACL。T75、T87 及 T78 其他卷没有此例外。
|
||||
- 通知描述了邀请与提交论文两种参与方式,XML 没有逐篇角色。统一暂存为 `proceedings_unspecified`,不猜测每篇属于 invited、main 或 workshop。
|
||||
- XML 共 36 条条目,其中 T78-1036 是无作者的场次讨论问题列表。精确识别并在 manifest 的 `excluded_materials` 保留其标识、标题与原因;其余 35 条作为具名贡献整理。该排除不推广到所有无作者条目,已识别条目发生变化也须重新核查。
|
||||
- 全部 35 条 XML 均无摘要。出版目录中的 accepted 是来源观察;通知提及录用/不录用通知时间,不提供任何可识别的拒稿决定,不能制造 rejected 或 withdrawn。
|
||||
|
||||
采集器仅生成元数据文件,不写业务数据库。真实来源逐条验收、PostgreSQL 身份预检、串行导入与独立验收仍是后续门槛。35 条来源不等于 35 个新增论文实体,更不等于完成全库去重。
|
||||
|
||||
## 1965 年:替代关系不改写为别名
|
||||
|
||||
Walker 回顾第 1 页(印刷页码 89)说明,1965 年 ICCL(后来称 COLING)替代了当年的年会。[官方 C65-1 目录](https://aclanthology.org/volumes/C65-1/)及 XML 仍以 COLING 标记该卷。这是限定年份的历史关系,不证明 COLING 与 ACL 是两个可全局互换的名字。
|
||||
|
||||
本次保留 C65 原有 `COLING` 归属,不添加 ACL 全局别名、不重放已导入记录、不将某一年的日历关系推广到整个系列,也不据此悄悄减少固定普查缺口。其他 1963–1977 年资料仍须逐年核查;本次不能证明这些年份整体属于非归档例外。
|
||||
|
||||
## 固定来源与恢复
|
||||
|
||||
服务器证据目录:`/data/paper-index/collected/acl-early-source-evidence-v1`。五份官方响应均 HTTP 200,另有两份来自既有官方 XML checkout 的文件,已与 Git 对象逐字节核对。未下载论文全文集合;两份 PDF 是本次历史/行政取证资料,不进入论文全文库。
|
||||
|
||||
| 文件 | SHA256 |
|
||||
|---|---|
|
||||
| `acl-venue.html` | `c60f64cb4d72e677aeb84a5844cde28a8d50579e1995f08e8c7801eed1e3fac5` |
|
||||
| `acl-history-1982.pdf` | `35c21bd4983975bcaccbb94bd8ab91c9019fc190feec1791255407cf18a7e49a` |
|
||||
| `acl-1978-notice.pdf` | `14b5deb471c55afb07d789529b95d0462a2a1a77df3dfba7fa1f5d3ab75eb411` |
|
||||
| `tinlap1978-volume.html` | `7e20d709f72f415819e4d17e3674b19dc7d871eabcf0cf8b4935f0392d775729` |
|
||||
| `coling1965-volume.html` | `0d2f2f49f2a311f9c0b80a8832f6476eb32f94f35abf36f6d2f2e10063ac880f` |
|
||||
| `T78.xml` | `f712f94cff0cba05f786b7ac31dad2ea9408fde2a17a04531130df623f57dd84` |
|
||||
| `C65.xml` | `572791610e20b7c6045d984d968e73a18a144bcb5c3da64112001c781826808e` |
|
||||
|
||||
XML commit:`a7be623d01c48c61405dec31c871121cdbed5a01`。测试 fixture `tests/fixtures/acl-t78.xml` 是该原始文件,不是手工模拟论文;不包含 PDF。PDF 哈希建立完整性,不单独证明语义,语义依据是上述具体页面。
|
||||
|
||||
在服务器使用既有命令,固定 checkout 并输出到新目录:
|
||||
|
||||
```sh
|
||||
paper-atlas collect acl \
|
||||
--checkout /data/paper-index/raw/metadata-checkouts/acl-anthology \
|
||||
--collection T78 \
|
||||
--out /data/paper-index/collected/acl-t78-source-v1
|
||||
```
|
||||
|
||||
该命令会校验/更新官方 metadata checkout,实际 commit 必须以生成清单为准;严格重现本次取证时,使用已冻结的 XML 和上述 commit,不把更新后的内容冒充旧快照。不要将无 collection 限制的全部历史输出直接作为增量队列。采集完成不代表导入完成,恢复前先检查当前单元和报告,不因日志安静重启已执行的写入。
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,52 @@
|
||||
# 补漏范围与恢复清单(2026-11-10)
|
||||
|
||||
用户确定:**后续主动补漏从 2020 年起**。2000 年起已经发现的名单、缺口、资料组合和身份核查记录保留在项目中,供下次接着做;已有旧数据继续保留。停止补采的年份标为范围外或延期,不算作已补齐。
|
||||
|
||||
本页是下一次工作的入口,结构化记录见 [backfill-handoff-2026-11-10.json](backfill-handoff-2026-11-10.json)。它保存 43 个会议的 2000–2026 已观察年份、缺年、官方目录文件入口、近期待办与恢复证据。旧的日期快照不改写,也不能当实时数据库。
|
||||
|
||||
## 当前范围和年份差集
|
||||
|
||||
依据生产只读快照 `2026-11-09 16:26:41.688509+00:00`,原全历史口径为 1266 个目标年份槽位:1208 已观察、43 个历史缺年、14 个 2026 缺年,以及 1 个 SysML 2018 非存档例外。原 135 个历史缺口中有 92 个通过数据补入解决;其余 43 个现在停止补采,不能因此宣称全部解决。
|
||||
|
||||
- **2020–2025**:没有整年完全缺记录的目标槽位,但逐篇、track、摘要、公开负面决定仍不完整。
|
||||
- **2026**:NeurIPS、ECCV、RecSys、CIKM、ICDM、SDM、IROS、RSS、CoRL、ICAPS、AAMAS、ICASSP、ISMIR、ACM MM,共 14 个槽位。须逐个核查是否已经公开出版;不能将尚未发布的资料说成漏采。
|
||||
- **2000–2019 延期线索**:WWW 2000、ICAPS 2001 两个整年缺口;官方历届记录分别见 [WWW](www-editions-2026-11-08.json)、[ICAPS](icaps-editions-2026-11-08.json)。SysML 2018 是非存档首届,见 [MLSys](mlsys-editions-2026-11-08.json),不能制造正式出版记录。
|
||||
- **2000 年前**:剩余 41 个历史缺年停止补采。旧来源文件继续保留作证据,不再作为当前工作队列。
|
||||
|
||||
以上只判断目标会议年份是否至少有一条投稿记录,绝不等于目录完整、独立论文数量或最终去重验收。
|
||||
|
||||
## 近期资料与恢复位置
|
||||
|
||||
服务器采集根目录为 `/data/paper-index/collected`,本地证据备份为 `/Users/zzyang/Documents/Codex/2026-11-06/paper-index`。这些目录提供大体积原始文件,仓库 JSON 提供路径、状态和可用哈希;恢复前先验证文件存在、哈希和生产实际状态。
|
||||
|
||||
| 资料 | 当前状态 | 下一步与证据入口 |
|
||||
| --- | --- | --- |
|
||||
| AAMAS 2026 | 官方目录 643 条,3 个 keynote 排除;640 个非 keynote 中含 1 个明确撤稿。尚未导入 | `aamas2026-primary-evidence-v1/` 保存官方 HTML、回执、逐项目录、审计和校验清单。八个标题差异/撤稿前缀条目需核查。先扩展正式 collector 并远程验证,再做新 PostgreSQL 身份预检。 |
|
||||
| CVPR 2026 Findings | 941 条摘要已核验,尚未导入 | `cvf-2026-verified/recovery-audit.json`;原 404 与 RVLF/FVLF 路径更正证据保留。先核对 manifests、track 和实体关系。 |
|
||||
| AAMAS 2025 | 476 个非 keynote 来源中,240 已验收、236 待身份核查 | `aamas2025-import225-v1/` 与 `aamas2025-import15-v1/` 是已消费批次,禁止重放。15 条最近验收报告哈希见 JSON。余下资料从当前身份候选与新只读基线继续。 |
|
||||
| RSS 2025 | 163 条目录中 162 已验收,`2025/p117` 待核查 | 保留 spinal/RMCC 关系问题;查看工作状态键 `rss2025_held7_primary_evidence_v1`,不能重跑已消费的 6/88/54/14 批次。 |
|
||||
| 近期公开负面决定首批 | 1625 条已验收:1624 rejected、1 withdrawn | `public-negative-postgresql-import-postcheck-v1.json`;仅该批次验收,不是公开拒稿全集,不要重放。 |
|
||||
|
||||
AAMAS 2026 官方目录分区:research 338、extended abstracts 185、AAAI 19、Blue Sky 17、doctoral 34、demo 44、JAAMAS 3,另有 keynote 3。JAAMAS 是期刊展示关系,不能凭目录再造一次独立审稿录用。官方撤稿条目在 research 分区,不能把全目录都标为当前 accepted。
|
||||
|
||||
AAAI 转投政策有年份冲突:[详细征稿页](https://cyprusconferences.org/aamas2026/call-for-papers-aaai-track/) 指 AAAI-26 rejected,[总览](https://cyprusconferences.org/aamas2026/calls/) 写 AAAI-25。保留两份证据;19 个 AAAI track 条目不能直接转成 19 条有原始 note 的拒稿决定。原投稿 ID、逐篇决定和版本关系仍需核实。
|
||||
|
||||
## 2000–2019 已发现但延期的身份与名单工作
|
||||
|
||||
- **AAMAS 2012**:官方 334 条,319 已独立验收、15 个身份条目保留待查。验收报告 `aamas2012-import-v1/postgresql-postcheck-v1.json`,哈希在 JSON;已验收 319 条及被替代的 316 条计划均不得重放。来源布局与用法见 [AAMAS 适配说明](../aamas-proceedings.md)。
|
||||
- **UAI 2008**:官方 programme 与归档资料确认了四篇独立论文,但部分原始 DBLP 键的标题、作者、页码、出版链接对应仍有冲突。JSON 保留四组候选、arXiv ID、官方题名及归档报告哈希。之前仅根据页码推断题名/作者错误的结论已撤回,未应用纠正;恢复时不得据此覆盖或合并。
|
||||
- **其他名单**:本目录全部 `*-editions-2026-11-08.json` 保留各会议官方年份及来源;[历史缺口](historical-gaps-2026-11-09.md)、[来源候选](historical-source-candidates-2026-11-09.json)、[来源范围](historical-source-scope-2026-11-09.md)、[主系列](historical-primary-series-2026-11-09.md)、[前身会议](historical-predecessors-2026-11-09.md)、[ICLR 档案](iclr-archive-2014-2016.md)继续留档。这些旧报告的“缺失”可能已被后续导入解决,恢复时必须对照当前状态,不能直接当可执行队列。
|
||||
|
||||
## 公开拒稿资料不能遗漏
|
||||
|
||||
结构化清单保留最近已查的政策 URL、限定年份和可用性结论。优先继续 UAI 2024–2026 的 opt-in 公开拒稿来源、EMNLP 2023 的公开集合范围及其他目标的逐年核查;平台使用 OpenReview 不能证明 rejected 公开。访问受限如实记录,不绕过 Challenge。
|
||||
|
||||
保留 rejected、desk rejected、withdrawn、retracted、resubmit 的区别;评审建议不等于最终决定。COLM 2026 的不公开政策只适用于已核实年份,不能推广到其他会议。PaperCopilot 与原 OpenReview、镜像间有重叠,候选计数不能相加。论文先被 A 拒绝再被 B 录用时,两次投稿、决定、来源与证据全部保留。
|
||||
|
||||
## 下次启动顺序
|
||||
|
||||
1. 读取本页和 JSON,再核实服务器工作状态、明确的 systemd invocation 与终态报告。权威工作状态 `/Users/zzyang/Documents/Codex/2026-11-06/paper-index/work-state.json` 中部分旧字段会过期;以最新验收和实际查询为准。
|
||||
2. 当前主动范围为 2020 起,2000–2019 只恢复线索,不自动启动其采集。先检查新来源是否已导入,验收过的队列禁止重放。
|
||||
3. 使用现有 `foundry-postgres` 的 `foundry_data.research`。当前已验证 writer 为提交 `714f745bd5c3fbc742be1efae0b31ab37a55e21f` 的固定 release;恢复时重新确认,不使用旧 `/data/paper-index/code` 替代。服务配置仅由服务器受保护文件读取,不复制密码。
|
||||
4. 对待导入资料做新只读身份预检,处理稳定标识、短标题、作者/版本差异、track 与跨会历史,再生成小范围增量队列。每批独立验收,SQLite 只读备份保持保护。
|
||||
5. 编译、测试或 wheel 构建仅在已确认 `ces-control:host` runner 执行。代码按工作分支交付 PR,等待用户手动合并。范围内导入结束后仍须全库去重、近似标题/作者冲突、来源-年份-track、摘要、约束、实际全文查询验收;不能用单批导入或 CI 成功替代。
|
||||
@@ -136,6 +136,9 @@ def read_pages(cache=None, snapshots=None):
|
||||
|
||||
|
||||
def _collect(year, out, *, cache=None, snapshots=None):
|
||||
if year == 2025:
|
||||
from paper_atlas.collectors.aamas2025 import collect as collect_2025
|
||||
return collect_2025(out, cache=cache, snapshots=snapshots)
|
||||
if year != 2012:
|
||||
raise ValueError('Only the independently reviewed AAMAS 2012 layout is supported')
|
||||
pages, receipts = read_pages(cache, snapshots)
|
||||
@@ -190,8 +193,8 @@ def _collect(year, out, *, cache=None, snapshots=None):
|
||||
|
||||
|
||||
def collect(year, out, *, cache=None, snapshots=None):
|
||||
if year != 2012:
|
||||
raise ValueError('Only the independently reviewed AAMAS 2012 layout is supported')
|
||||
if year not in (2012, 2025):
|
||||
raise ValueError('Only independently reviewed AAMAS 2012/2025 layouts are supported')
|
||||
out = Path(out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
with (out/'aamas.collect.lock').open('a') as lock:
|
||||
@@ -201,7 +204,7 @@ def collect(year, out, *, cache=None, snapshots=None):
|
||||
|
||||
def main(argv=None, *, prog=None):
|
||||
parser = argparse.ArgumentParser(prog=prog)
|
||||
parser.add_argument('--year', type=int, choices=[2012], required=True)
|
||||
parser.add_argument('--year', type=int, choices=[2012, 2025], required=True)
|
||||
parser.add_argument('--out', required=True)
|
||||
source = parser.add_mutually_exclusive_group(required=True)
|
||||
source.add_argument('--cache')
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
"""Reviewed AAMAS 2025 publisher TOC; HTML metadata only, never paper PDFs."""
|
||||
from collections import Counter, defaultdict
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import tempfile
|
||||
|
||||
from paper_atlas.db.submissions import timestamp
|
||||
from paper_atlas.io.html import Document, Node, clean
|
||||
from paper_atlas.io.http import fetch
|
||||
|
||||
ROOT = 'https://www.ifaamas.org/Proceedings/aamas2025/'
|
||||
URLS = {'index': ROOT, 'contents': ROOT+'forms/contents.htm'}
|
||||
TITLE = 'AAMAS 2025: Proceedings of the 2025 International Conference on Autonomous Agents and Multiagent Systems'
|
||||
SECTIONS = {
|
||||
'1': ('Keynote Talks', 'keynote', 'keynote', 3),
|
||||
'2': ('Research Paper Track', 'full_paper', 'proceedings_unspecified', 260),
|
||||
'3': ('Extended Abstracts', 'extended_abstract', 'proceedings_unspecified', 148),
|
||||
'4': ('Blue Sky Ideas', 'blue_sky_paper', 'blue_sky', 12),
|
||||
'5': ('Doctoral Consortium', 'doctoral_abstract', 'doctoral', 25),
|
||||
'6': ('Demo Track', 'demo', 'demo', 22),
|
||||
'7': ('JAAMAS Track', 'journal_presentation', 'jaamas', 9),
|
||||
}
|
||||
COMPETITION_TITLE = '[COMP24] The Automated Negotiating Agents Competition (ANAC) 2024 Challenges and Results'
|
||||
PAPER = re.compile(r'\.\./pdfs/p([0-9]+)\.pdf')
|
||||
|
||||
|
||||
def one(items, label):
|
||||
if len(items) != 1:
|
||||
raise ValueError('Expected one '+label)
|
||||
return items[0]
|
||||
|
||||
|
||||
def parse_contents(html, evidence):
|
||||
doc = Document(html)
|
||||
if [clean(n.text()) for n in doc.root.walk() if n.tag == 'title'] != [TITLE]:
|
||||
raise ValueError('Unexpected AAMAS proceedings title/year')
|
||||
for paragraph in doc.root.walk():
|
||||
if paragraph.tag != 'p' or any(n is not paragraph and n.tag == 'p' for n in paragraph.walk()):
|
||||
continue
|
||||
anchors = [n.attrs['name'] for n in paragraph.walk() if n.tag == 'a' and n.attrs.get('name') in SECTIONS]
|
||||
if anchors and (len(anchors) != 1 or clean(paragraph.text()) != SECTIONS[anchors[0]][0]):
|
||||
raise ValueError('Publisher section heading differs from reviewed layout')
|
||||
section = None
|
||||
headers = []
|
||||
records, excluded, counts, seen = [], [], Counter(), set()
|
||||
for node in doc.root.walk():
|
||||
if node.tag == 'a' and node.attrs.get('name') in SECTIONS:
|
||||
section = node.attrs['name']
|
||||
headers.append(section)
|
||||
if node.tag != 'p' or any(n is not node and n.tag == 'p' for n in node.walk()):
|
||||
continue
|
||||
links = [n for n in node.walk() if n.tag == 'a' and PAPER.fullmatch(n.attrs.get('href', ''))]
|
||||
if not links:
|
||||
continue
|
||||
link = one(links, 'paper link per paragraph')
|
||||
if section not in SECTIONS:
|
||||
raise ValueError('Paper outside a reviewed publisher section')
|
||||
page = int(PAPER.fullmatch(link.attrs['href'])[1])
|
||||
if page in seen:
|
||||
raise ValueError('Repeated publisher paper/page ID')
|
||||
seen.add(page)
|
||||
title = clean(link.text())
|
||||
if not title:
|
||||
raise ValueError('Empty paper title')
|
||||
# Names are outside italicized affiliations and separated by explicit BRs.
|
||||
font = one([n for n in node.walk() if n.tag == 'font' and n.attrs.get('color', '').lower() == '#222222'], 'ordered author block')
|
||||
authors, affiliations = [], []
|
||||
name, affiliation = '', ''
|
||||
def flush():
|
||||
nonlocal name, affiliation
|
||||
if clean(name) or clean(affiliation):
|
||||
if not clean(name):
|
||||
raise ValueError('Affiliation without author')
|
||||
authors.append(clean(name)); affiliations.append(clean(affiliation))
|
||||
name, affiliation = '', ''
|
||||
for child in font.children:
|
||||
if isinstance(child, str):
|
||||
name += child
|
||||
elif child.tag == 'br':
|
||||
flush()
|
||||
elif child.tag == 'i':
|
||||
affiliation += child.text()
|
||||
else:
|
||||
raise ValueError('Unreviewed author markup: '+child.tag)
|
||||
flush()
|
||||
if not authors:
|
||||
raise ValueError('Empty author list')
|
||||
heading, publication_format, track, _ = SECTIONS[section]
|
||||
counts[section] += 1
|
||||
if section == '1':
|
||||
excluded.append({'publisher_page': page, 'title': title, 'authors': authors, 'reason': 'publisher keynote talk, not a research-paper submission'})
|
||||
continue
|
||||
note = 'Explicit publisher section; section topics are not research types.'
|
||||
if track == 'proceedings_unspecified':
|
||||
note = 'Published format is known; individual main/special submission track is not provided.'
|
||||
if page == 3000:
|
||||
if section != '6' or title != COMPETITION_TITLE:
|
||||
raise ValueError('Competition report differs from reviewed publisher entry')
|
||||
publication_format, track = 'competition_report', 'proceedings_unspecified'
|
||||
note = 'Publisher places this competition report in Demo Track; no individual demonstration submission is inferred.'
|
||||
records.append({
|
||||
'source': 'aamas_proceedings', 'external_id': '2025/p'+str(page),
|
||||
'url': URLS['contents']+'#'+section, 'title': title, 'abstract': None,
|
||||
'authors': authors, 'identifiers': {}, 'venue': 'AAMAS', 'venue_aliases': [],
|
||||
'year': 2025, 'track': track, 'status': 'accepted',
|
||||
'observed_at': evidence['observed_at'],
|
||||
'raw_decision': 'listed in official AAMAS 2025 published proceedings '+heading,
|
||||
'tags': [{'namespace': 'aamas_proceedings:publication_format', 'value': publication_format}],
|
||||
'metadata': {'publisher_page': page, 'paper_url': ROOT+'pdfs/p'+str(page)+'.pdf',
|
||||
'section_url': URLS['contents'], 'section_anchor': section,
|
||||
'section_heading': heading, 'publication_format': publication_format,
|
||||
'author_affiliations': affiliations, 'source_evidence': evidence,
|
||||
'track_scope_note': note, 'decision_date_precision': 'not_supplied',
|
||||
'missing_abstract_reason': 'Publisher HTML supplies no abstract; research PDF not downloaded.',
|
||||
'publication_evidence': 'Published proceedings entry, not an original decision note.'}})
|
||||
if headers != list(SECTIONS) or dict(counts) != {k: v[3] for k, v in SECTIONS.items()}:
|
||||
raise ValueError('Publisher section inventory differs from reviewed 2025 layout')
|
||||
if len(records) != 476 or len(excluded) != 3 or sum(r['metadata']['publication_format'] == 'competition_report' for r in records) != 1:
|
||||
raise ValueError('Unexpected publication partition')
|
||||
return records, excluded
|
||||
|
||||
|
||||
def read_pages(cache, snapshots):
|
||||
if bool(cache) == bool(snapshots):
|
||||
raise ValueError('Choose exactly one cache or frozen snapshot directory')
|
||||
frozen = json.loads((Path(snapshots)/'receipts.json').read_text())['receipts'] if snapshots else []
|
||||
pages = {}
|
||||
for key, url in URLS.items():
|
||||
if snapshots:
|
||||
r = one([r for r in frozen if r['url'] == url], 'frozen response receipt')
|
||||
filename = Path(r['file']); base = Path(snapshots).resolve()
|
||||
if filename.is_absolute() or '..' in filename.parts or not (base/filename).resolve().is_relative_to(base) or r.get('http_status') != 200:
|
||||
raise ValueError('Unsafe or unsuccessful publisher response')
|
||||
data = (base/filename).read_bytes()
|
||||
if len(data) > 8*1024**2 or len(data) != r['bytes'] or hashlib.sha256(data).hexdigest() != r['sha256']:
|
||||
raise ValueError('Frozen publisher response checksum/size mismatch')
|
||||
observed = r['at']
|
||||
else:
|
||||
data = fetch(url, cache, 8*1024**2, delay=1)
|
||||
r = json.loads((Path(cache)/hashlib.sha256(url.encode()).hexdigest()).with_suffix('.json').read_text())
|
||||
observed = r['retrieved_at']
|
||||
timestamp(observed)
|
||||
pages[key] = data, {'url': url, 'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data),
|
||||
'file': 'evidence/'+hashlib.sha256(data).hexdigest()+'.html',
|
||||
'observed_at': observed, 'http_status': 200, 'response_receipt': r}
|
||||
return pages
|
||||
|
||||
|
||||
def collect(out, *, cache=None, snapshots=None):
|
||||
pages = read_pages(cache, snapshots)
|
||||
if [clean(n.text()) for n in Document(pages['index'][0].decode('utf-8')).root.walk() if n.tag == 'title'] != [TITLE]:
|
||||
raise ValueError('Unexpected AAMAS 2025 introduction')
|
||||
records, excluded = parse_contents(pages['contents'][0].decode('utf-8'), pages['contents'][1])
|
||||
grouped = defaultdict(list)
|
||||
for row in records: grouped[row['track']].append(row)
|
||||
out = Path(out); out.mkdir(parents=True, exist_ok=True); files = []
|
||||
with tempfile.TemporaryDirectory(prefix='.aamas2025-', dir=out) as temporary:
|
||||
stage = Path(temporary); (stage/'evidence').mkdir()
|
||||
for data, e in pages.values(): (stage/e['file']).write_bytes(data)
|
||||
for track, group in sorted(grouped.items()):
|
||||
p = stage/f'aamas-proceedings-2025-{track}.jsonl'
|
||||
p.write_text(''.join(json.dumps(r, ensure_ascii=False)+'\n' for r in group))
|
||||
m = {'source': 'aamas_proceedings', 'parser_version': 2, 'venue': 'AAMAS', 'year': 2025,
|
||||
'track': track, 'count': len(group), 'file': str(out/p.name), 'file_sha256': hashlib.sha256(p.read_bytes()).hexdigest(),
|
||||
'list_url': URLS['contents'], 'status': 'partial', 'missing_abstract_count': len(group),
|
||||
'publication_format_counts': dict(Counter(r['metadata']['publication_format'] for r in group)),
|
||||
'note': 'Complete published TOC partition; HTML abstracts absent, identities and submission-track scope require independent review.'}
|
||||
p.with_suffix('.manifest.json').write_text(json.dumps(m, ensure_ascii=False, indent=2)+'\n'); files.append(m)
|
||||
result = {'source': 'aamas_proceedings', 'year': 2025, 'finished_at': timestamp(),
|
||||
'status': 'collected_pending_identity_review', 'records': len(records), 'files': files,
|
||||
'excluded_publications': excluded, 'source_receipts': [e for _, e in pages.values()],
|
||||
'publication_format_counts': dict(Counter(r['metadata']['publication_format'] for r in records)),
|
||||
'database_written': False, 'pdfs_downloaded': False,
|
||||
'note': 'Neither absence from proceedings nor aggregate rejection/withdrawal counts identify an individual negative decision.'}
|
||||
report = 'aamas-proceedings-2025.json'; (stage/report).write_text(json.dumps(result, ensure_ascii=False, indent=2)+'\n')
|
||||
(out/'evidence').mkdir(exist_ok=True)
|
||||
for p in (stage/'evidence').iterdir(): p.replace(out/'evidence'/p.name)
|
||||
for p in stage.iterdir():
|
||||
if p.is_file() and p.name != report: p.replace(out/p.name)
|
||||
(stage/report).replace(out/report)
|
||||
return result
|
||||
@@ -6,6 +6,7 @@ import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
|
||||
from paper_atlas.io.git_metadata import checkout_metadata
|
||||
from paper_atlas.collectors.acl_historical import event_scope, material_reason
|
||||
|
||||
VENUES={'acl':'ACL','emnlp':'EMNLP','naacl':'NAACL','eacl':'EACL','coling':'COLING','conll':'CoNLL'}
|
||||
|
||||
@@ -29,7 +30,10 @@ def paper_id(collection,volume,number):
|
||||
return f'{collection}-{int(volume):0{width}d}{int(number):0{digits}d}'
|
||||
|
||||
|
||||
def volume_scope(meta,volume):
|
||||
def volume_scope(meta,volume,collection=None):
|
||||
historical=event_scope(meta,volume,collection)
|
||||
if historical:
|
||||
return historical['canonical_venue'],historical['venue_aliases'],historical['track']
|
||||
codes=[text(n).lower() for n in meta.findall('venue')]
|
||||
book=text(meta.find('booktitle'))
|
||||
lower=book.lower()
|
||||
@@ -59,7 +63,7 @@ def collect_xml(path,out,commit):
|
||||
vid=volume.attrib['id']
|
||||
meta=volume.find('meta')
|
||||
if meta is None:raise ValueError('Volume metadata missing')
|
||||
scope=volume_scope(meta,vid)
|
||||
scope=volume_scope(meta,vid,collection)
|
||||
key=f'{collection}-{vid}'
|
||||
if scope is None:
|
||||
results.append({'volume':key,'status':'excluded_by_venue','venue_codes':[text(n) for n in meta.findall('venue')],'booktitle':text(meta.find('booktitle')),'year':text(meta.find('year'))})
|
||||
@@ -70,12 +74,18 @@ def collect_xml(path,out,commit):
|
||||
year=int(year_text)
|
||||
records=[]
|
||||
frontmatter=0
|
||||
excluded_materials=[]
|
||||
historical=event_scope(meta,vid,collection)
|
||||
for paper in volume.findall('paper'):
|
||||
pid=paper.attrib['id']
|
||||
if int(pid)==0:
|
||||
frontmatter+=1
|
||||
continue
|
||||
identifier=paper_id(collection,vid,pid)
|
||||
reason=material_reason(collection,vid,paper)
|
||||
if reason:
|
||||
excluded_materials.append({'external_id':identifier,'title':text(paper.find('title')),'reason':reason})
|
||||
continue
|
||||
doi=text(paper.find('doi'))
|
||||
if doi.startswith('10.18653/v1/') and doi.split('10.18653/v1/',1)[1].casefold()!=identifier.casefold():
|
||||
raise ValueError('Official DOI disagrees with constructed Anthology ID: '+identifier)
|
||||
@@ -83,6 +93,8 @@ def collect_xml(path,out,commit):
|
||||
if not title:raise ValueError('Missing paper title: '+identifier)
|
||||
authors=[{'given':text(a.find('first')),'family':text(a.find('last'))} for a in paper.findall('author')]
|
||||
records.append({'source':'acl_anthology','external_id':identifier,'url':f'https://aclanthology.org/{identifier}/','title':title,'abstract':text(paper.find('abstract')) or None,'authors':authors,'identifiers':{'acl':identifier,'doi':doi},'venue':venue,'venue_aliases':aliases if len(aliases)>1 else [],'year':year,'track':track,'status':'accepted','raw_decision':'listed in authoritative ACL Anthology proceedings XML','metadata':{'collection':collection,'volume':vid,'booktitle':text(meta.find('booktitle')),'venue_codes':[text(n) for n in meta.findall('venue')],'pages':text(paper.find('pages')),'publisher':text(meta.find('publisher')),'metadata_commit':commit,'source_xml':f'https://github.com/acl-org/acl-anthology/blob/{commit}/data/xml/{collection}.xml'}})
|
||||
if historical:
|
||||
records[-1]['metadata']['historical_event_scope']=historical
|
||||
if not records:
|
||||
results.append({'volume':key,'status':'unavailable','reason':'no research paper elements after excluding front matter'})
|
||||
continue
|
||||
@@ -93,7 +105,7 @@ def collect_xml(path,out,commit):
|
||||
for record in records:output.write(json.dumps(record,ensure_ascii=False)+'\n')
|
||||
part.replace(target)
|
||||
missing=sum(not row['abstract'] for row in records)
|
||||
manifest={'source':'acl_anthology','venue':venue,'year':year,'track':track,'count':len(records),'xml_paper_elements':len(volume.findall('paper')),'excluded_frontmatter':frontmatter,'missing_abstract_count':missing,'list_url':f'https://aclanthology.org/{key}/','metadata_commit':commit,'file':str(target),'status':'partial' if missing else 'collected'}
|
||||
manifest={'source':'acl_anthology','venue':venue,'year':year,'track':track,'count':len(records),'xml_paper_elements':len(volume.findall('paper')),'excluded_frontmatter':frontmatter,'excluded_materials':excluded_materials,'missing_abstract_count':missing,'list_url':f'https://aclanthology.org/{key}/','metadata_commit':commit,'file':str(target),'status':'partial' if missing else 'collected'}
|
||||
target.with_suffix('.manifest.json').write_text(json.dumps(manifest,indent=2))
|
||||
results.append(manifest)
|
||||
print(json.dumps(manifest),flush=True)
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Edition-specific historical scope; never alias every TINLAP year to ACL."""
|
||||
|
||||
HISTORY_URL = 'https://aclanthology.org/www.mt-archive.info/70/ACL-1982-Walker-1.pdf'
|
||||
NOTICE_URL = 'https://aclanthology.org/J77-4001.pdf'
|
||||
BOOK = 'Theoretical Issues in Natural Language Processing-2'
|
||||
MATERIAL_TITLE = 'List of questions suggested for consideration in each session'
|
||||
|
||||
|
||||
def text(node):
|
||||
return ' '.join(''.join(node.itertext()).split()) if node is not None else ''
|
||||
|
||||
|
||||
def event_scope(meta, volume, collection):
|
||||
if (collection, volume) != ('T78', '1'):
|
||||
return None
|
||||
if ([text(n) for n in meta.findall('venue')] != ['tinlap']
|
||||
or [text(n) for n in meta.findall('year')] != ['1978']
|
||||
or [text(n) for n in meta.findall('booktitle')] != [BOOK]
|
||||
or [(text(n.find('first')), text(n.find('last'))) for n in meta.findall('editor')]
|
||||
!= [('David L.', 'Waltz')]):
|
||||
raise ValueError('Reviewed T78-1 scope metadata changed; independent review required')
|
||||
return {
|
||||
'collection': 'T78', 'volume': '1', 'original_venue': 'TINLAP',
|
||||
'event_year': 1978, 'event_dates': ['1978-07-25', '1978-07-27'],
|
||||
'relationship': 'ACL annual meeting took the form of TINLAP-2',
|
||||
'canonical_venue': 'ACL+TINLAP', 'venue_aliases': ['ACL', 'TINLAP'],
|
||||
'track': 'proceedings_unspecified',
|
||||
'track_scope_note': 'Invited and submitted contributions are described at volume level; individual roles are not identified in XML.',
|
||||
'evidence': [
|
||||
{'url': NOTICE_URL, 'sha256': '14b5deb471c55afb07d789529b95d0462a2a1a77df3dfba7fa1f5d3ab75eb411',
|
||||
'pdf_pages': [9, 10], 'printed_pages': ['9', '10'],
|
||||
'scope': 'Contemporary annual meeting notice, not individual decision notes; ACM confirmation was pending in this notice.'},
|
||||
{'url': HISTORY_URL, 'sha256': '35c21bd4983975bcaccbb94bd8ab91c9019fc190feec1791255407cf18a7e49a',
|
||||
'pdf_pages': [3], 'printed_pages': ['91'],
|
||||
'scope': 'Retrospective annual meeting table and TNLP abbreviation confirm the 1978 event association.'},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def material_reason(collection, volume, paper):
|
||||
if (collection, volume, paper.attrib.get('id')) != ('T78', '1', '36'):
|
||||
return None
|
||||
if text(paper.find('title')) != MATERIAL_TITLE or paper.findall('author'):
|
||||
raise ValueError('Reviewed T78-1036 material changed; independent review required')
|
||||
return 'Session question list, not an authored research contribution; no decision inferred.'
|
||||
+4281
File diff suppressed because it is too large
Load Diff
+9968
File diff suppressed because it is too large
Load Diff
Vendored
+54
@@ -0,0 +1,54 @@
|
||||
<HTML><HEAD><TITLE>AAMAS 2025: Proceedings of the 2025 International Conference on Autonomous Agents and Multiagent Systems</TITLE>
|
||||
<META http-equiv=Content-Type content="text/html; charset=utf-8"><LINK
|
||||
href="forms/index_files/factemplate.css" type=text/css rel=stylesheet>
|
||||
<META content="MSHTML 6.00.2900.3562" name=GENERATOR></HEAD>
|
||||
<BODY bgcolor="#010101" bgproperties="fixed"><DIV style="TEXT-ALIGN: center" >
|
||||
<table width="724" border="0" align="center" cellpadding="0" cellspacing="0" valign="top">
|
||||
<td colspan="2" align="center" valign="top" bgcolor="#FFFFFF"><img src="forms/index_files/aamas25.png" width="750" height="300" alt=""/></td>
|
||||
</tr>
|
||||
<tr bgcolor="#FFFFFF">
|
||||
<td width="125" align="center" valign="top" bgcolor="#FFFFFF"><table width="125" border="1" align="center" cellpadding="1" cellspacing="1" bordercolor="#444444" style="margin-bottom: 2">
|
||||
<tr>
|
||||
<td bgcolor="#FFFFFF"><p style="text-indent: 0; margin-left: 6; margin-right: 6; margin-top: 5; margin-bottom: 4" align="left"><A href="forms/index.htm">Main
|
||||
Page</A></p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td bgcolor="#FFFFFF"><p style="text-indent: 0; margin-left: 6; margin-right: 6; margin-top: 5; margin-bottom: 4" align="left"><A href="forms/contents.htm">Table
|
||||
of Contents</A></p></td>
|
||||
</tr>
|
||||
</table></td>
|
||||
<td width="599"><p style="text-indent: 0; margin-left: 12; margin-right: 14; margin-top: 12; margin-bottom: 12" align="left"><font color="#010101" size="4" face="Arial, Helvetica, sans-serif"><strong>Welcome
|
||||
to the Proceedings of AAMAS-2025</strong></font></p>
|
||||
<p style="text-indent: 0; margin-left: 12; margin-right: 14; margin-top: 12; margin-bottom: 12;" align="left"><font color="#333333" size="2" face="Times New Roman, Times, serif">These
|
||||
are the proceedings of the 24th International Conference on Autonomous
|
||||
Agents and Multiagent Systems (AAMAS-2025). They are published by the
|
||||
International Foundation for Autonomous Agents and Multiagent Systems
|
||||
(IFAAMAS).</font></p></td>
|
||||
</tr>
|
||||
<tr bgcolor="#FFFFFF">
|
||||
<td align="center" valign="middle" bgcolor="#FFFFFF"> </td>
|
||||
<td><table width="96%" border="0" cellspacing="2" cellpadding="2">
|
||||
<tr align="left" valign="top">
|
||||
<td> <p style="text-indent: 0; margin-left: 12; margin-right: 6; margin-top: 0; margin-bottom: 6;" align="left"><font color="#333333" face="Times New Roman, Times, serif"><font size="2"><strong>ISBN: 979-8-4007-1426-9</strong></font></font></p></td>
|
||||
<td> <p style="text-indent: 0; margin-left: 12; margin-right: 6; margin-top: 0; margin-bottom: 6;" align="left"><font color="#333333" face="Times New Roman, Times, serif"></font></p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="42%"><div align="left"><img src="forms/index_files/ifaamas.jpg" width="256" height="104"></div></td>
|
||||
<td width="58%"><div align="center"><img src="forms/index_files/ACM-In-Cooperation_medium.jpg" width="285" height="86"></div></td>
|
||||
</tr>
|
||||
</table>
|
||||
<p style="text-indent: 0; margin-left: 12; margin-right: 14; margin-top: 12; margin-bottom: 12;" align="left"><font color="#333333" size="2" face="Times New Roman, Times, serif">Copyright
|
||||
© 2025 by International Foundation for Autonomous Agents and Multiagent
|
||||
Systems (IFAAMAS). Permission to make digital or hard copies of portions
|
||||
of this work for personal or classroom use is granted without fee provided
|
||||
that copies are not made or distributed for profit or commercial advantage
|
||||
and that copies bear this notice and the full citation on the first page.
|
||||
Copyright for components of this work owned by others than IFAAMAS must
|
||||
be honored. Abstracting with credit is permitted. To copy otherwise, to
|
||||
republish, to post on servers or to redistribute to lists, requires prior
|
||||
specific permission and/or a fee. </font></p></td>
|
||||
</tr>
|
||||
</table>
|
||||
<p> </p>
|
||||
|
||||
</BODY></HTML>
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"status": "two_successful_official_metadata_pages_recovered_third403_not_retried",
|
||||
"receipts": [
|
||||
{
|
||||
"file": "index.html",
|
||||
"url": "https://www.ifaamas.org/Proceedings/aamas2025/",
|
||||
"bytes": 4107,
|
||||
"sha256": "850b7825c66d0147ac8a8bbf03a554590e4a9e8f4b5c576e043999bbd43317f4",
|
||||
"at": "2026-11-09T12:31:49.347577+00:00",
|
||||
"at_precision": "filesystem_write_time_recovered",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"receipt_recovered": true,
|
||||
"scope": "Collectorasserted200/htmlbeforewrite;request3acceptedprogram403;noheaders/finalredirectURLpersisted;no refetch or rawbytesmutation."
|
||||
},
|
||||
{
|
||||
"file": "contents.html",
|
||||
"url": "https://www.ifaamas.org/Proceedings/aamas2025/forms/contents.htm",
|
||||
"bytes": 370650,
|
||||
"sha256": "732847b51972f34c721a49d8bd963a80a029a48f3ec08022c9a0d5ddc2945549",
|
||||
"at": "2026-11-09T12:31:52.006642+00:00",
|
||||
"at_precision": "filesystem_write_time_recovered",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"receipt_recovered": true,
|
||||
"scope": "Collectorasserted200/htmlbeforewrite;request3acceptedprogram403;noheaders/finalredirectURLpersisted;no refetch or rawbytesmutation."
|
||||
}
|
||||
],
|
||||
"failed_url": "https://aamas2025.org/index.php/conference/program/accepted-papers/",
|
||||
"not_attempted_url": "https://aamas2025.org/index.php/conference/calls/submission-instructions-main-technical-track/",
|
||||
"original_error": {
|
||||
"error": "HTTP Error 403: Forbidden",
|
||||
"type": "HTTPError",
|
||||
"downloaded": 2
|
||||
},
|
||||
"research_fulltext_downloaded": false,
|
||||
"database_written": false
|
||||
}
|
||||
Vendored
+273
@@ -0,0 +1,273 @@
|
||||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<collection id="T78">
|
||||
<volume id="1" type="proceedings">
|
||||
<!-- https://aclanthology.org/T78-1/ -->
|
||||
<meta>
|
||||
<booktitle>Theoretical Issues in Natural Language <fixed-case>P</fixed-case>rocessing-2</booktitle>
|
||||
<editor id="david-l-waltz"><first>David L.</first><last>Waltz</last></editor>
|
||||
<venue>tinlap</venue>
|
||||
<year>1978</year>
|
||||
</meta>
|
||||
<frontmatter>
|
||||
<!-- https://aclanthology.org/T78-1000/ -->
|
||||
<pdf hash="fb6164ad"/>
|
||||
<bibkey>tinlap-1978-theoretical</bibkey>
|
||||
</frontmatter>
|
||||
<paper id="1">
|
||||
<!-- https://aclanthology.org/T78-1001/ -->
|
||||
<title>Testing The Psychological Reality of a Representational Model</title>
|
||||
<author><first>Dedre</first><last>Gentner</last></author>
|
||||
<pdf hash="f723a956"/>
|
||||
<bibkey>gentner-1978-testing-psychological</bibkey>
|
||||
</paper>
|
||||
<paper id="2">
|
||||
<!-- https://aclanthology.org/T78-1002/ -->
|
||||
<title>What Makes Something “Ad Hoc”</title>
|
||||
<author id="roger-c-schank"><first>Roger C.</first><last>Schank</last></author>
|
||||
<pdf hash="636853f9"/>
|
||||
<bibkey>schank-1978-makes-something</bibkey>
|
||||
</paper>
|
||||
<paper id="3">
|
||||
<!-- https://aclanthology.org/T78-1003/ -->
|
||||
<title>The Relation of Grammar to Cognition–a Synopsis</title>
|
||||
<author><first>Leonard</first><last>Talmy</last></author>
|
||||
<pdf hash="101c67dc"/>
|
||||
<bibkey>talmy-1978-relation-grammar</bibkey>
|
||||
</paper>
|
||||
<paper id="4">
|
||||
<!-- https://aclanthology.org/T78-1004/ -->
|
||||
<title>On primitives, prototypes, and other semantic anomalies</title>
|
||||
<author><first>Terry</first><last>Winograd</last></author>
|
||||
<pdf hash="f8e47023"/>
|
||||
<bibkey>winograd-1978-primitives-prototypes</bibkey>
|
||||
</paper>
|
||||
<paper id="5">
|
||||
<!-- https://aclanthology.org/T78-1005/ -->
|
||||
<title>Taxonomic Lattice Structures for Situation Recognition</title>
|
||||
<author id="william-a-woods"><first>William A.</first><last>Woods</last></author>
|
||||
<pdf hash="92957ddf"/>
|
||||
<bibkey>woods-1978-taxonomic-lattice</bibkey>
|
||||
</paper>
|
||||
<paper id="6">
|
||||
<!-- https://aclanthology.org/T78-1006/ -->
|
||||
<title>Description Formation and Discourse Model Synthesis</title>
|
||||
<author id="bonnie-webber"><first>Bonnie Lynn</first><last>Webber</last></author>
|
||||
<pdf hash="ffded1d2"/>
|
||||
<bibkey>webber-1978-description-formation</bibkey>
|
||||
</paper>
|
||||
<paper id="7">
|
||||
<!-- https://aclanthology.org/T78-1007/ -->
|
||||
<title>The Processing of Referring Expressions within a Semantic Network</title>
|
||||
<author><first>John R.</first><last>Anderson</last></author>
|
||||
<pdf hash="8bcc083a"/>
|
||||
<bibkey>anderson-1978-processing-referring</bibkey>
|
||||
</paper>
|
||||
<paper id="8">
|
||||
<!-- https://aclanthology.org/T78-1008/ -->
|
||||
<title>Reference Diaries</title>
|
||||
<author><first>Herbert H.</first><last>Clark</last></author>
|
||||
<author><first>Catherine</first><last>Marshall</last></author>
|
||||
<pdf hash="d06b7ea2"/>
|
||||
<bibkey>clark-marshall-1978-reference-diaries</bibkey>
|
||||
</paper>
|
||||
<paper id="9">
|
||||
<!-- https://aclanthology.org/T78-1009/ -->
|
||||
<title>Subsequent Reference: Syntactic and Rhetorical Constraints</title>
|
||||
<author id="david-d-mcdonald"><first>David D.</first><last>McDonald</last></author>
|
||||
<pdf hash="2d424559"/>
|
||||
<bibkey>mcdonald-1978-subsequent-reference</bibkey>
|
||||
</paper>
|
||||
<paper id="10">
|
||||
<!-- https://aclanthology.org/T78-1010/ -->
|
||||
<title>Some Psycholinguistic Constraints on the Construction and Interpretation of Definite Descriptions</title>
|
||||
<author><first>Andrew</first><last>Ortony</last></author>
|
||||
<pdf hash="69807727"/>
|
||||
<bibkey>ortony-1978-psycholinguistic-constraints</bibkey>
|
||||
</paper>
|
||||
<paper id="11">
|
||||
<!-- https://aclanthology.org/T78-1011/ -->
|
||||
<title>Bound Variables and Other Anaphors</title>
|
||||
<author id="barbara-h-partee"><first>Barbara H.</first><last>Partee</last></author>
|
||||
<pdf hash="e8e6a510"/>
|
||||
<bibkey>partee-1978-bound-variables</bibkey>
|
||||
</paper>
|
||||
<paper id="12">
|
||||
<!-- https://aclanthology.org/T78-1012/ -->
|
||||
<title>The Use of Focus as a Tool for Disambiguation of Definite Noun Phrases</title>
|
||||
<author id="candace-l-sidner"><first>Candace L.</first><last>Sidner</last></author>
|
||||
<pdf hash="7b96d2b7"/>
|
||||
<bibkey>sidner-1978-use-focus</bibkey>
|
||||
</paper>
|
||||
<paper id="13">
|
||||
<!-- https://aclanthology.org/T78-1013/ -->
|
||||
<title>Focusing in Dialog</title>
|
||||
<author id="barbara-j-grosz"><first>Barbara J.</first><last>Grosz</last></author>
|
||||
<pdf hash="8c8d4c2e"/>
|
||||
<bibkey>grosz-1978-focusing-dialog</bibkey>
|
||||
</paper>
|
||||
<paper id="14">
|
||||
<!-- https://aclanthology.org/T78-1014/ -->
|
||||
<title>Topic Levels</title>
|
||||
<author><first>Joseph E.</first><last>Grimes</last></author>
|
||||
<pdf hash="f485b2ba"/>
|
||||
<bibkey>grimes-1978-topic-levels</bibkey>
|
||||
</paper>
|
||||
<paper id="15">
|
||||
<!-- https://aclanthology.org/T78-1015/ -->
|
||||
<title>Toward a Rational Model of Discourse Comprehension</title>
|
||||
<author><first>Jerry L.</first><last>Morgan</last></author>
|
||||
<pdf hash="5e12a597"/>
|
||||
<bibkey>morgan-1978-toward-rational</bibkey>
|
||||
</paper>
|
||||
<paper id="16">
|
||||
<!-- https://aclanthology.org/T78-1016/ -->
|
||||
<title>Assent and Compliance in Children’s Language Comprehension</title>
|
||||
<author><first>David R.</first><last>Olson</last></author>
|
||||
<pdf hash="708bc4a0"/>
|
||||
<bibkey>olson-1978-assent-compliance</bibkey>
|
||||
</paper>
|
||||
<paper id="17">
|
||||
<!-- https://aclanthology.org/T78-1017/ -->
|
||||
<title>Speech Acts as a Basis for Understanding Dialogue Coherence</title>
|
||||
<author id="c-raymond-perrault"><first>C. Raymond</first><last>Perrault</last></author>
|
||||
<author id="james-allen"><first>James F.</first><last>Allen</last></author>
|
||||
<pdf hash="d111402b"/>
|
||||
<bibkey>perrault-allen-1978-speech</bibkey>
|
||||
</paper>
|
||||
<paper id="18">
|
||||
<!-- https://aclanthology.org/T78-1018/ -->
|
||||
<title>A Framework for Comparing Language Experiences (with particular emphasis on: The Effect of Audience on Discourse Models)</title>
|
||||
<author><first>Andee</first><last>Rubin</last></author>
|
||||
<pdf hash="5d9a6150"/>
|
||||
<bibkey>rubin-1978-framework-comparing</bibkey>
|
||||
</paper>
|
||||
<paper id="19">
|
||||
<!-- https://aclanthology.org/T78-1019/ -->
|
||||
<title>Intentlonallty and Human Conversations</title>
|
||||
<author id="jaime-g-carbonell"><first>Jaime G.</first><last>Carbonell Jr</last></author>
|
||||
<pdf hash="c50fc619"/>
|
||||
<bibkey>carbonell-jr-1978-intentlonallty-human</bibkey>
|
||||
</paper>
|
||||
<paper id="20">
|
||||
<!-- https://aclanthology.org/T78-1020/ -->
|
||||
<title>On the Interdependence of Language and Perception</title>
|
||||
<author id="david-l-waltz"><first>David L.</first><last>Waltz</last></author>
|
||||
<pdf hash="3c708191"/>
|
||||
<bibkey>waltz-1978-interdependence-language</bibkey>
|
||||
</paper>
|
||||
<paper id="21">
|
||||
<!-- https://aclanthology.org/T78-1021/ -->
|
||||
<title>The Problem of Naming Shapes: Vision-Language Interface</title>
|
||||
<author id="ruzena-bajcsy"><first>R.</first><last>Bajcsy</last></author>
|
||||
<author id="aravind-joshi"><first>A. K.</first><last>Joshi</last></author>
|
||||
<pdf hash="50331b4e"/>
|
||||
<bibkey>bajcsy-joshi-1978-problem-naming</bibkey>
|
||||
</paper>
|
||||
<paper id="22">
|
||||
<!-- https://aclanthology.org/T78-1022/ -->
|
||||
<title>An Argument about the Composition of Conceptual Structure</title>
|
||||
<author><first>Ray</first><last>Jackendoff</last></author>
|
||||
<pdf hash="57d001f8"/>
|
||||
<bibkey>jackendoff-1978-argument-composition</bibkey>
|
||||
</paper>
|
||||
<paper id="23">
|
||||
<!-- https://aclanthology.org/T78-1023/ -->
|
||||
<title>On the Ontological Status of Visual Mental Images</title>
|
||||
<author><first>Stephen Michael</first><last>Kosslyn</last></author>
|
||||
<pdf hash="fd06e265"/>
|
||||
<bibkey>kosslyn-1978-ontological-status</bibkey>
|
||||
</paper>
|
||||
<paper id="24">
|
||||
<!-- https://aclanthology.org/T78-1024/ -->
|
||||
<title>What has language to do with perception? Some speculations on the <i>Lingua Mentis</i></title>
|
||||
<author><first>Zenon W.</first><last>Pylyshyn</last></author>
|
||||
<pdf hash="a2f3a044"/>
|
||||
<bibkey>pylyshyn-1978-language-perception</bibkey>
|
||||
</paper>
|
||||
<paper id="25">
|
||||
<!-- https://aclanthology.org/T78-1025/ -->
|
||||
<title>Semantic Primitives in Language and Vision</title>
|
||||
<author id="yorick-wilks"><first>Yorick</first><last>Wilks</last></author>
|
||||
<pdf hash="b4e13896"/>
|
||||
<bibkey>wilks-1978-semantic-primitives</bibkey>
|
||||
</paper>
|
||||
<paper id="26">
|
||||
<!-- https://aclanthology.org/T78-1026/ -->
|
||||
<title>A Note on Partial Match of Descriptions. Can One Simultaneously Question (Retrieve) and Inform (Update)?</title>
|
||||
<author id="aravind-joshi"><first>Aravind K.</first><last>Joshi</last></author>
|
||||
<pdf hash="74969243"/>
|
||||
<bibkey>joshi-1978-note-partial</bibkey>
|
||||
</paper>
|
||||
<paper id="27">
|
||||
<!-- https://aclanthology.org/T78-1027/ -->
|
||||
<title>With a Spoon in Hand This Must Be the Eating Frame</title>
|
||||
<author><first>Eugene</first><last>Charniak</last></author>
|
||||
<pdf hash="f1be68bb"/>
|
||||
<bibkey>charniak-1978-spoon-hand</bibkey>
|
||||
</paper>
|
||||
<paper id="28">
|
||||
<!-- https://aclanthology.org/T78-1028/ -->
|
||||
<title>Fragments of a Theory of Human Plausible Reasoning</title>
|
||||
<author><first>Allan</first><last>Collins</last></author>
|
||||
<pdf hash="67b4bca2"/>
|
||||
<bibkey>collins-1978-fragments-theory</bibkey>
|
||||
</paper>
|
||||
<paper id="29">
|
||||
<!-- https://aclanthology.org/T78-1029/ -->
|
||||
<title>Indirect Responses to Loaded Questions</title>
|
||||
<author><first>S. Jerrold</first><last>Kaplan</last></author>
|
||||
<pdf hash="135037f9"/>
|
||||
<bibkey>kaplan-1978-indirect-responses</bibkey>
|
||||
</paper>
|
||||
<paper id="30">
|
||||
<!-- https://aclanthology.org/T78-1030/ -->
|
||||
<title>On Reasoning by Default</title>
|
||||
<author><first>Raymond</first><last>Reiter</last></author>
|
||||
<pdf hash="0de53d33"/>
|
||||
<bibkey>reiter-1978-reasoning-default</bibkey>
|
||||
</paper>
|
||||
<paper id="31">
|
||||
<!-- https://aclanthology.org/T78-1031/ -->
|
||||
<title>Path-Based and Node-Based Inference in Semantic Networks</title>
|
||||
<author id="stuart-c-shapiro"><first>Stuart C.</first><last>Shapiro</last></author>
|
||||
<pdf hash="2ddc5535"/>
|
||||
<bibkey>shapiro-1978-path-based</bibkey>
|
||||
</paper>
|
||||
<paper id="32">
|
||||
<!-- https://aclanthology.org/T78-1032/ -->
|
||||
<title>The Representation of Derivable Information in Memory: When What Might Have Been Left Unsaid Is Said</title>
|
||||
<author><first>Rand J.</first><last>Spiro</last></author>
|
||||
<author><first>Joseph</first><last>Esposito</last></author>
|
||||
<author><first>Richard J.</first><last>Vondruska</last></author>
|
||||
<pdf hash="7758fb45"/>
|
||||
<bibkey>spiro-etal-1978-representation-derivable</bibkey>
|
||||
</paper>
|
||||
<paper id="33">
|
||||
<!-- https://aclanthology.org/T78-1033/ -->
|
||||
<title>A Heuristic for Paradigms</title>
|
||||
<author><first>Joseph E.</first><last>Grimes</last></author>
|
||||
<pdf hash="53b42c16"/>
|
||||
<bibkey>grimes-1978-heuristic-paradigms</bibkey>
|
||||
</paper>
|
||||
<paper id="34">
|
||||
<!-- https://aclanthology.org/T78-1034/ -->
|
||||
<title>A Computational Account of Some Constraints on Language</title>
|
||||
<author id="mitch-marcus"><first>Mitchell</first><last>Marcus</last></author>
|
||||
<pdf hash="c9724c43"/>
|
||||
<bibkey>marcus-1978-computational-account</bibkey>
|
||||
</paper>
|
||||
<paper id="35">
|
||||
<!-- https://aclanthology.org/T78-1035/ -->
|
||||
<title>Remarks on Processing, Constraints, and the Lexicon</title>
|
||||
<author id="thomas-wasow"><first>Thomas</first><last>Wasow</last></author>
|
||||
<pdf hash="8beed275"/>
|
||||
<bibkey>wasow-1978-remarks-processing</bibkey>
|
||||
</paper>
|
||||
<paper id="36">
|
||||
<!-- https://aclanthology.org/T78-1036/ -->
|
||||
<title>List of questions suggested for consideration in each session</title>
|
||||
<pdf hash="1950b908"/>
|
||||
<bibkey>nn-1978-list-questions</bibkey>
|
||||
</paper>
|
||||
</volume>
|
||||
</collection>
|
||||
@@ -0,0 +1,85 @@
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from paper_atlas.collectors import aamas, aamas2025
|
||||
|
||||
|
||||
FIXTURE = Path(__file__).parent/'fixtures/aamas2025'
|
||||
|
||||
|
||||
class Aamas2025Tests(unittest.TestCase):
|
||||
def parse(self, html=None):
|
||||
return aamas2025.parse_contents(html or (FIXTURE/'contents.html').read_text(),
|
||||
{'observed_at': '2026-11-09T12:31:50+00:00', 'url': aamas2025.URLS['contents']})
|
||||
|
||||
def test_real_toc_preserves_every_title_author_and_affiliation(self):
|
||||
expected = json.loads((FIXTURE/'expected-authors.json').read_text())
|
||||
self.assertEqual(hashlib.sha256((FIXTURE/'contents.html').read_bytes()).hexdigest(), expected['source_sha256'])
|
||||
rows, excluded = self.parse()
|
||||
self.assertEqual((len(rows), len(excluded)), (476, 3))
|
||||
by_url = {r['paper_url']: r for r in expected['entries']}
|
||||
for row in rows:
|
||||
original = by_url[row['metadata']['paper_url']]
|
||||
self.assertEqual(row['title'], original['title'])
|
||||
self.assertEqual(row['authors'], [a['name'] for a in original['authors']])
|
||||
self.assertEqual(row['metadata']['author_affiliations'], [a['affiliation'] for a in original['authors']])
|
||||
self.assertIsNone(row['abstract'])
|
||||
self.assertEqual(row['identifiers'], {})
|
||||
self.assertEqual(row['status'], 'accepted')
|
||||
self.assertNotIn('decision_date', row)
|
||||
self.assertTrue(all(t['namespace'] == 'aamas_proceedings:publication_format' for t in row['tags']))
|
||||
|
||||
def test_publication_format_and_track_are_separate(self):
|
||||
rows, excluded = self.parse()
|
||||
groups = {k: sum(r['track'] == k for r in rows) for k in {r['track'] for r in rows}}
|
||||
self.assertEqual(groups, {'proceedings_unspecified': 409, 'blue_sky': 12, 'doctoral': 25, 'demo': 21, 'jaamas': 9})
|
||||
competition = next(r for r in rows if r['external_id'] == '2025/p3000')
|
||||
self.assertEqual(competition['year'], 2025)
|
||||
self.assertEqual(competition['metadata']['publication_format'], 'competition_report')
|
||||
self.assertEqual(competition['metadata']['section_heading'], 'Demo Track')
|
||||
self.assertEqual(competition['track'], 'proceedings_unspecified')
|
||||
self.assertEqual(len({r['external_id'] for r in rows}), 476)
|
||||
self.assertEqual([r['publisher_page'] for r in excluded], [1, 3, 4])
|
||||
|
||||
def test_refuses_changed_edition_inventory_or_author_markup(self):
|
||||
original = (FIXTURE/'contents.html').read_text()
|
||||
for html in [original.replace('AAMAS 2025:', 'AAMAS 2026:', 1),
|
||||
original.replace('../pdfs/p5.pdf', '../pdfs/p14.pdf', 1),
|
||||
original.replace('Paper Track</font>', 'Workshop Track</font>'),
|
||||
original.replace('Amir Abolfazli', '', 1),
|
||||
original.replace('[COMP24]', '[CHANGED]', 1)]:
|
||||
with self.subTest(change=html[:60]):
|
||||
with self.assertRaises(ValueError): self.parse(html)
|
||||
|
||||
def test_frozen_collection_is_partial_and_never_fetches_pdfs(self):
|
||||
with tempfile.TemporaryDirectory() as tmp, patch.object(aamas2025, 'fetch', side_effect=AssertionError('No HTTP in frozen mode')):
|
||||
out = Path(tmp)/'out'
|
||||
report = aamas.collect(2025, out, snapshots=FIXTURE)
|
||||
self.assertEqual(report['records'], 476)
|
||||
self.assertFalse(report['database_written'])
|
||||
self.assertFalse(report['pdfs_downloaded'])
|
||||
self.assertEqual(len(report['files']), 5)
|
||||
for m in report['files']:
|
||||
self.assertEqual(m['status'], 'partial')
|
||||
self.assertEqual(m['missing_abstract_count'], m['count'])
|
||||
self.assertEqual(hashlib.sha256(Path(m['file']).read_bytes()).hexdigest(), m['file_sha256'])
|
||||
receipt = report['source_receipts'][0]['response_receipt']
|
||||
self.assertEqual(receipt['at_precision'], 'filesystem_write_time_recovered')
|
||||
self.assertTrue((out/'aamas-proceedings-2025.json').is_file())
|
||||
|
||||
def test_rejects_tampered_or_unsafe_frozen_response(self):
|
||||
import shutil
|
||||
for mode in ['tamper', 'path', 'status']:
|
||||
with self.subTest(mode=mode), tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp); source = root/'source'; shutil.copytree(FIXTURE, source)
|
||||
p = source/'receipts.json'; r = json.loads(p.read_text())
|
||||
if mode == 'tamper': (source/'contents.html').write_text('bad')
|
||||
elif mode == 'path': r['receipts'][0]['file'] = '../index.html'
|
||||
else: r['receipts'][0]['http_status'] = 403
|
||||
p.write_text(json.dumps(r))
|
||||
with self.assertRaises(ValueError): aamas.collect(2025, root/'out', snapshots=source)
|
||||
self.assertFalse((root/'out/aamas-proceedings-2025.json').exists())
|
||||
@@ -99,7 +99,7 @@ class AamasProceedingsTests(unittest.TestCase):
|
||||
aamas.collect(2012, root/'out', snapshots=source)
|
||||
self.assertFalse((root/'out/aamas-proceedings-2012.json').exists())
|
||||
with self.assertRaises(ValueError):
|
||||
aamas.collect(2025, root/'future', snapshots=source)
|
||||
aamas.collect(2026, root/'future', snapshots=source)
|
||||
self.assertFalse((root/'future').exists())
|
||||
|
||||
def test_ids_colliding_across_sections_do_not_produce_a_completed_collection(self):
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
"""Pinned T78 evidence, edition-local aliases and an explicitly identified material."""
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
from paper_atlas.collectors.acl import collect_xml, volume_scope
|
||||
from paper_atlas.collectors.acl_historical import event_scope, material_reason
|
||||
from paper_atlas.db.store import Store
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
FIXTURE = Path(__file__).parent/'fixtures/acl-t78.xml'
|
||||
COMMIT = 'a7be623d01c48c61405dec31c871121cdbed5a01'
|
||||
|
||||
|
||||
class AclHistoricalTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.root = ET.parse(FIXTURE).getroot()
|
||||
self.volume = self.root.find('volume')
|
||||
self.meta = self.volume.find('meta')
|
||||
|
||||
def test_frozen_volume_matches_official_snapshot(self):
|
||||
self.assertEqual(hashlib.sha256(FIXTURE.read_bytes()).hexdigest(), 'f712f94cff0cba05f786b7ac31dad2ea9408fde2a17a04531130df623f57dd84')
|
||||
self.assertEqual(len(self.volume.findall('paper')), 36)
|
||||
self.assertEqual(volume_scope(self.meta, '1', 'T78'), ('ACL+TINLAP', ['ACL', 'TINLAP'], 'proceedings_unspecified'))
|
||||
|
||||
def test_unknown_year_collection_or_volume_is_not_globally_aliased(self):
|
||||
for collection, volume in [('T75', '1'), ('T87', '1'), ('T78', '2'), ('C65', '1')]:
|
||||
with self.subTest(collection=collection, volume=volume):
|
||||
self.assertIsNone(event_scope(self.meta, volume, collection))
|
||||
self.assertIsNone(volume_scope(self.meta, volume, collection))
|
||||
self.assertIsNone(volume_scope(self.meta, '1'))
|
||||
|
||||
def test_known_volume_scope_rejects_critical_metadata_drift(self):
|
||||
for field, replacement in [('venue', 'acl'), ('year', '1979'), ('booktitle', 'Another TINLAP'), ('editor/first', 'Another')]:
|
||||
changed = copy.deepcopy(self.meta)
|
||||
changed.find(field).text = replacement
|
||||
with self.subTest(field=field), self.assertRaises(ValueError):
|
||||
event_scope(changed, '1', 'T78')
|
||||
changed = copy.deepcopy(self.meta)
|
||||
ET.SubElement(changed, 'year').text = '1978'
|
||||
with self.assertRaises(ValueError):
|
||||
event_scope(changed, '1', 'T78')
|
||||
|
||||
def test_evidence_retains_original_series_dates_and_role_boundary(self):
|
||||
scope = event_scope(self.meta, '1', 'T78')
|
||||
self.assertEqual(scope['original_venue'], 'TINLAP')
|
||||
self.assertEqual(scope['event_dates'], ['1978-07-25', '1978-07-27'])
|
||||
self.assertEqual([e['pdf_pages'] for e in scope['evidence']], [[9, 10], [3]])
|
||||
self.assertIn('not identified', scope['track_scope_note'])
|
||||
self.assertIn('pending', scope['evidence'][0]['scope'])
|
||||
|
||||
def test_only_exact_session_questions_material_is_excluded(self):
|
||||
paper = copy.deepcopy(self.volume.findall('paper')[-1])
|
||||
self.assertIn('Session question list', material_reason('T78', '1', paper))
|
||||
self.assertIsNone(material_reason('C65', '1', paper))
|
||||
self.assertIsNone(material_reason('T78', '2', paper))
|
||||
paper.set('id', '37')
|
||||
self.assertIsNone(material_reason('T78', '1', paper))
|
||||
paper.set('id', '36')
|
||||
ET.SubElement(paper, 'author')
|
||||
with self.assertRaises(ValueError):
|
||||
material_reason('T78', '1', paper)
|
||||
|
||||
def test_all_authored_source_fields_and_material_receipt_survive_collection(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
manifest, = collect_xml(FIXTURE, temporary, COMMIT)
|
||||
rows = [json.loads(raw) for raw in Path(manifest['file']).read_text().splitlines()]
|
||||
self.assertEqual(manifest['count'], 35)
|
||||
self.assertEqual(manifest['xml_paper_elements'], 36)
|
||||
self.assertEqual(manifest['missing_abstract_count'], 35)
|
||||
self.assertEqual(manifest['excluded_materials'][0]['external_id'], 'T78-1036')
|
||||
for number, row in enumerate(rows, 1):
|
||||
paper = self.volume.findall('paper')[number-1]
|
||||
self.assertEqual(row['external_id'], f'T78-1{number:03d}')
|
||||
self.assertEqual(row['title'], ' '.join(''.join(paper.find('title').itertext()).split()))
|
||||
self.assertEqual(row['metadata']['venue_codes'], ['tinlap'])
|
||||
self.assertEqual(row['metadata']['historical_event_scope']['original_venue'], 'TINLAP')
|
||||
self.assertEqual(row['metadata']['metadata_commit'], COMMIT)
|
||||
self.assertEqual((row['status'], row['track'], row['abstract']), ('accepted', 'proceedings_unspecified', None))
|
||||
|
||||
def test_old_coling_scope_is_unchanged_by_calendar_replacement_evidence(self):
|
||||
meta = ET.fromstring('<meta><venue>coling</venue><year>1965</year><booktitle>COLING 1965</booktitle></meta>')
|
||||
self.assertEqual(volume_scope(meta, '1', 'C65'), ('COLING', ['COLING'], 'main'))
|
||||
self.assertIsNone(event_scope(meta, '1', 'C65'))
|
||||
|
||||
def test_native_ingestion_does_not_alias_other_tinlap_years_or_infer_negative_status(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
manifest, = collect_xml(FIXTURE, root/'source', COMMIT)
|
||||
db = root/'sample.sqlite'
|
||||
store = Store(db)
|
||||
pid, rid = store.record('test_control', 'tinlap1975', 'https://example.org/tinlap1975',
|
||||
'An earlier TINLAP contribution', None, ['Jane Doe'])
|
||||
store.decision(pid, rid, 'TINLAP', 1975, 'main', 'tinlap1975', 'https://example.org/tinlap1975', 'accepted', 'test control')
|
||||
store.conn.commit()
|
||||
store.conn.close()
|
||||
ingest_file(db, manifest['file'])
|
||||
ingest_file(db, manifest['file'])
|
||||
with sqlite3.connect(db) as c:
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0], 36)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM submissions').fetchone()[0], 36)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM decisions').fetchone()[0], 36)
|
||||
self.assertEqual(c.execute("SELECT count(*) FROM decisions WHERE status!='accepted'").fetchone()[0], 0)
|
||||
self.assertIsNone(c.execute("SELECT 1 FROM source_records WHERE external_id='T78-1036'").fetchone())
|
||||
aliases = c.execute('SELECT v.acronym,a.alias FROM venue_aliases a JOIN venues v ON v.id=a.venue_id ORDER BY a.alias').fetchall()
|
||||
self.assertEqual(aliases, [('ACL+TINLAP', 'ACL'), ('ACL+TINLAP', 'TINLAP')])
|
||||
metadata = json.loads(c.execute("SELECT metadata_json FROM source_records WHERE external_id='T78-1001'").fetchone()[0])
|
||||
self.assertEqual(metadata['historical_event_scope']['event_year'], 1978)
|
||||
self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
Reference in New Issue
Block a user