docs: set 2020 backfill scope and preserve recovery inventory since 2000

Reviewed-on: #42

.gitea/workflows/check.yml

@@ -52,6 +52,7 @@ jobs:
               subprocess.run([python,'-I','-B','-m','paper_atlas','collect','aamas-proceedings','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','collect','aamas-proceedings','--year','2025','--snapshots',str(src/'tests/fixtures/aamas2025'),'--out',str(root/'installed-aamas2025')],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','collect','iclr-archive','--help'],cwd=outside,check=True)
+              subprocess.run([python,'-I','-B','-m','paper_atlas','collect','acl','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','postgres','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','edition-tracks','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','joint-editions','--help'],cwd=outside,check=True)

README.md

@@ -6,10 +6,14 @@ Paper Atlas 将 arXiv、AI 会议论文集及公开投稿决定整理到可追
 
 项目仍在建设中。候选范围包含 43 个会议（含 ACM MM、MLSys），尚未完成全部历史数据和最终去重复核。实际进度以数据库、来源清单和覆盖报告为准，而不是仓库文件数量。
 
+当前主动补漏从 **2020 年**起；2000 年起已发现的名单、缺口与待办保存在[补漏恢复清单](docs/coverage/backfill-handoff-2026-11-10.md)，下次从这里接续。已有旧数据保留，范围排除不算补齐。
+
 历史 ICLR 2014–2016 官方录用页由 `collect iclr-archive` 适配，保留主会/workshop及重复展示证据；运行和身份核验边界见 [历史档案说明](docs/coverage/iclr-archive-2014-2016.md)。
 
 ## 项目结构
 
+ACL 1978 的年会与 TINLAP-2 来源关系按精确卷适配，保留原系列和不明逐篇角色；1965 的替代关系不转成全局别名。来源、排除条目和恢复边界见 [ACL 早期来源核查](docs/coverage/acl-early-sources-2026-11-09.md)。
+
 ```text
 paper_atlas/
   cli.py                 统一命令入口

docs/coverage/README.md

@@ -1,5 +1,7 @@
 # 会议覆盖与公开决定核查
 
+**当前工作范围：2020 年起主动补漏。** 2000 年起已发现资料保留，恢复工作从 [补漏范围与恢复清单（2026-11-10）](backfill-handoff-2026-11-10.md) 开始；下文旧日期快照不改写，较早年份排除不算补齐。
+
 这是 2026-11-08 的首份核查基线，**完整逐年政策普查和最终去重仍未完成**。收录 43 个目标，不能把“有记录”当作“完整覆盖”。结构化清单见 [inventory-2026-11-08.json](inventory-2026-11-08.json)。清单中的 `policy_checks` 均限定所列年份和轨道；空数组表示尚未核实，绝不表示没有公开拒稿。
 
 ## 快照与复核

docs/coverage/acl-early-sources-2026-11-09.md

@@ -0,0 +1,52 @@
+# ACL 早期年份：会议存在、出版目录与逐篇资料分开核验
+
+固定普查中 ACL 1963–1978 仍有年份缺口。不能因为会议年表连续，就假设每年都有独立的正式论文集；也不能因为当前目录始于 1979，就断言此前没有摘要、通知或其他公开资料。本次不修改固定的 1,266 个目标年份基准。
+
+## 1978 年：限定 T78-1 的来源关系
+
+[同期年会通知](https://aclanthology.org/J77-4001.pdf)的 PDF 第 9–10 页（印刷页码 9–10）明确说明，1978 年 ACL 年会采用第二届 Theoretical Issues in Natural Language Processing 的形式，地点为 Urbana/Champaign，日期为 7 月 25–27 日。通知中的 ACM 联合赞助确认仍处于待确认状态，不能将该预告改写为已完成的行政决定。
+
+[1982 年 Donald E. Walker 的历史回顾](https://aclanthology.org/www.mt-archive.info/70/ACL-1982-Walker-1.pdf)第 3 页（印刷页码 91）的年会表及缩写说明，独立对应了 1978 年的时间、地点与 TNLP/TINLAP-2 关系。两份扫描件相关页已渲染并逐页目视核对，不仅依靠 OCR。
+
+[官方卷目录](https://aclanthology.org/volumes/T78-1/)与固定 XML `T78.xml` 保留原系列名 TINLAP、完整卷名、1978 年及编辑 David L. Waltz。适配仅接受精确的 collection `T78`、volume `1` 和上述字段组合；关键元数据变化会拒绝处理，等待重新核查。
+
+- 规范事件名使用 `ACL+TINLAP`，原系列名 `TINLAP`、原始 XML 的 `venue_codes: [tinlap]` 及证据均保留。
+- ACL/TINLAP 别名只挂在这一规范事件系列上，不把所有 TINLAP 年份都算成 ACL。T75、T87 及 T78 其他卷没有此例外。
+- 通知描述了邀请与提交论文两种参与方式，XML 没有逐篇角色。统一暂存为 `proceedings_unspecified`，不猜测每篇属于 invited、main 或 workshop。
+- XML 共 36 条条目，其中 T78-1036 是无作者的场次讨论问题列表。精确识别并在 manifest 的 `excluded_materials` 保留其标识、标题与原因；其余 35 条作为具名贡献整理。该排除不推广到所有无作者条目，已识别条目发生变化也须重新核查。
+- 全部 35 条 XML 均无摘要。出版目录中的 accepted 是来源观察；通知提及录用／不录用通知时间，不提供任何可识别的拒稿决定，不能制造 rejected 或 withdrawn。
+
+采集器仅生成元数据文件，不写业务数据库。真实来源逐条验收、PostgreSQL 身份预检、串行导入与独立验收仍是后续门槛。35 条来源不等于 35 个新增论文实体，更不等于完成全库去重。
+
+## 1965 年：替代关系不改写为别名
+
+Walker 回顾第 1 页（印刷页码 89）说明，1965 年 ICCL（后来称 COLING）替代了当年的年会。[官方 C65-1 目录](https://aclanthology.org/volumes/C65-1/)及 XML 仍以 COLING 标记该卷。这是限定年份的历史关系，不证明 COLING 与 ACL 是两个可全局互换的名字。
+
+本次保留 C65 原有 `COLING` 归属，不添加 ACL 全局别名、不重放已导入记录、不将某一年的日历关系推广到整个系列，也不据此悄悄减少固定普查缺口。其他 1963–1977 年资料仍须逐年核查；本次不能证明这些年份整体属于非归档例外。
+
+## 固定来源与恢复
+
+服务器证据目录：`/data/paper-index/collected/acl-early-source-evidence-v1`。五份官方响应均 HTTP 200，另有两份来自既有官方 XML checkout 的文件，已与 Git 对象逐字节核对。未下载论文全文集合；两份 PDF 是本次历史／行政取证资料，不进入论文全文库。
+
+| 文件 | SHA256 |
+|---|---|
+| `acl-venue.html` | `c60f64cb4d72e677aeb84a5844cde28a8d50579e1995f08e8c7801eed1e3fac5` |
+| `acl-history-1982.pdf` | `35c21bd4983975bcaccbb94bd8ab91c9019fc190feec1791255407cf18a7e49a` |
+| `acl-1978-notice.pdf` | `14b5deb471c55afb07d789529b95d0462a2a1a77df3dfba7fa1f5d3ab75eb411` |
+| `tinlap1978-volume.html` | `7e20d709f72f415819e4d17e3674b19dc7d871eabcf0cf8b4935f0392d775729` |
+| `coling1965-volume.html` | `0d2f2f49f2a311f9c0b80a8832f6476eb32f94f35abf36f6d2f2e10063ac880f` |
+| `T78.xml` | `f712f94cff0cba05f786b7ac31dad2ea9408fde2a17a04531130df623f57dd84` |
+| `C65.xml` | `572791610e20b7c6045d984d968e73a18a144bcb5c3da64112001c781826808e` |
+
+XML commit：`a7be623d01c48c61405dec31c871121cdbed5a01`。测试 fixture `tests/fixtures/acl-t78.xml` 是该原始文件，不是手工模拟论文；不包含 PDF。PDF 哈希建立完整性，不单独证明语义，语义依据是上述具体页面。
+
+在服务器使用既有命令，固定 checkout 并输出到新目录：
+
+```sh
+paper-atlas collect acl \
+  --checkout /data/paper-index/raw/metadata-checkouts/acl-anthology \
+  --collection T78 \
+  --out /data/paper-index/collected/acl-t78-source-v1
+```
+
+该命令会校验／更新官方 metadata checkout，实际 commit 必须以生成清单为准；严格重现本次取证时，使用已冻结的 XML 和上述 commit，不把更新后的内容冒充旧快照。不要将无 collection 限制的全部历史输出直接作为增量队列。采集完成不代表导入完成，恢复前先检查当前单元和报告，不因日志安静重启已执行的写入。

docs/coverage/backfill-handoff-2026-11-10.md

@@ -0,0 +1,52 @@
+# 补漏范围与恢复清单（2026-11-10）
+
+用户确定：**后续主动补漏从 2020 年起**。2000 年起已经发现的名单、缺口、资料组合和身份核查记录保留在项目中，供下次接着做；已有旧数据继续保留。停止补采的年份标为范围外或延期，不算作已补齐。
+
+本页是下一次工作的入口，结构化记录见 [backfill-handoff-2026-11-10.json](backfill-handoff-2026-11-10.json)。它保存 43 个会议的 2000–2026 已观察年份、缺年、官方目录文件入口、近期待办与恢复证据。旧的日期快照不改写，也不能当实时数据库。
+
+## 当前范围和年份差集
+
+依据生产只读快照 `2026-11-09 16:26:41.688509+00:00`，原全历史口径为 1266 个目标年份槽位：1208 已观察、43 个历史缺年、14 个 2026 缺年，以及 1 个 SysML 2018 非存档例外。原 135 个历史缺口中有 92 个通过数据补入解决；其余 43 个现在停止补采，不能因此宣称全部解决。
+
+- **2020–2025**：没有整年完全缺记录的目标槽位，但逐篇、track、摘要、公开负面决定仍不完整。
+- **2026**：NeurIPS、ECCV、RecSys、CIKM、ICDM、SDM、IROS、RSS、CoRL、ICAPS、AAMAS、ICASSP、ISMIR、ACM MM，共 14 个槽位。须逐个核查是否已经公开出版；不能将尚未发布的资料说成漏采。
+- **2000–2019 延期线索**：WWW 2000、ICAPS 2001 两个整年缺口；官方历届记录分别见 [WWW](www-editions-2026-11-08.json)、[ICAPS](icaps-editions-2026-11-08.json)。SysML 2018 是非存档首届，见 [MLSys](mlsys-editions-2026-11-08.json)，不能制造正式出版记录。
+- **2000 年前**：剩余 41 个历史缺年停止补采。旧来源文件继续保留作证据，不再作为当前工作队列。
+
+以上只判断目标会议年份是否至少有一条投稿记录，绝不等于目录完整、独立论文数量或最终去重验收。
+
+## 近期资料与恢复位置
+
+服务器采集根目录为 `/data/paper-index/collected`，本地证据备份为 `/Users/zzyang/Documents/Codex/2026-11-06/paper-index`。这些目录提供大体积原始文件，仓库 JSON 提供路径、状态和可用哈希；恢复前先验证文件存在、哈希和生产实际状态。
+
+| 资料 | 当前状态 | 下一步与证据入口 |
+| --- | --- | --- |
+| AAMAS 2026 | 官方目录 643 条，3 个 keynote 排除；640 个非 keynote 中含 1 个明确撤稿。尚未导入 | `aamas2026-primary-evidence-v1/` 保存官方 HTML、回执、逐项目录、审计和校验清单。八个标题差异/撤稿前缀条目需核查。先扩展正式 collector 并远程验证，再做新 PostgreSQL 身份预检。 |
+| CVPR 2026 Findings | 941 条摘要已核验，尚未导入 | `cvf-2026-verified/recovery-audit.json`；原 404 与 RVLF/FVLF 路径更正证据保留。先核对 manifests、track 和实体关系。 |
+| AAMAS 2025 | 476 个非 keynote 来源中，240 已验收、236 待身份核查 | `aamas2025-import225-v1/` 与 `aamas2025-import15-v1/` 是已消费批次，禁止重放。15 条最近验收报告哈希见 JSON。余下资料从当前身份候选与新只读基线继续。 |
+| RSS 2025 | 163 条目录中 162 已验收，`2025/p117` 待核查 | 保留 spinal/RMCC 关系问题；查看工作状态键 `rss2025_held7_primary_evidence_v1`，不能重跑已消费的 6/88/54/14 批次。 |
+| 近期公开负面决定首批 | 1625 条已验收：1624 rejected、1 withdrawn | `public-negative-postgresql-import-postcheck-v1.json`；仅该批次验收，不是公开拒稿全集，不要重放。 |
+
+AAMAS 2026 官方目录分区：research 338、extended abstracts 185、AAAI 19、Blue Sky 17、doctoral 34、demo 44、JAAMAS 3，另有 keynote 3。JAAMAS 是期刊展示关系，不能凭目录再造一次独立审稿录用。官方撤稿条目在 research 分区，不能把全目录都标为当前 accepted。
+
+AAAI 转投政策有年份冲突：[详细征稿页](https://cyprusconferences.org/aamas2026/call-for-papers-aaai-track/) 指 AAAI-26 rejected，[总览](https://cyprusconferences.org/aamas2026/calls/) 写 AAAI-25。保留两份证据；19 个 AAAI track 条目不能直接转成 19 条有原始 note 的拒稿决定。原投稿 ID、逐篇决定和版本关系仍需核实。
+
+## 2000–2019 已发现但延期的身份与名单工作
+
+- **AAMAS 2012**：官方 334 条，319 已独立验收、15 个身份条目保留待查。验收报告 `aamas2012-import-v1/postgresql-postcheck-v1.json`，哈希在 JSON；已验收 319 条及被替代的 316 条计划均不得重放。来源布局与用法见 [AAMAS 适配说明](../aamas-proceedings.md)。
+- **UAI 2008**：官方 programme 与归档资料确认了四篇独立论文，但部分原始 DBLP 键的标题、作者、页码、出版链接对应仍有冲突。JSON 保留四组候选、arXiv ID、官方题名及归档报告哈希。之前仅根据页码推断题名/作者错误的结论已撤回，未应用纠正；恢复时不得据此覆盖或合并。
+- **其他名单**：本目录全部 `*-editions-2026-11-08.json` 保留各会议官方年份及来源；[历史缺口](historical-gaps-2026-11-09.md)、[来源候选](historical-source-candidates-2026-11-09.json)、[来源范围](historical-source-scope-2026-11-09.md)、[主系列](historical-primary-series-2026-11-09.md)、[前身会议](historical-predecessors-2026-11-09.md)、[ICLR 档案](iclr-archive-2014-2016.md)继续留档。这些旧报告的“缺失”可能已被后续导入解决，恢复时必须对照当前状态，不能直接当可执行队列。
+
+## 公开拒稿资料不能遗漏
+
+结构化清单保留最近已查的政策 URL、限定年份和可用性结论。优先继续 UAI 2024–2026 的 opt-in 公开拒稿来源、EMNLP 2023 的公开集合范围及其他目标的逐年核查；平台使用 OpenReview 不能证明 rejected 公开。访问受限如实记录，不绕过 Challenge。
+
+保留 rejected、desk rejected、withdrawn、retracted、resubmit 的区别；评审建议不等于最终决定。COLM 2026 的不公开政策只适用于已核实年份，不能推广到其他会议。PaperCopilot 与原 OpenReview、镜像间有重叠，候选计数不能相加。论文先被 A 拒绝再被 B 录用时，两次投稿、决定、来源与证据全部保留。
+
+## 下次启动顺序
+
+1. 读取本页和 JSON，再核实服务器工作状态、明确的 systemd invocation 与终态报告。权威工作状态 `/Users/zzyang/Documents/Codex/2026-11-06/paper-index/work-state.json` 中部分旧字段会过期；以最新验收和实际查询为准。
+2. 当前主动范围为 2020 起，2000–2019 只恢复线索，不自动启动其采集。先检查新来源是否已导入，验收过的队列禁止重放。
+3. 使用现有 `foundry-postgres` 的 `foundry_data.research`。当前已验证 writer 为提交 `714f745bd5c3fbc742be1efae0b31ab37a55e21f` 的固定 release；恢复时重新确认，不使用旧 `/data/paper-index/code` 替代。服务配置仅由服务器受保护文件读取，不复制密码。
+4. 对待导入资料做新只读身份预检，处理稳定标识、短标题、作者/版本差异、track 与跨会历史，再生成小范围增量队列。每批独立验收，SQLite 只读备份保持保护。
+5. 编译、测试或 wheel 构建仅在已确认 `ces-control:host` runner 执行。代码按工作分支交付 PR，等待用户手动合并。范围内导入结束后仍须全库去重、近似标题/作者冲突、来源-年份-track、摘要、约束、实际全文查询验收；不能用单批导入或 CI 成功替代。

paper_atlas/collectors/acl.py

@@ -6,6 +6,7 @@ import xml.etree.ElementTree as ET
 from pathlib import Path
 
 from paper_atlas.io.git_metadata import checkout_metadata
+from paper_atlas.collectors.acl_historical import event_scope, material_reason
 
 VENUES={'acl':'ACL','emnlp':'EMNLP','naacl':'NAACL','eacl':'EACL','coling':'COLING','conll':'CoNLL'}
 
@@ -29,7 +30,10 @@ def paper_id(collection,volume,number):
     return f'{collection}-{int(volume):0{width}d}{int(number):0{digits}d}'
 
 
-def volume_scope(meta,volume):
+def volume_scope(meta,volume,collection=None):
+    historical=event_scope(meta,volume,collection)
+    if historical:
+        return historical['canonical_venue'],historical['venue_aliases'],historical['track']
     codes=[text(n).lower() for n in meta.findall('venue')]
     book=text(meta.find('booktitle'))
     lower=book.lower()
@@ -59,7 +63,7 @@ def collect_xml(path,out,commit):
         vid=volume.attrib['id']
         meta=volume.find('meta')
         if meta is None:raise ValueError('Volume metadata missing')
-        scope=volume_scope(meta,vid)
+        scope=volume_scope(meta,vid,collection)
         key=f'{collection}-{vid}'
         if scope is None:
             results.append({'volume':key,'status':'excluded_by_venue','venue_codes':[text(n) for n in meta.findall('venue')],'booktitle':text(meta.find('booktitle')),'year':text(meta.find('year'))})
@@ -70,12 +74,18 @@ def collect_xml(path,out,commit):
         year=int(year_text)
         records=[]
         frontmatter=0
+        excluded_materials=[]
+        historical=event_scope(meta,vid,collection)
         for paper in volume.findall('paper'):
             pid=paper.attrib['id']
             if int(pid)==0:
                 frontmatter+=1
                 continue
             identifier=paper_id(collection,vid,pid)
+            reason=material_reason(collection,vid,paper)
+            if reason:
+                excluded_materials.append({'external_id':identifier,'title':text(paper.find('title')),'reason':reason})
+                continue
             doi=text(paper.find('doi'))
             if doi.startswith('10.18653/v1/') and doi.split('10.18653/v1/',1)[1].casefold()!=identifier.casefold():
                 raise ValueError('Official DOI disagrees with constructed Anthology ID: '+identifier)
@@ -83,6 +93,8 @@ def collect_xml(path,out,commit):
             if not title:raise ValueError('Missing paper title: '+identifier)
             authors=[{'given':text(a.find('first')),'family':text(a.find('last'))} for a in paper.findall('author')]
             records.append({'source':'acl_anthology','external_id':identifier,'url':f'https://aclanthology.org/{identifier}/','title':title,'abstract':text(paper.find('abstract')) or None,'authors':authors,'identifiers':{'acl':identifier,'doi':doi},'venue':venue,'venue_aliases':aliases if len(aliases)>1 else [],'year':year,'track':track,'status':'accepted','raw_decision':'listed in authoritative ACL Anthology proceedings XML','metadata':{'collection':collection,'volume':vid,'booktitle':text(meta.find('booktitle')),'venue_codes':[text(n) for n in meta.findall('venue')],'pages':text(paper.find('pages')),'publisher':text(meta.find('publisher')),'metadata_commit':commit,'source_xml':f'https://github.com/acl-org/acl-anthology/blob/{commit}/data/xml/{collection}.xml'}})
+            if historical:
+                records[-1]['metadata']['historical_event_scope']=historical
         if not records:
             results.append({'volume':key,'status':'unavailable','reason':'no research paper elements after excluding front matter'})
             continue
@@ -93,7 +105,7 @@ def collect_xml(path,out,commit):
             for record in records:output.write(json.dumps(record,ensure_ascii=False)+'\n')
         part.replace(target)
         missing=sum(not row['abstract'] for row in records)
-        manifest={'source':'acl_anthology','venue':venue,'year':year,'track':track,'count':len(records),'xml_paper_elements':len(volume.findall('paper')),'excluded_frontmatter':frontmatter,'missing_abstract_count':missing,'list_url':f'https://aclanthology.org/{key}/','metadata_commit':commit,'file':str(target),'status':'partial' if missing else 'collected'}
+        manifest={'source':'acl_anthology','venue':venue,'year':year,'track':track,'count':len(records),'xml_paper_elements':len(volume.findall('paper')),'excluded_frontmatter':frontmatter,'excluded_materials':excluded_materials,'missing_abstract_count':missing,'list_url':f'https://aclanthology.org/{key}/','metadata_commit':commit,'file':str(target),'status':'partial' if missing else 'collected'}
         target.with_suffix('.manifest.json').write_text(json.dumps(manifest,indent=2))
         results.append(manifest)
         print(json.dumps(manifest),flush=True)

paper_atlas/collectors/acl_historical.py

@@ -0,0 +1,45 @@
+"""Edition-specific historical scope; never alias every TINLAP year to ACL."""
+
+HISTORY_URL = 'https://aclanthology.org/www.mt-archive.info/70/ACL-1982-Walker-1.pdf'
+NOTICE_URL = 'https://aclanthology.org/J77-4001.pdf'
+BOOK = 'Theoretical Issues in Natural Language Processing-2'
+MATERIAL_TITLE = 'List of questions suggested for consideration in each session'
+
+
+def text(node):
+    return ' '.join(''.join(node.itertext()).split()) if node is not None else ''
+
+
+def event_scope(meta, volume, collection):
+    if (collection, volume) != ('T78', '1'):
+        return None
+    if ([text(n) for n in meta.findall('venue')] != ['tinlap']
+            or [text(n) for n in meta.findall('year')] != ['1978']
+            or [text(n) for n in meta.findall('booktitle')] != [BOOK]
+            or [(text(n.find('first')), text(n.find('last'))) for n in meta.findall('editor')]
+               != [('David L.', 'Waltz')]):
+        raise ValueError('Reviewed T78-1 scope metadata changed; independent review required')
+    return {
+        'collection': 'T78', 'volume': '1', 'original_venue': 'TINLAP',
+        'event_year': 1978, 'event_dates': ['1978-07-25', '1978-07-27'],
+        'relationship': 'ACL annual meeting took the form of TINLAP-2',
+        'canonical_venue': 'ACL+TINLAP', 'venue_aliases': ['ACL', 'TINLAP'],
+        'track': 'proceedings_unspecified',
+        'track_scope_note': 'Invited and submitted contributions are described at volume level; individual roles are not identified in XML.',
+        'evidence': [
+            {'url': NOTICE_URL, 'sha256': '14b5deb471c55afb07d789529b95d0462a2a1a77df3dfba7fa1f5d3ab75eb411',
+             'pdf_pages': [9, 10], 'printed_pages': ['9', '10'],
+             'scope': 'Contemporary annual meeting notice, not individual decision notes; ACM confirmation was pending in this notice.'},
+            {'url': HISTORY_URL, 'sha256': '35c21bd4983975bcaccbb94bd8ab91c9019fc190feec1791255407cf18a7e49a',
+             'pdf_pages': [3], 'printed_pages': ['91'],
+             'scope': 'Retrospective annual meeting table and TNLP abbreviation confirm the 1978 event association.'},
+        ],
+    }
+
+
+def material_reason(collection, volume, paper):
+    if (collection, volume, paper.attrib.get('id')) != ('T78', '1', '36'):
+        return None
+    if text(paper.find('title')) != MATERIAL_TITLE or paper.findall('author'):
+        raise ValueError('Reviewed T78-1036 material changed; independent review required')
+    return 'Session question list, not an authored research contribution; no decision inferred.'

tests/fixtures/acl-t78.xml

@@ -0,0 +1,273 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<collection id="T78">
+  <volume id="1" type="proceedings">
+    <!-- https://aclanthology.org/T78-1/ -->
+    <meta>
+      <booktitle>Theoretical Issues in Natural Language <fixed-case>P</fixed-case>rocessing-2</booktitle>
+      <editor id="david-l-waltz"><first>David L.</first><last>Waltz</last></editor>
+      <venue>tinlap</venue>
+      <year>1978</year>
+    </meta>
+    <frontmatter>
+      <!-- https://aclanthology.org/T78-1000/ -->
+      <pdf hash="fb6164ad"/>
+      <bibkey>tinlap-1978-theoretical</bibkey>
+    </frontmatter>
+    <paper id="1">
+      <!-- https://aclanthology.org/T78-1001/ -->
+      <title>Testing The Psychological Reality of a Representational Model</title>
+      <author><first>Dedre</first><last>Gentner</last></author>
+      <pdf hash="f723a956"/>
+      <bibkey>gentner-1978-testing-psychological</bibkey>
+    </paper>
+    <paper id="2">
+      <!-- https://aclanthology.org/T78-1002/ -->
+      <title>What Makes Something “Ad Hoc”</title>
+      <author id="roger-c-schank"><first>Roger C.</first><last>Schank</last></author>
+      <pdf hash="636853f9"/>
+      <bibkey>schank-1978-makes-something</bibkey>
+    </paper>
+    <paper id="3">
+      <!-- https://aclanthology.org/T78-1003/ -->
+      <title>The Relation of Grammar to Cognition–a Synopsis</title>
+      <author><first>Leonard</first><last>Talmy</last></author>
+      <pdf hash="101c67dc"/>
+      <bibkey>talmy-1978-relation-grammar</bibkey>
+    </paper>
+    <paper id="4">
+      <!-- https://aclanthology.org/T78-1004/ -->
+      <title>On primitives, prototypes, and other semantic anomalies</title>
+      <author><first>Terry</first><last>Winograd</last></author>
+      <pdf hash="f8e47023"/>
+      <bibkey>winograd-1978-primitives-prototypes</bibkey>
+    </paper>
+    <paper id="5">
+      <!-- https://aclanthology.org/T78-1005/ -->
+      <title>Taxonomic Lattice Structures for Situation Recognition</title>
+      <author id="william-a-woods"><first>William A.</first><last>Woods</last></author>
+      <pdf hash="92957ddf"/>
+      <bibkey>woods-1978-taxonomic-lattice</bibkey>
+    </paper>
+    <paper id="6">
+      <!-- https://aclanthology.org/T78-1006/ -->
+      <title>Description Formation and Discourse Model Synthesis</title>
+      <author id="bonnie-webber"><first>Bonnie Lynn</first><last>Webber</last></author>
+      <pdf hash="ffded1d2"/>
+      <bibkey>webber-1978-description-formation</bibkey>
+    </paper>
+    <paper id="7">
+      <!-- https://aclanthology.org/T78-1007/ -->
+      <title>The Processing of Referring Expressions within a Semantic Network</title>
+      <author><first>John R.</first><last>Anderson</last></author>
+      <pdf hash="8bcc083a"/>
+      <bibkey>anderson-1978-processing-referring</bibkey>
+    </paper>
+    <paper id="8">
+      <!-- https://aclanthology.org/T78-1008/ -->
+      <title>Reference Diaries</title>
+      <author><first>Herbert H.</first><last>Clark</last></author>
+      <author><first>Catherine</first><last>Marshall</last></author>
+      <pdf hash="d06b7ea2"/>
+      <bibkey>clark-marshall-1978-reference-diaries</bibkey>
+    </paper>
+    <paper id="9">
+      <!-- https://aclanthology.org/T78-1009/ -->
+      <title>Subsequent Reference: Syntactic and Rhetorical Constraints</title>
+      <author id="david-d-mcdonald"><first>David D.</first><last>McDonald</last></author>
+      <pdf hash="2d424559"/>
+      <bibkey>mcdonald-1978-subsequent-reference</bibkey>
+    </paper>
+    <paper id="10">
+      <!-- https://aclanthology.org/T78-1010/ -->
+      <title>Some Psycholinguistic Constraints on the Construction and Interpretation of Definite Descriptions</title>
+      <author><first>Andrew</first><last>Ortony</last></author>
+      <pdf hash="69807727"/>
+      <bibkey>ortony-1978-psycholinguistic-constraints</bibkey>
+    </paper>
+    <paper id="11">
+      <!-- https://aclanthology.org/T78-1011/ -->
+      <title>Bound Variables and Other Anaphors</title>
+      <author id="barbara-h-partee"><first>Barbara H.</first><last>Partee</last></author>
+      <pdf hash="e8e6a510"/>
+      <bibkey>partee-1978-bound-variables</bibkey>
+    </paper>
+    <paper id="12">
+      <!-- https://aclanthology.org/T78-1012/ -->
+      <title>The Use of Focus as a Tool for Disambiguation of Definite Noun Phrases</title>
+      <author id="candace-l-sidner"><first>Candace L.</first><last>Sidner</last></author>
+      <pdf hash="7b96d2b7"/>
+      <bibkey>sidner-1978-use-focus</bibkey>
+    </paper>
+    <paper id="13">
+      <!-- https://aclanthology.org/T78-1013/ -->
+      <title>Focusing in Dialog</title>
+      <author id="barbara-j-grosz"><first>Barbara J.</first><last>Grosz</last></author>
+      <pdf hash="8c8d4c2e"/>
+      <bibkey>grosz-1978-focusing-dialog</bibkey>
+    </paper>
+    <paper id="14">
+      <!-- https://aclanthology.org/T78-1014/ -->
+      <title>Topic Levels</title>
+      <author><first>Joseph E.</first><last>Grimes</last></author>
+      <pdf hash="f485b2ba"/>
+      <bibkey>grimes-1978-topic-levels</bibkey>
+    </paper>
+    <paper id="15">
+      <!-- https://aclanthology.org/T78-1015/ -->
+      <title>Toward a Rational Model of Discourse Comprehension</title>
+      <author><first>Jerry L.</first><last>Morgan</last></author>
+      <pdf hash="5e12a597"/>
+      <bibkey>morgan-1978-toward-rational</bibkey>
+    </paper>
+    <paper id="16">
+      <!-- https://aclanthology.org/T78-1016/ -->
+      <title>Assent and Compliance in Children’s Language Comprehension</title>
+      <author><first>David R.</first><last>Olson</last></author>
+      <pdf hash="708bc4a0"/>
+      <bibkey>olson-1978-assent-compliance</bibkey>
+    </paper>
+    <paper id="17">
+      <!-- https://aclanthology.org/T78-1017/ -->
+      <title>Speech Acts as a Basis for Understanding Dialogue Coherence</title>
+      <author id="c-raymond-perrault"><first>C. Raymond</first><last>Perrault</last></author>
+      <author id="james-allen"><first>James F.</first><last>Allen</last></author>
+      <pdf hash="d111402b"/>
+      <bibkey>perrault-allen-1978-speech</bibkey>
+    </paper>
+    <paper id="18">
+      <!-- https://aclanthology.org/T78-1018/ -->
+      <title>A Framework for Comparing Language Experiences (with particular emphasis on: The Effect of Audience on Discourse Models)</title>
+      <author><first>Andee</first><last>Rubin</last></author>
+      <pdf hash="5d9a6150"/>
+      <bibkey>rubin-1978-framework-comparing</bibkey>
+    </paper>
+    <paper id="19">
+      <!-- https://aclanthology.org/T78-1019/ -->
+      <title>Intentlonallty and Human Conversations</title>
+      <author id="jaime-g-carbonell"><first>Jaime G.</first><last>Carbonell Jr</last></author>
+      <pdf hash="c50fc619"/>
+      <bibkey>carbonell-jr-1978-intentlonallty-human</bibkey>
+    </paper>
+    <paper id="20">
+      <!-- https://aclanthology.org/T78-1020/ -->
+      <title>On the Interdependence of Language and Perception</title>
+      <author id="david-l-waltz"><first>David L.</first><last>Waltz</last></author>
+      <pdf hash="3c708191"/>
+      <bibkey>waltz-1978-interdependence-language</bibkey>
+    </paper>
+    <paper id="21">
+      <!-- https://aclanthology.org/T78-1021/ -->
+      <title>The Problem of Naming Shapes: Vision-Language Interface</title>
+      <author id="ruzena-bajcsy"><first>R.</first><last>Bajcsy</last></author>
+      <author id="aravind-joshi"><first>A. K.</first><last>Joshi</last></author>
+      <pdf hash="50331b4e"/>
+      <bibkey>bajcsy-joshi-1978-problem-naming</bibkey>
+    </paper>
+    <paper id="22">
+      <!-- https://aclanthology.org/T78-1022/ -->
+      <title>An Argument about the Composition of Conceptual Structure</title>
+      <author><first>Ray</first><last>Jackendoff</last></author>
+      <pdf hash="57d001f8"/>
+      <bibkey>jackendoff-1978-argument-composition</bibkey>
+    </paper>
+    <paper id="23">
+      <!-- https://aclanthology.org/T78-1023/ -->
+      <title>On the Ontological Status of Visual Mental Images</title>
+      <author><first>Stephen Michael</first><last>Kosslyn</last></author>
+      <pdf hash="fd06e265"/>
+      <bibkey>kosslyn-1978-ontological-status</bibkey>
+    </paper>
+    <paper id="24">
+      <!-- https://aclanthology.org/T78-1024/ -->
+      <title>What has language to do with perception? Some speculations on the <i>Lingua Mentis</i></title>
+      <author><first>Zenon W.</first><last>Pylyshyn</last></author>
+      <pdf hash="a2f3a044"/>
+      <bibkey>pylyshyn-1978-language-perception</bibkey>
+    </paper>
+    <paper id="25">
+      <!-- https://aclanthology.org/T78-1025/ -->
+      <title>Semantic Primitives in Language and Vision</title>
+      <author id="yorick-wilks"><first>Yorick</first><last>Wilks</last></author>
+      <pdf hash="b4e13896"/>
+      <bibkey>wilks-1978-semantic-primitives</bibkey>
+    </paper>
+    <paper id="26">
+      <!-- https://aclanthology.org/T78-1026/ -->
+      <title>A Note on Partial Match of Descriptions. Can One Simultaneously Question (Retrieve) and Inform (Update)?</title>
+      <author id="aravind-joshi"><first>Aravind K.</first><last>Joshi</last></author>
+      <pdf hash="74969243"/>
+      <bibkey>joshi-1978-note-partial</bibkey>
+    </paper>
+    <paper id="27">
+      <!-- https://aclanthology.org/T78-1027/ -->
+      <title>With a Spoon in Hand This Must Be the Eating Frame</title>
+      <author><first>Eugene</first><last>Charniak</last></author>
+      <pdf hash="f1be68bb"/>
+      <bibkey>charniak-1978-spoon-hand</bibkey>
+    </paper>
+    <paper id="28">
+      <!-- https://aclanthology.org/T78-1028/ -->
+      <title>Fragments of a Theory of Human Plausible Reasoning</title>
+      <author><first>Allan</first><last>Collins</last></author>
+      <pdf hash="67b4bca2"/>
+      <bibkey>collins-1978-fragments-theory</bibkey>
+    </paper>
+    <paper id="29">
+      <!-- https://aclanthology.org/T78-1029/ -->
+      <title>Indirect Responses to Loaded Questions</title>
+      <author><first>S. Jerrold</first><last>Kaplan</last></author>
+      <pdf hash="135037f9"/>
+      <bibkey>kaplan-1978-indirect-responses</bibkey>
+    </paper>
+    <paper id="30">
+      <!-- https://aclanthology.org/T78-1030/ -->
+      <title>On Reasoning by Default</title>
+      <author><first>Raymond</first><last>Reiter</last></author>
+      <pdf hash="0de53d33"/>
+      <bibkey>reiter-1978-reasoning-default</bibkey>
+    </paper>
+    <paper id="31">
+      <!-- https://aclanthology.org/T78-1031/ -->
+      <title>Path-Based and Node-Based Inference in Semantic Networks</title>
+      <author id="stuart-c-shapiro"><first>Stuart C.</first><last>Shapiro</last></author>
+      <pdf hash="2ddc5535"/>
+      <bibkey>shapiro-1978-path-based</bibkey>
+    </paper>
+    <paper id="32">
+      <!-- https://aclanthology.org/T78-1032/ -->
+      <title>The Representation of Derivable Information in Memory: When What Might Have Been Left Unsaid Is Said</title>
+      <author><first>Rand J.</first><last>Spiro</last></author>
+      <author><first>Joseph</first><last>Esposito</last></author>
+      <author><first>Richard J.</first><last>Vondruska</last></author>
+      <pdf hash="7758fb45"/>
+      <bibkey>spiro-etal-1978-representation-derivable</bibkey>
+    </paper>
+    <paper id="33">
+      <!-- https://aclanthology.org/T78-1033/ -->
+      <title>A Heuristic for Paradigms</title>
+      <author><first>Joseph E.</first><last>Grimes</last></author>
+      <pdf hash="53b42c16"/>
+      <bibkey>grimes-1978-heuristic-paradigms</bibkey>
+    </paper>
+    <paper id="34">
+      <!-- https://aclanthology.org/T78-1034/ -->
+      <title>A Computational Account of Some Constraints on Language</title>
+      <author id="mitch-marcus"><first>Mitchell</first><last>Marcus</last></author>
+      <pdf hash="c9724c43"/>
+      <bibkey>marcus-1978-computational-account</bibkey>
+    </paper>
+    <paper id="35">
+      <!-- https://aclanthology.org/T78-1035/ -->
+      <title>Remarks on Processing, Constraints, and the Lexicon</title>
+      <author id="thomas-wasow"><first>Thomas</first><last>Wasow</last></author>
+      <pdf hash="8beed275"/>
+      <bibkey>wasow-1978-remarks-processing</bibkey>
+    </paper>
+    <paper id="36">
+      <!-- https://aclanthology.org/T78-1036/ -->
+      <title>List of questions suggested for consideration in each session</title>
+      <pdf hash="1950b908"/>
+      <bibkey>nn-1978-list-questions</bibkey>
+    </paper>
+  </volume>
+</collection>

tests/test_acl_historical.py

@@ -0,0 +1,114 @@
+"""Pinned T78 evidence, edition-local aliases and an explicitly identified material."""
+import copy
+import hashlib
+import json
+from pathlib import Path
+import sqlite3
+import tempfile
+import unittest
+import xml.etree.ElementTree as ET
+
+from paper_atlas.collectors.acl import collect_xml, volume_scope
+from paper_atlas.collectors.acl_historical import event_scope, material_reason
+from paper_atlas.db.store import Store
+from paper_atlas.ingestion.records import ingest_file
+
+FIXTURE = Path(__file__).parent/'fixtures/acl-t78.xml'
+COMMIT = 'a7be623d01c48c61405dec31c871121cdbed5a01'
+
+
+class AclHistoricalTests(unittest.TestCase):
+    def setUp(self):
+        self.root = ET.parse(FIXTURE).getroot()
+        self.volume = self.root.find('volume')
+        self.meta = self.volume.find('meta')
+
+    def test_frozen_volume_matches_official_snapshot(self):
+        self.assertEqual(hashlib.sha256(FIXTURE.read_bytes()).hexdigest(), 'f712f94cff0cba05f786b7ac31dad2ea9408fde2a17a04531130df623f57dd84')
+        self.assertEqual(len(self.volume.findall('paper')), 36)
+        self.assertEqual(volume_scope(self.meta, '1', 'T78'), ('ACL+TINLAP', ['ACL', 'TINLAP'], 'proceedings_unspecified'))
+
+    def test_unknown_year_collection_or_volume_is_not_globally_aliased(self):
+        for collection, volume in [('T75', '1'), ('T87', '1'), ('T78', '2'), ('C65', '1')]:
+            with self.subTest(collection=collection, volume=volume):
+                self.assertIsNone(event_scope(self.meta, volume, collection))
+                self.assertIsNone(volume_scope(self.meta, volume, collection))
+        self.assertIsNone(volume_scope(self.meta, '1'))
+
+    def test_known_volume_scope_rejects_critical_metadata_drift(self):
+        for field, replacement in [('venue', 'acl'), ('year', '1979'), ('booktitle', 'Another TINLAP'), ('editor/first', 'Another')]:
+            changed = copy.deepcopy(self.meta)
+            changed.find(field).text = replacement
+            with self.subTest(field=field), self.assertRaises(ValueError):
+                event_scope(changed, '1', 'T78')
+        changed = copy.deepcopy(self.meta)
+        ET.SubElement(changed, 'year').text = '1978'
+        with self.assertRaises(ValueError):
+            event_scope(changed, '1', 'T78')
+
+    def test_evidence_retains_original_series_dates_and_role_boundary(self):
+        scope = event_scope(self.meta, '1', 'T78')
+        self.assertEqual(scope['original_venue'], 'TINLAP')
+        self.assertEqual(scope['event_dates'], ['1978-07-25', '1978-07-27'])
+        self.assertEqual([e['pdf_pages'] for e in scope['evidence']], [[9, 10], [3]])
+        self.assertIn('not identified', scope['track_scope_note'])
+        self.assertIn('pending', scope['evidence'][0]['scope'])
+
+    def test_only_exact_session_questions_material_is_excluded(self):
+        paper = copy.deepcopy(self.volume.findall('paper')[-1])
+        self.assertIn('Session question list', material_reason('T78', '1', paper))
+        self.assertIsNone(material_reason('C65', '1', paper))
+        self.assertIsNone(material_reason('T78', '2', paper))
+        paper.set('id', '37')
+        self.assertIsNone(material_reason('T78', '1', paper))
+        paper.set('id', '36')
+        ET.SubElement(paper, 'author')
+        with self.assertRaises(ValueError):
+            material_reason('T78', '1', paper)
+
+    def test_all_authored_source_fields_and_material_receipt_survive_collection(self):
+        with tempfile.TemporaryDirectory() as temporary:
+            manifest, = collect_xml(FIXTURE, temporary, COMMIT)
+            rows = [json.loads(raw) for raw in Path(manifest['file']).read_text().splitlines()]
+        self.assertEqual(manifest['count'], 35)
+        self.assertEqual(manifest['xml_paper_elements'], 36)
+        self.assertEqual(manifest['missing_abstract_count'], 35)
+        self.assertEqual(manifest['excluded_materials'][0]['external_id'], 'T78-1036')
+        for number, row in enumerate(rows, 1):
+            paper = self.volume.findall('paper')[number-1]
+            self.assertEqual(row['external_id'], f'T78-1{number:03d}')
+            self.assertEqual(row['title'], ' '.join(''.join(paper.find('title').itertext()).split()))
+            self.assertEqual(row['metadata']['venue_codes'], ['tinlap'])
+            self.assertEqual(row['metadata']['historical_event_scope']['original_venue'], 'TINLAP')
+            self.assertEqual(row['metadata']['metadata_commit'], COMMIT)
+            self.assertEqual((row['status'], row['track'], row['abstract']), ('accepted', 'proceedings_unspecified', None))
+
+    def test_old_coling_scope_is_unchanged_by_calendar_replacement_evidence(self):
+        meta = ET.fromstring('<meta><venue>coling</venue><year>1965</year><booktitle>COLING 1965</booktitle></meta>')
+        self.assertEqual(volume_scope(meta, '1', 'C65'), ('COLING', ['COLING'], 'main'))
+        self.assertIsNone(event_scope(meta, '1', 'C65'))
+
+    def test_native_ingestion_does_not_alias_other_tinlap_years_or_infer_negative_status(self):
+        with tempfile.TemporaryDirectory() as temporary:
+            root = Path(temporary)
+            manifest, = collect_xml(FIXTURE, root/'source', COMMIT)
+            db = root/'sample.sqlite'
+            store = Store(db)
+            pid, rid = store.record('test_control', 'tinlap1975', 'https://example.org/tinlap1975',
+                                    'An earlier TINLAP contribution', None, ['Jane Doe'])
+            store.decision(pid, rid, 'TINLAP', 1975, 'main', 'tinlap1975', 'https://example.org/tinlap1975', 'accepted', 'test control')
+            store.conn.commit()
+            store.conn.close()
+            ingest_file(db, manifest['file'])
+            ingest_file(db, manifest['file'])
+            with sqlite3.connect(db) as c:
+                self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0], 36)
+                self.assertEqual(c.execute('SELECT count(*) FROM submissions').fetchone()[0], 36)
+                self.assertEqual(c.execute('SELECT count(*) FROM decisions').fetchone()[0], 36)
+                self.assertEqual(c.execute("SELECT count(*) FROM decisions WHERE status!='accepted'").fetchone()[0], 0)
+                self.assertIsNone(c.execute("SELECT 1 FROM source_records WHERE external_id='T78-1036'").fetchone())
+                aliases = c.execute('SELECT v.acronym,a.alias FROM venue_aliases a JOIN venues v ON v.id=a.venue_id ORDER BY a.alias').fetchall()
+                self.assertEqual(aliases, [('ACL+TINLAP', 'ACL'), ('ACL+TINLAP', 'TINLAP')])
+                metadata = json.loads(c.execute("SELECT metadata_json FROM source_records WHERE external_id='T78-1001'").fetchone()[0])
+                self.assertEqual(metadata['historical_event_scope']['event_year'], 1978)
+                self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(), [])