Recognize evidenced early ICML and IROS primary editions

Reviewed-on: #35

.gitea/workflows/check.yml

@@ -48,6 +48,7 @@ jobs:
               outside.mkdir()
               subprocess.run([str(venv/'bin/paper-atlas'),'--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','collect','cvf','--help'],cwd=outside,check=True)
+              subprocess.run([python,'-I','-B','-m','paper_atlas','collect','www-archive','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','postgres','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','edition-tracks','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','joint-editions','--help'],cwd=outside,check=True)

README.md

@@ -60,6 +60,8 @@ paper-atlas collect colm --cache /data/paper-index/raw/http-cache \
   --out /data/paper-index/collected
 ```
 
+早期主系列的Workshop名称按[ICML/IROS精确届次证据](docs/coverage/historical-primary-series-2026-11-09.md)识别，保留原名与范围证据；代码支持不代表已补齐生产数据。
+
 `paper-atlas collect --help` 查看来源；`paper-atlas import --help` 查看导入方式。完整批次步骤见[使用指南](docs/usage.md)。
 
 共享实例名为 `foundry-postgres`，数据位于 `foundry_data` 库的 `research` schema；其他项目可按权限复用同一组表。库名通过 libpq service 配置、schema 通过 search_path 或 `--pg-schema` 指定，不写死在查询中。PostgreSQL 凭据放在服务器权限为 `0600` 的 libpq service 文件中；命令行只使用服务名称。`paper-atlas maintain postgres --help` 提供迁移入口：先完整核验 21 张业务表，再发布目标 schema。SQLite FTS5 内部表由 PostgreSQL GIN 索引替代。切换后 SQLite 为只读备份；`import records`、`import arxiv` 和 `maintain backfill` 使用 `--pg-service foundry-research` 写入主库，整个写入会话只允许一个协调写入者，会议文件失败时整体回滚并保留失败批次记录。
@@ -88,6 +90,7 @@ CI 在已授权的远程 runner 上运行测试、构建 wheel，并从源码目
 
 开发通过工作分支和 PR 交付，由用户手动合并。
 
+早期WWW档案的可重放元数据采集、论文/海报分轨与待核查条目见 [WWW档案采集](docs/www-archive.md)。
 历史联合会议的既有届次归属可通过 [`maintain joint-editions`](docs/joint-editions.md) 先只读规划，再按计划哈希执行；保留论文与投稿决定来源，不重导入。
 同年多届的 `event_key`、显式迁移、旧文件刷新与查询边界见 [会议事件身份](docs/event-identity.md)。
 研究类型与自定义标签可通过 [`annotate`](docs/annotations.md) 显式初始化、添加和撤回；两者独立查询，保留证据及完整更正历史，不从会议轨道推断研究类型。

docs/coverage/historical-primary-series-2026-11-09.md

@@ -0,0 +1,33 @@
+# ICML 与 IROS 早期主系列的来源范围
+
+本轮修正采集阶段的届次识别，不修改任何现有生产投稿轨道或历史。正式主系列曾以Workshop命名，不等于附属workshop；也不能反过来把所有旧workshop升为主会。
+
+## 已核实的精确范围
+
+| 会议 | 论文集键 | 依据与行为 |
+| --- | --- | --- |
+| ICML | `conf/icml/1988`、`1989`、`1990`、`1991`、`1992` | [官方历史](https://icml.cc/Conferences/2008/past_icmls.shtml.html)将其列为第5–9届，年份与举办地对应固定目录中的完整标题。1989/1991/1992的Workshop名称不再导致主系列被默认排除；1988/1990保留main并增加来源范围证据 |
+| IROS | `conf/iros/1988`、`1989`、`1990` | [IEEE RAS官方历届目录](https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/)列出三届与论文集链接，地点及日期对应固定目录标题；这些是IROS主系列 |
+
+规则同时要求family、booktitle基础别名（ML或IROS）、空轨道后缀、完整精确论文集键、匹配的事件年份及固定完整标题。仍执行原有书目年份核查：与事件年不同且标题无明确事件年时拒绝推断。ML91原始标题没有四位年份，使用精确1991卷键、完整标题、逐篇书目年和官方第8届记录，不从两位标题代码猜世纪。
+
+`1991w`/`1988w`等卷键、booktitle中的Workshops后缀、不同届次/地点/完整标题、未列入的年份均不会获得上述主系列豁免。AISTATS/COLT既有精确规则保留；真正的附属workshop仍受默认不采集workshop的边界约束。
+
+## 原始字段、决定与后续导入
+
+采集器仍保存原始proceedings标题（包括Workshop）、书目年、原始作者名、完整来源书目字段及稳定标识；新增 `track_scope_note` 与官方 `track_scope_evidence_url`。接受状态仍只源自该论文的inproceedings书目证据；不从历史目录生成占位论文或拒稿，也不重新导入已经存在的记录。
+
+新规则的输出必须重新做原始来源/标题/作者/稳定标识与现有库冲突预检，再按增量导入。此前被隔离的ICML1988/1990记录不能仅凭代码规则而直接解除全部冲突。当前正在运行的历史批次使用原固定版本和文件，不能在运行中替换其源码或输入。
+
+## 验证与证据
+
+新增回归覆盖：ICML三届Workshop名称、IROS最初三届、ICML原Conference名称的两届、错误卷键/后缀/标题/未知年份，以及归一化记录经过原生文件导入后保留轨道证据。构建与测试仅在已授权的ces-control远程runner执行，本机仅做语法和文本/JSON一致性检查；结果以相应commit的CI回执为准。
+
+官方HTML在工作证据目录 `paper-index/historical-primary-series-evidence-v1/`，附响应状态、字节数、时间与SHA256；仅取元数据网页，没有下载论文全文。
+
+| 来源 | UTC采样时间 | HTTP | 字节数 | SHA256 |
+| --- | --- | ---: | ---: | --- |
+| [icml-history](https://icml.cc/Conferences/2008/past_icmls.shtml.html) | 2026-11-08T18:36:43.043135+00:00 | 200 | 11993 | `80cce886fe985551339a52b13c4077acf827e8fcc01555f854dc24077bdebdf0` |
+| [iros-history](https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/) | 2026-11-08T18:36:45.687171+00:00 | 200 | 466059 | `fdb4594621e68567e1099f5c88aae53de835b7b8146025148c3b1968df990697` |
+
+本次不关闭历史年度缺口，也不替代全部导入后的全库去重、来源/年度/track覆盖、摘要和实际查询验收。

docs/coverage/www-1995-posters-reviewed-2026-11-09.json

@@ -0,0 +1,201 @@
+{
+  "format": 1,
+  "source_cache_sha256": "8a65562f3490497cada2919ec77827134beabd5ec9c3690dec3625e2318aa54e",
+  "records": [
+    {
+      "entry_id": "45",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/45/index.html",
+      "individual_sha256": "83bede6afc9a7039fae2dd1f28274a394418b34f5d5632590b913617580bda1a",
+      "title": "WISE - World Wide Information System for R&D; Efforts",
+      "authors": [
+        "Fritz Loseries",
+        "Veronika Samara",
+        "Bodo Urban"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "16",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/16/index.html",
+      "individual_sha256": "cf76da903eaeaa27d19857fdaa13220f6821559bf4ac22844fae10abbaa6f3f3",
+      "title": "The World Wide Web as a Tool for Collaborative Inter-Agency Projects: Creation of an Environmental Metadata Database for the Analysis of the Venice Lagoon Ecosystem.",
+      "authors": [
+        "Mirko Labbri"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "17",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/17/index.html",
+      "individual_sha256": "0578fec9c82e8f2f98c4de0f1bcc44011ebb84c53353fdcc423a69cf70673bc2",
+      "title": "NetRepreneur",
+      "authors": [
+        "Kari-Pekka Turtiainen",
+        "Antti Auer"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "29",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/29/index.html",
+      "individual_sha256": "624ad3f685de1163411e33e90488975d848a63243fd7ffbed0862c6fb58c4140",
+      "title": "Mosaic TueV 2.4.3: Multilingual GUI and Improvements towards User Friendliness",
+      "authors": [
+        "Mike Bretz",
+        "Uwe Koch",
+        "Paul de Bra"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "13",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/13/index.html",
+      "individual_sha256": "02409321a3403a94d8362a9862f4960386ea9c344fb47f4bcdf9eae128e8b46e",
+      "title": "Using a WWW Browser as an Alternative User Interface for Interactive Applications",
+      "authors": [
+        "Frank Lonczewski"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "36",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/36/index.html",
+      "individual_sha256": "ed9dada4ea5903cb8f0a4e75f2958416a6e32647fbf3530e58edd9abf26070ce",
+      "title": "HyperSource: A Hypermedia Program Development And Documentation System",
+      "authors": [
+        "Jan Oliver Borchers"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "57",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/57/index.html",
+      "individual_sha256": "3ad4206ad28a19823172cd2845f636fe7a461c251940e9a93deb8f6484b8a984",
+      "title": "Progressive Image Transmission Capability (PROTRAC) on the World Wide Web",
+      "authors": [
+        "Vojislav Lalich-Petrich",
+        "Gaurav Bhatia",
+        "Larry Davis"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "38",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/38/index.html",
+      "individual_sha256": "940370acc5e6a954b197053fe451e466a904d1f0ac82c8fe4e6ca455faf2c927",
+      "title": "Measuring the Web with Lycos",
+      "authors": [
+        "Michael L. Mauldin"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": "Official abstract listing has no abstract. Canonical HTML has body sections but no explicit abstract; body is not substituted as a summary."
+    },
+    {
+      "entry_id": "05",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/05/index.html",
+      "individual_sha256": "1858be87d6e7f47476e1cb8fb4793c0fce3b8a6ff653bcfd4a35089e0520f177",
+      "title": "Virtual Places - Adding people to the web",
+      "authors": [
+        "Yosi Mass"
+      ],
+      "author_source": "author_index",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": "Official abstract listing and canonical HTML have no abstract. Canonical page identifies a contact and links PostScript; no full-text download. Author identity comes from author index, not inferred from contact role."
+    },
+    {
+      "entry_id": "37",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/37/index.html",
+      "individual_sha256": "b700a0a51b7a82a5f88373ae0283bb369d1cf6f455acff7d91cd7313fa8b3adf",
+      "title": "The Internet Public Library: Serving a Global Community",
+      "authors": [
+        "Susan Schweitzer"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "41",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/41/index.html",
+      "individual_sha256": "397b474109f9c1240a77d0f490f2b02dd35fae93febbb7397d1b2a31759a8ec1",
+      "title": "Graphical Browsing in the WorldWideWeb and the MultiMedia Forum",
+      "authors": [
+        "Rainer Pagé",
+        "Klaus Süllow"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "47",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/47/index.html",
+      "individual_sha256": "52eb4bfdd658730a222c98170b72dabf9cbcc5ffd2ada1e900b826baf2b02833",
+      "title": "Interoperating Advanced Hypermedia Systems with WWW",
+      "authors": [
+        "Jian Zhang",
+        "Martin Fruehauf",
+        "Thomas Kirste"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "50",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/50/index.html",
+      "individual_sha256": "8dc8b0561bd8b1b106c5b209a973c695c3cc9df5288450d4a01fe32541476b7e",
+      "title": "Web Enabled Agile Manufacturing",
+      "authors": [
+        "S. V. Iyer",
+        "T. Singh",
+        "U. Ravaioli",
+        "R. E. DeVor",
+        "S. G. Kapoor",
+        "G. Veknatasubramian"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "55",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/55/index.html",
+      "individual_sha256": "21780cd4047f513e8c9aeced5247002e27cf71719f08de85c6531b57a9e5058d",
+      "title": "Weblets: Fundamental Building Blocks for WWW Tools",
+      "authors": [
+        "Thomas J. Watt Jr."
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    },
+    {
+      "entry_id": "59",
+      "individual_url": "https://archives.iw3c2.org/www3/proceedings/posters/59/index.html",
+      "individual_sha256": "fed4cc9dcaee350a12c5e4b6fa9c99be9bead25a81632512d31566400d3d0fad",
+      "title": "Usenet and the World-Wide Web",
+      "authors": [
+        "M. L. Grant"
+      ],
+      "author_source": "canonical_byline",
+      "reason": "Reviewed canonical H1 and explicit authorship for this exact linked poster entry; retain abstract-heading and author-index title variants without fuzzy matching.",
+      "missing_abstract_reason": ""
+    }
+  ]
+}

docs/www-archive.md

@@ -0,0 +1,111 @@
+# Early WWW archive collector
+
+`paper-atlas collect www-archive` is a maintained metadata adapter. It reads official
+IW3C2 archive HTML, keeps exact source hashes, and separates importable rows from
+program entries that still need review. It fetches no PDF, follows no paper-body or
+reference link, and makes no database connection.
+
+For a fresh source snapshot on the server:
+
+```sh
+paper-atlas collect www-archive \
+  --http-cache /data/paper-index/cache/www-archive \
+  --source-cache /data/paper-index/collected/www-archive-source-v2 \
+  --out /data/paper-index/collected/www-archive-normalized-v2
+```
+
+For reproducible offline replay of the already verified snapshot:
+
+```sh
+paper-atlas collect www-archive \
+  --source-cache /data/paper-index/collected/www-early-archive-source-v1 \
+  --source-cache-sha256 c9388ccef44486e0684c9ee9e8620a465cdc4574e6bc9ea9391d743e01a3312c \
+  --out /data/paper-index/collected/www-archive-normalized-v1
+```
+
+Use a new output directory for another batch. Existing snapshots and normalized
+batches are never overwritten. The manifest hash binds every source URL, byte count,
+response hash and cache filename. Plain files must remain directly within the cache;
+symlinks, escaping paths, duplicate URLs, non-200 responses and altered bytes fail.
+The command uses UTF-8 where valid and explicitly records an ISO-8859-1 fallback for
+older pages; original bytes remain authoritative for encoding corrections.
+
+## Evidence and outputs
+
+- WWW3 Spring 1995 papers use `event_key=www-1995-spring`, `track=main`. The author,
+  session and abstract indexes must have the same nonempty entry-ID set. The ID is
+  local to this exact event and proceedings section, never a DOI or OpenReview ID.
+- Posters remain `track=poster`. An abstract without a linked author-index entry
+  stays in review even when another index has a placeholder `#` link for it.
+- Paper URLs must agree between author and session indexes. A malformed whitespace
+  URL in the abstract list may use the exact event/entry-ID match only when both
+  independent indexes corroborate the same clean URL. The original href is retained.
+- Canonical names come from the explicitly labeled author index. The source lists
+  them alphabetically; this is **not** original byline order. Both the raw index names
+  and the full abstract-page byline/affiliation text remain in metadata. No person ID
+  or cross-source identity is inferred from a name. Conflicting titles or missing
+  abstracts stay in review. Topic keywords are not treated as an abstract.
+- WWW1 1994 formal presentations and WWW2 1994 mixed categories remain review
+  candidates. The first list contains uncertain business-track annotations and a
+  count discrepancy; the second contains panels, tutorials, products and other
+  activities. Neither presentation listing nor a missing full-text link establishes
+  a regular-paper acceptance or a rejection. Raw category/title/name claims are kept.
+
+The main and poster JSONL files have independent manifests with explicit event key,
+track, count and SHA256. `source_listing_count` records the full recognized abstract
+listing, while `held_entry_count` records excluded entries; neither is a ready-row
+count. Their status stays `partial`: a verified linked subset does
+not prove complete event coverage. `review-candidates.jsonl` contains nested
+candidates and reasons, with no importable decision status. It is excluded from
+`import-queue.json`. `report.json` accounts for ready/held records and every source
+used. Newlines delimit JSONL; Unicode U+2028/U+2029 inside metadata are not line breaks.
+The output is published only after all required source and scope checks pass.
+
+## Before importing
+
+This independent collector branch starts from main; it does not include or silently
+install the event-aware writer. Its output requires the reviewed integrated runtime
+from PR31 (or a main revision containing that implementation). The production event
+schema migration has been independently accepted, preserving existing edition1212
+and its 59 Fall submissions; that is not an import of the Spring proceedings.
+
+Inspect held entries and original author names, preflight every ready row against
+current PostgreSQL stable identifiers and candidate identities, and verify the
+source/year/track/event coverage before running `import records`. When a ready subset
+excludes held entries, preserve the independently known listing total in the existing
+incremental-coverage protocol: bind a freshly checked parent count and expected count,
+and set the full expected count from this manifest. A new source uses previous count
+0 and previous expected count null; an existing source requires its actual current
+values. These live parent values are not guessed by the offline collector. Do not automatically
+run this queue against an older writer. Preserve extended versions and independent
+submission/decision evidence, and keep unresolved identities held. A normalization
+report or passing tests alone do not establish production import, complete historical
+coverage or final global deduplication.
+
+Initial fixed-cache replay (64ced7a, CI1057): 38 main papers and38 posters ready;
+372 entries held (50 first-conference fields,307 mixed program entries,15 posters).
+Independent report44dfd070 verified source hashes,counts and author evidence, finding
+one existing exact-title candidate. That publication is explicitly linked to
+arXiv1811.12161 by its WWW1995/journal-reference declaration; arXiv1810.07232 is a
+different1997 invited chapter and remains separate despite the overlap note. This
+is a preparation observation, not evidence of completed production import or final
+global deduplication.
+
+## 以单篇官方页面核实待处理 poster
+
+作者索引漏链接、摘要目录使用短标题或标题存在拼写差异时，不放宽全局标题匹配。可以先取得单篇官方 HTML，人工核对 H1、署名区与原目录同一 entry URL，再提供带 SHA256 的逐条核查文件：
+
+```sh
+paper-atlas collect www-archive \
+  --source-cache /data/paper-index/collected/www-poster-reviewed-source-v2 \
+  --source-cache-sha256 8a65562f3490497cada2919ec77827134beabd5ec9c3690dec3625e2318aa54e \
+  --reviewed-posters docs/coverage/www-1995-posters-reviewed-2026-11-09.json \
+  --reviewed-posters-sha256 <核查文件的SHA256> \
+  --out <新的输出目录>
+```
+
+[1995 年15条poster的核查输入](coverage/www-1995-posters-reviewed-2026-11-09.json) 绑定61页来源快照：原46页和补取的15页单篇HTML。程序验证输入文件及快照哈希、单篇页面哈希、完全一致的 canonical H1、原摘要目录中的同一URL/entry ID及作者证据，不通过相似标题猜身份。规范标题采用单篇H1，摘要目录标题及作者索引标题均在 metadata 中保留。
+
+`author_source=canonical_byline` 表示名字出现在H1之后、摘要/关键词或下一节标题之前的署名区域，由逐条核查明确选出，不从全文引用或机构猜作者；保留原始缩写、顺序、重音和拼写。只有 `author_source=author_index` 时使用原作者索引顺序并逐项核对；05的单篇页面写的是联系人，作者身份使用原作者索引证据，不由联系人角色推断。核查输入是审阅后的数据声明，程序的字符串/哈希检查不代替人工识别署名语义。
+
+38和05的来源没有可明确引用的摘要。保留 `abstract=null` 和逐条缺失原因，manifest准确报告 missing_abstract_count，继续标partial；正文、联系方式及PostScript链接不当作摘要，也不下载正文。缺摘要不等于拒稿或未发表。本批输出仍包含原先已验收记录，正式导入前必须只取新增external ID，检查实时覆盖父记录并按incremental模式准备，不能直接重导入全部输出。

paper_atlas/cli.py

@@ -23,6 +23,7 @@ COMMANDS = {
     ('collect','mlsys'): ('paper_atlas.collectors.proceedings', {'source':'mlsys'}, 'MLSys 官方历年标题与摘要'),
     ('collect','cvf'): ('paper_atlas.collectors.vision', {'source':'cvf'}, 'CVPR / ICCV 官方开放目录'),
     ('collect','ecva'): ('paper_atlas.collectors.vision', {'source':'ecva'}, 'ECCV 官方开放目录'),
+    ('collect','www-archive'): ('paper_atlas.collectors.www_archive', {}, '核验早期WWW官方档案并分开待核查条目'),
     ('collect','dblp'): ('paper_atlas.collectors.dblp_records', {}, '核对并整理 DBLP 历史发表记录'),
     ('catalogue','dblp'): ('paper_atlas.collectors.dblp', {}, '盘点 DBLP 历史会议目录'),
     ('import','arxiv'): ('paper_atlas.ingestion.arxiv', {}, '导入 arXiv 元数据 ZIP'),

paper_atlas/collectors/dblp_scope.py

@@ -54,7 +54,34 @@ HISTORICAL_WORKSHOP_TITLES = {
 }
 
 
+# Exact pinned volume titles corroborated by official series histories. The
+# original title (including Workshop) remains in source metadata. In particular
+# ML91 has no four-digit year in its title; its exact key/title and bibliographic
+# year are checked independently, rather than guessing a title year.
+HISTORICAL_PRIMARY_EVIDENCE = {
+    'icml': ('ML', 'https://icml.cc/Conferences/2008/past_icmls.shtml.html'),
+    'iros': ('IROS', 'https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/'),
+}
+HISTORICAL_PRIMARY_TITLES = {
+    ('icml', 1988): 'Machine Learning, Proceedings of the Fifth International Conference on Machine Learning, Ann Arbor, Michigan, USA, June 12-14, 1988',
+    ('icml', 1989): 'Proceedings of the Sixth International Workshop on Machine Learning (ML 1989), Cornell University, Ithaca, New York, USA, June 26-27, 1989',
+    ('icml', 1990): 'Machine Learning, Proceedings of the Seventh International Conference on Machine Learning, Austin, Texas, USA, June 21-23, 1990',
+    ('icml', 1991): 'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA',
+    ('icml', 1992): 'Proceedings of the Ninth International Workshop on Machine Learning (ML 1992), Aberdeen, Scotland, UK, July 1-3, 1992',
+    ('iros', 1988): "IEEE International Workshop on Intelligent Robots and Systems '88, Proceedings. IROS 1988, Tokyo, Japan, October 31 - November 2, 1988",
+    ('iros', 1989): "IEEE International Workshop on Intelligent Robots and Systems '89, Proceedings. IROS 1989, September 4-6, 1989, Tsukuba, Japan",
+    ('iros', 1990): "IEEE International Workshop on Intelligent Robots and Systems '90, Towards a New Frontier of Applications, Proceedings. IROS 1990, July 3-6, 1990, Ibaraki, Japan",
+}
+
+
 def historical_track(family, base, suffix, volume_key, year, title):
+    primary = HISTORICAL_PRIMARY_EVIDENCE.get(family)
+    if (primary and not suffix and base.casefold() == primary[0].casefold()
+            and volume_key == f'conf/{family}/{year}'
+            and title == HISTORICAL_PRIMARY_TITLES.get((family, year))):
+        return ('main',
+                'Historical primary series; exact proceedings key, complete title and bibliographic/event year corroborated by official series history. Workshop in this original title is not an affiliated track.',
+                primary[1])
     if suffix or base.casefold() != family or volume_key != f'conf/{family}/{year}':
         return None
     prefix = HISTORICAL_WORKSHOP_TITLES.get((family, year))

paper_atlas/collectors/www_archive.py

@@ -0,0 +1,340 @@
+"""Reproducible early WWW metadata; unresolved program entries stay outside imports."""
+import argparse
+from collections import defaultdict
+import hashlib
+from html.parser import HTMLParser
+import json
+from pathlib import Path
+import re
+import tempfile
+from urllib.parse import urljoin, urlsplit
+
+from paper_atlas.io.html import Document, clean
+from paper_atlas.io.http import fetch
+
+ROOT = 'https://archives.iw3c2.org/'
+SPRING = ROOT+'www3/proceedings/'
+FALL = ROOT+'www2/Proceedings/WWW2_Proceedings.html'
+SEEDS = [ROOT+'www1/PrelimProcs.html', ROOT+'www1/Proceedings.html', FALL,
+         SPRING+'proceedings.html', *[SPRING+'papers/'+x+'.html' for x in ('author','session','abstract')],
+         *[SPRING+'posters/'+x+'.html' for x in ('author','abstract')]]
+
+
+def decode(raw):
+    try:
+        return raw.decode('utf-8-sig'), 'utf-8'
+    except UnicodeDecodeError:
+        return raw.decode('iso-8859-1'), 'iso-8859-1 fallback; original bytes retained'
+
+
+def text(html):
+    return re.sub(r'[ \t\r\n\f\v]+',' ',Document(html).root.text()).strip(' \t\r\n\f\v')
+
+
+def anchors(html):
+    return [(n.attrs.get('href',''),clean(n.text())) for n in Document(html).root.walk() if n.tag=='a']
+
+
+def source_url(url):
+    p=urlsplit(url)
+    if p.scheme!='https' or p.netloc!='archives.iw3c2.org' or p.query or p.fragment or not p.path.endswith('.html'):
+        raise ValueError('Only exact official archive HTML source URLs are allowed')
+    return url
+
+
+class Sources:
+    def __init__(self, directory, expected_sha256):
+        self.root=Path(directory).resolve();manifest=self.root/'manifest.json';raw=manifest.read_bytes()
+        if hashlib.sha256(raw).hexdigest()!=expected_sha256:
+            raise ValueError('Source cache manifest checksum mismatch')
+        self.rows={};self.used=set()
+        for row in json.loads(raw)['sources']:
+            url=source_url(row['url'])
+            if url in self.rows or row.get('status')!=200:
+                raise ValueError('Duplicate or incomplete source response')
+            p=self.root/row['file']
+            if p.is_symlink() or p.parent!=self.root or not p.is_file():
+                raise ValueError('Source cache paths must be plain files directly within the cache')
+            data=p.read_bytes()
+            if len(data)>2*1024**2 or len(data)!=row['bytes'] or hashlib.sha256(data).hexdigest()!=row['sha256']:
+                raise ValueError('Source response checksum or size mismatch')
+            self.rows[url]={'raw':data,'receipt':row}
+
+    def read(self,url):
+        source_url(url)
+        if url not in self.rows:
+            raise ValueError('Required source missing from cache: '+url)
+        self.used.add(url)
+        return decode(self.rows[url]['raw'])[0]
+
+    def proof(self,urls):
+        return [{'url':url,'sha256':self.rows[url]['receipt']['sha256'],
+                 'encoding':decode(self.rows[url]['raw'])[1]} for url in sorted(set(urls))]
+
+
+def download_sources(http_cache, directory):
+    directory=Path(directory)
+    if directory.exists():
+        raise ValueError('Refusing to overwrite a source snapshot')
+    directory.mkdir(parents=True)
+    responses={url:fetch(url,http_cache,2*1024**2) for url in SEEDS}
+    html=decode(responses[FALL])[0]
+    for href,label in anchors(html):
+        if href!='Author_Inst.html' and href.endswith('.html'):
+            url=source_url(urljoin(FALL,href))
+            if not url.startswith(ROOT+'www2/Proceedings/'):
+                raise ValueError('Unexpected fall category source')
+            responses[url]=fetch(url,http_cache,2*1024**2)
+    receipts=[]
+    for i,(url,raw) in enumerate(sorted(responses.items())):
+        name='source-'+str(i).zfill(2)+'.html';(directory/name).write_bytes(raw)
+        receipts.append({'url':url,'status':200,'file':name,'bytes':len(raw),'sha256':hashlib.sha256(raw).hexdigest()})
+    manifest=directory/'manifest.json';manifest.write_text(json.dumps({'sources':receipts},indent=2)+'\n')
+    return hashlib.sha256(manifest.read_bytes()).hexdigest()
+
+
+def identity(href,track):
+    url=urljoin(SPRING+track+'/',href);p=urlsplit(url)
+    match=re.fullmatch(r'/www3/proceedings/'+track+r'/(\d+)/[^?#]+',p.path)
+    if p.scheme=='https' and p.netloc=='archives.iw3c2.org' and not p.query and not p.fragment and match:
+        return match[1],url
+    return None
+
+
+class AuthorIndex(HTMLParser):
+    def __init__(self,html,track):
+        super().__init__(convert_charrefs=True);self.track=track;self.name='';self.bold=None;self.link=None;self.link_text=[]
+        self.entries=defaultdict(lambda:{'authors':[],'urls':set(),'titles':set()});self.feed(html)
+
+    def handle_starttag(self,tag,attrs):
+        if tag=='b':self.bold=[]
+        if tag=='a':self.link=dict(attrs).get('href');self.link_text=[]
+
+    def handle_data(self,data):
+        if self.bold is not None:self.bold.append(data)
+        if self.link is not None:self.link_text.append(data)
+
+    def handle_endtag(self,tag):
+        if tag=='b' and self.bold is not None:
+            self.name=clean(''.join(self.bold));self.bold=None
+        if tag=='a' and self.link is not None:
+            own=identity(self.link,self.track)
+            if own:
+                key,url=own;entry=self.entries[key];entry['urls'].add(url);entry['titles'].add(clean(''.join(self.link_text)))
+                if not self.name:raise ValueError('Paper link has no archive author context')
+                if self.name not in entry['authors']:entry['authors'].append(self.name)
+            self.link=None
+
+
+def sections(html):
+    """Use explicit h3 boundaries; old HTML omits many p/li closing tags."""
+    result=[]
+    for chunk in re.split(r'<h3\b[^>]*>',html,flags=re.I)[1:]:
+        parts=re.split(r'</h3\s*>',chunk,maxsplit=1,flags=re.I)
+        if len(parts)!=2:raise ValueError('Unclosed archive entry heading')
+        heading,body=parts
+        body=re.split(r'<(?:hr|h2|address)\b',body,maxsplit=1,flags=re.I)[0]
+        author=re.search(r'<i\b[^>]*>(.*?)</i\s*>',body,re.I|re.S)
+        paragraphs=[]
+        for paragraph in re.findall(r'<p\b[^>]*>(.*?)(?=</p\s*>|<p\b|$)',body,re.I|re.S):
+            value=text(paragraph)
+            if value and not value.casefold().startswith('keywords:'):paragraphs.append(value)
+        result.append({'heading':heading,'title':text(heading),'abstract':'\n\n'.join(paragraphs)or None,
+                       'author_byline_raw':text(author[1])if author else None})
+    if not result:raise ValueError('No explicit archive entry headings')
+    return result
+
+
+def reviewed_posters(sources, path, expected_sha256, cache_sha256):
+    if path is None:
+        if expected_sha256:raise ValueError('Reviewed poster path required')
+        return {},None
+    raw=Path(path).read_bytes()
+    if hashlib.sha256(raw).hexdigest()!=expected_sha256:raise ValueError('Reviewed poster checksum mismatch')
+    review=json.loads(raw)
+    if set(review)!={'format','source_cache_sha256','records'} or review['format']!=1 or review['source_cache_sha256']!=cache_sha256:
+        raise ValueError('Reviewed poster snapshot binding differs')
+    if not isinstance(review['records'],list) or not 0<len(review['records'])<=100:
+        raise ValueError('Bounded nonempty poster review required')
+    result={}
+    for row in review['records']:
+        if set(row)!={'entry_id','individual_url','individual_sha256','title','authors','author_source','reason','missing_abstract_reason'}:
+            raise ValueError('Unexpected reviewed poster fields')
+        key=row['entry_id'];own=identity(row['individual_url'],'posters')
+        if not isinstance(key,str) or not own or own[0]!=key or key in result:
+            raise ValueError('Reviewed poster identity duplicated or outside canonical entry')
+        if row['author_source'] not in ('canonical_byline','author_index') or not isinstance(row['authors'],list) or not row['authors'] or any(not isinstance(a,str) or not a.strip()for a in row['authors']):
+            raise ValueError('Explicit author evidence required')
+        if len(set(row['authors']))!=len(row['authors']) or not isinstance(row['reason'],str) or not row['reason'].strip():
+            raise ValueError('Reviewed author order/reason invalid')
+        if sources.rows.get(own[1],{}).get('receipt',{}).get('sha256')!=row['individual_sha256']:
+            raise ValueError('Individual poster page checksum differs')
+        page=sources.read(own[1]);headings=re.findall(r'<h1\b[^>]*>(.*?)</h1\s*>',page,re.I|re.S)
+        if len(headings)!=1 or text(headings[0])!=row['title']:
+            raise ValueError('Reviewed title differs from exact canonical H1')
+        header=re.split(r'</h1\s*>',page,maxsplit=1,flags=re.I)[1]
+        header=re.split(r'<h[23]\b',header,maxsplit=1,flags=re.I)[0]
+        header=re.split(r'\b(?:keywords|abstract)\s*:',text(header),maxsplit=1,flags=re.I)[0]
+        if row['author_source']=='canonical_byline' and any(a.casefold() not in header.casefold()for a in row['authors']):
+            raise ValueError('Reviewed author absent from canonical byline region')
+        result[key]={**row,'canonical_byline_region_raw':header}
+    return result,hashlib.sha256(raw).hexdigest()
+
+
+def reviewed_row(sources,key,url,item,authors,review):
+    if url!=review['individual_url']:
+        raise ValueError('Reviewed canonical page differs from abstract-entry URL')
+    if authors and authors['urls']!={url}:
+        raise ValueError('Reviewed poster author-index URL disagrees')
+    if review['author_source']=='author_index':
+        names=[]
+        for name in authors['authors']if authors else []:
+            parts=name.split(',');names.append(clean(parts[1]+' '+parts[0])if len(parts)==2 else name)
+        if names!=review['authors']:raise ValueError('Reviewed author-index evidence differs')
+    missing=review['missing_abstract_reason']
+    if not isinstance(missing,str) or bool(missing.strip())==bool(item['abstract']):
+        raise ValueError('Missing abstract requires explicit reason; never invent a summary')
+    urls=[SPRING+'posters/author.html',SPRING+'posters/abstract.html',url]
+    return {'source':'www_archive','external_id':'www3:posters:'+key,'url':url,'title':review['title'],
+            'abstract':item['abstract'],'authors':review['authors'],'identifiers':{},'venue':'WWW','year':1995,
+            'track':'poster','event_key':'www-1995-spring','status':'accepted','raw_decision':'listed in official posters proceedings',
+            'metadata':{'entry_id':key,'source_evidence':sources.proof(urls),'abstract_heading_title_raw':item['title'],
+                        'author_byline_raw':item['author_byline_raw'],'author_index_raw':authors['authors']if authors else [],
+                        'author_index_titles_raw':sorted(authors['titles'])if authors else [],'abstract_href_raw':url,
+                        'author_order':review['author_source'],'reviewed_poster_identity':review,
+                        'missing_abstract_reason':missing or None},
+            'tags':[{'namespace':'www_archive:proceedings_section','value':'posters'}]}
+
+
+def spring_rows(sources,track,reviews=None):
+    base=SPRING+track+'/';author_url=base+'author.html';abstract_url=base+'abstract.html'
+    authors=AuthorIndex(sources.read(author_url),track).entries
+    listing=sections(sources.read(abstract_url));by_id={};held=[];session={}
+    if track=='papers':
+        session_url=base+'session.html'
+        for href,title in anchors(sources.read(session_url)):
+            own=identity(href,track)
+            if own:
+                key,url=own
+                if key in session:raise ValueError('Duplicate main session entry ID')
+                session[key]=url
+    for item in listing:
+        links=[identity(h,track)for h,t in anchors(item['heading'])if identity(h,track)]
+        if len(links)!=1:
+            held.append({'reason':'unverified_heading_identity','evidence':abstract_url,'candidate':item});continue
+        key,url=links[0]
+        if key in by_id:raise ValueError('Duplicate abstract entry ID')
+        by_id[key]=(url,item)
+    if not by_id:raise ValueError('No stable abstract entry IDs')
+    if track=='papers' and (set(authors)!=set(by_id) or set(session)!=set(by_id)):
+        raise ValueError('Main author/session/abstract entry ID sets disagree')
+    if set(authors)-set(by_id):raise ValueError('Author index has entries outside the abstract listing')
+    reviews=reviews or {}
+    if set(reviews)-set(by_id):raise ValueError('Reviewed poster outside abstract listing')
+    rows=[]
+    for key,(url,item) in by_id.items():
+        if key in reviews:
+            if track!='posters':raise ValueError('Reviews only apply to poster entries')
+            rows.append(reviewed_row(sources,key,url,item,authors.get(key),reviews[key]));continue
+        if key not in authors:
+            held.append({'reason':'abstract_entry_missing_linked_author_index','entry_id':key,'evidence':abstract_url,'candidate':item});continue
+        alternatives=authors[key]['urls']
+        if len(alternatives)!=1:raise ValueError('Ambiguous author-index paper URL')
+        canonical=next(iter(alternatives))
+        if track=='papers' and session[key]!=canonical:raise ValueError('Main author/session paper URLs disagree')
+        if url!=canonical and (track!='papers' or not re.search(r'\s',url)):
+            raise ValueError('Uncorroborated archive URL discrepancy')
+        if not item['abstract']:
+            held.append({'reason':'missing_abstract','entry_id':key,'evidence':abstract_url,'candidate':item});continue
+        if item['title'].casefold() not in {t.casefold() for t in authors[key]['titles']}:
+            held.append({'reason':'abstract_and_author_index_titles_disagree','entry_id':key,'evidence':abstract_url,'candidate':item,'author_index_titles':sorted(authors[key]['titles'])});continue
+        names=[]
+        for name in authors[key]['authors']:
+            parts=name.split(',')
+            names.append(clean(parts[1]+' '+parts[0])if len(parts)==2 else name)
+        proof_urls=[author_url,abstract_url]+([session_url]if track=='papers'else[])
+        rows.append({'source':'www_archive','external_id':'www3:'+track+':'+key,'url':canonical,
+                     'title':item['title'],'abstract':item['abstract'],'authors':names,'identifiers':{},
+                     'venue':'WWW','year':1995,'track':'main'if track=='papers'else'poster','event_key':'www-1995-spring',
+                     'status':'accepted','raw_decision':'listed in official '+track+' proceedings',
+                     'metadata':{'entry_id':key,'source_evidence':sources.proof(proof_urls),
+                                 'author_index_raw':authors[key]['authors'],'author_index_titles_raw':sorted(authors[key]['titles']),'author_order':'archive author-index order, not original byline rank',
+                                 'author_byline_raw':item['author_byline_raw'],'abstract_href_raw':url,
+                                 'url_corroborated_by_author_and_session':url!=canonical},
+                     'tags':[{'namespace':'www_archive:proceedings_section','value':track}]})
+    return rows,held,{'abstract_entries':len(by_id),'linked_author_entries':len(authors),'ready':len(rows),'held':len(held)}
+
+
+def early_review(sources):
+    held=[];url=ROOT+'www1/PrelimProcs.html';html=sources.read(url)
+    for ordinal,chunk in enumerate(re.split(r'<li\b[^>]*>',html,flags=re.I)[1:],1):
+        title=re.search(r'<em>Title:</em>\s*<b>(.*?)</b>',chunk,re.I|re.S)
+        if not title:raise ValueError('Unclassified first-conference entry')
+        names=re.findall(r'<em>Author:</em>(.*?)(?=<br\b)',chunk,re.I|re.S)
+        held.append({'reason':'formal_presentation_scope_and_count_need_review','event_key':'www-1994-spring',
+                     'ordinal':ordinal,'evidence':url,'candidate':{'title':text(title[1]),
+                     'authors_raw':[re.sub(r'\s*<[^>]+>','',text(n)).strip()for n in names],
+                     'keywords_raw':text(re.search(r'<em>Keywords:</em>(.*?)(?=<br\b)',chunk,re.I|re.S)[1]) if re.search(r'<em>Keywords:</em>(.*?)(?=<br\b)',chunk,re.I|re.S)else None},
+                     'source_evidence':sources.proof([url])})
+    if not held:raise ValueError('First-conference presentation listing empty')
+    category_urls={}
+    for href,label in anchors(sources.read(FALL)):
+        if href!='Author_Inst.html' and href.endswith('.html'):
+            category_url=source_url(urljoin(FALL,href))
+            if not category_url.startswith(ROOT+'www2/Proceedings/'):raise ValueError('Unexpected fall category')
+            category_urls[category_url]=label
+    if not category_urls:raise ValueError('Fall archive category listing empty')
+    for url,label in sorted(category_urls.items()):
+        html=sources.read(url)
+        for ordinal,chunk in enumerate(re.split(r'<dt\b[^>]*>',html,flags=re.I)[1:],1):
+            fields=re.split(r'<dd\b[^>]*>',chunk,maxsplit=1,flags=re.I)
+            title=text(fields[0]);raw_byline=text(re.split(r'</dl\b',fields[1],maxsplit=1,flags=re.I)[0]) if len(fields)==2 else None
+            if title:
+                held.append({'reason':'mixed_program_category_not_verified_paper_acceptance','event_key':'www-1994-fall',
+                             'category':label,'ordinal':ordinal,'evidence':url,'candidate':{'title':title,'authors_raw':raw_byline},
+                             'source_evidence':sources.proof([FALL,url])})
+    return held,{'first_conference_fields':sum(r['event_key']=='www-1994-spring'for r in held),'fall_categories':len(category_urls),
+                 'fall_program_entries':sum(r['event_key']=='www-1994-fall'for r in held)}
+
+
+def collect(source_cache,cache_sha256,out,*,reviewed=None,reviewed_sha256=None):
+    out=Path(out)
+    if out.exists():raise ValueError('Refusing to overwrite an existing normalized batch')
+    sources=Sources(source_cache,cache_sha256);groups={};held=[];counts={}
+    reviews,review_sha=reviewed_posters(sources,reviewed,reviewed_sha256,cache_sha256)
+    for track in ('papers','posters'):
+        rows,review,counts[track]=spring_rows(sources,track,reviews if track=='posters'else None);groups[track]=rows;held.extend(review)
+    review,counts['earlier']=early_review(sources);held.extend(review)
+    for item in held:
+        if 'source_evidence' not in item:item['source_evidence']=sources.proof([item['evidence']])
+    out.parent.mkdir(parents=True,exist_ok=True)
+    with tempfile.TemporaryDirectory(prefix='.www-archive-',dir=out.parent)as tmp:
+        stage=Path(tmp);queue=[]
+        for track,rows in groups.items():
+            if not rows:continue
+            filename='www-1995-spring-'+('main'if track=='papers'else'poster')+'.jsonl';raw=''.join(json.dumps(r,ensure_ascii=False)+'\n'for r in rows).encode();(stage/filename).write_bytes(raw)
+            manifest={'source':'www_archive','parser_version':2 if review_sha else 1,'venue':'WWW','year':1995,'track':rows[0]['track'],'event_key':'www-1995-spring',
+                      'list_url':SPRING+track+'/abstract.html','count':len(rows),'status':'partial','file_sha256':hashlib.sha256(raw).hexdigest(),
+                      'missing_abstract_count':sum(not bool((r['abstract']or '').strip())for r in rows),'source_listing_count':counts[track]['abstract_entries'],'held_entry_count':counts[track]['held'],'source_cache_sha256':cache_sha256,'scope':'Verified linked archive entries; held entries and other event tracks excluded, not complete event coverage.'}
+            (stage/filename).with_suffix('.manifest.json').write_text(json.dumps(manifest,indent=2)+'\n');queue.append(str(out/filename))
+        (stage/'review-candidates.jsonl').write_text(''.join(json.dumps(r,ensure_ascii=False)+'\n'for r in held))
+        report={'status':'normalized_not_imported','source_cache_sha256':cache_sha256,'counts':counts,'ready':sum(len(r)for r in groups.values()),'held':len(held),
+                'source_evidence':sources.proof(sources.used),'files':queue,'scope':'Early WWW archive only; no PDF fetching, no rejected/withdrawn inference, no database access. Event-aware importer required.'}
+        if review_sha:report['reviewed_posters_sha256']=review_sha
+        (stage/'report.json').write_text(json.dumps(report,ensure_ascii=False,indent=2)+'\n')
+        (stage/'import-queue.json').write_text(json.dumps({'files':queue},indent=2)+'\n')
+        stage.rename(out)
+    return report
+
+
+def main(argv=None,*,prog=None):
+    p=argparse.ArgumentParser(prog=prog);p.add_argument('--source-cache',required=True);p.add_argument('--source-cache-sha256');p.add_argument('--http-cache');p.add_argument('--out',required=True)
+    p.add_argument('--reviewed-posters');p.add_argument('--reviewed-posters-sha256')
+    args=p.parse_args(argv)
+    if bool(args.http_cache)==bool(args.source_cache_sha256):p.error('Choose a fresh --http-cache download or an existing --source-cache-sha256 snapshot')
+    sha=download_sources(args.http_cache,args.source_cache)if args.http_cache else args.source_cache_sha256
+    if bool(args.reviewed_posters)!=bool(args.reviewed_posters_sha256):p.error('Reviewed poster file and SHA256 must be supplied together')
+    print(json.dumps(collect(args.source_cache,sha,args.out,reviewed=args.reviewed_posters,reviewed_sha256=args.reviewed_posters_sha256),ensure_ascii=False),flush=True)
+
+
+if __name__=='__main__':main()

tests/test_historical_primary_series.py

@@ -0,0 +1,115 @@
+"""Regression coverage for primary series once named Workshop, with strict scope boundaries."""
+from contextlib import closing
+import hashlib
+import json
+from pathlib import Path
+import sqlite3
+import tempfile
+import unittest
+
+from paper_atlas.collectors.dblp_records import normalize_row
+from paper_atlas.collectors.dblp_scope import scope
+from paper_atlas.ingestion.records import ingest_file
+
+ICML_HISTORY = 'https://icml.cc/Conferences/2008/past_icmls.shtml.html'
+IROS_HISTORY = 'https://www.ieee-ras.org/conferences-workshops/financially-co-sponsored/iros/iros-past-and-future-venues/'
+
+
+class HistoricalPrimarySeriesTests(unittest.TestCase):
+    def example(self, family, year, title, book=None, key=None):
+        key = key or f'conf/{family}/{year}'
+        book = book or ('ML' if family == 'icml' else 'IROS')
+        row = {'record_type':'inproceedings', 'key':f'conf/{family}/Paper{year}', 'publtype':None,
+               'fields':{'title':['An independently identified historical research paper'],
+                         'author':['Jane Doe'], 'booktitle':[book], 'year':[str(year)],
+                         'crossref':[key], 'ee':[f'https://doi.org/10.1234/{family}.{year}']}}
+        volume = {'record_type':'proceedings', 'key':key,
+                  'fields':{'title':[title], 'year':[str(year)]}}
+        return row, volume
+
+    def test_icml_workshop_primary_editions_are_not_silently_filtered(self):
+        examples = {
+            1989:'Proceedings of the Sixth International Workshop on Machine Learning (ML 1989), Cornell University, Ithaca, New York, USA, June 26-27, 1989',
+            1991:'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA',
+            1992:'Proceedings of the Ninth International Workshop on Machine Learning (ML 1992), Aberdeen, Scotland, UK, July 1-3, 1992',
+        }
+        for year, title in examples.items():
+            with self.subTest(year=year):
+                row, volume = self.example('icml', year, title)
+                record, reason = normalize_row(row, {volume['key']:volume}, {'ICML'})
+                self.assertIsNone(reason)
+                self.assertEqual((record['venue'],record['year'],record['track'],record['status']), ('ICML',year,'main','accepted'))
+                self.assertEqual(record['metadata']['proceedings_title'], title)
+                self.assertEqual(record['metadata']['track_scope_evidence_url'], ICML_HISTORY)
+                self.assertEqual(record['metadata']['bibliographic_year'], year)
+
+    def test_iros_first_three_editions_are_primary_series(self):
+        examples = {
+            1988:"IEEE International Workshop on Intelligent Robots and Systems '88, Proceedings. IROS 1988, Tokyo, Japan, October 31 - November 2, 1988",
+            1989:"IEEE International Workshop on Intelligent Robots and Systems '89, Proceedings. IROS 1989, September 4-6, 1989, Tsukuba, Japan",
+            1990:"IEEE International Workshop on Intelligent Robots and Systems '90, Towards a New Frontier of Applications, Proceedings. IROS 1990, July 3-6, 1990, Ibaraki, Japan",
+        }
+        for year, title in examples.items():
+            with self.subTest(year=year):
+                row, volume = self.example('iros', year, title)
+                record, reason = normalize_row(row, {volume['key']:volume}, {'IROS'})
+                self.assertIsNone(reason)
+                self.assertEqual((record['venue'],record['year'],record['track']), ('IROS',year,'main'))
+                self.assertEqual(record['metadata']['proceedings_title'], title)
+                self.assertEqual(record['metadata']['track_scope_evidence_url'], IROS_HISTORY)
+
+    def test_main_named_icml_predecessors_also_carry_primary_scope_provenance(self):
+        examples = {
+            1988:'Machine Learning, Proceedings of the Fifth International Conference on Machine Learning, Ann Arbor, Michigan, USA, June 12-14, 1988',
+            1990:'Machine Learning, Proceedings of the Seventh International Conference on Machine Learning, Austin, Texas, USA, June 21-23, 1990',
+        }
+        for year, title in examples.items():
+            row, volume = self.example('icml', year, title)
+            edition, reason = scope(row, volume)
+            self.assertIsNone(reason)
+            self.assertEqual(edition['track'], 'main')
+            self.assertEqual(edition['track_scope_evidence_url'], ICML_HISTORY)
+
+    def test_wrong_keys_titles_suffixes_and_unknown_years_keep_workshop_boundary(self):
+        title = 'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA'
+        for year, changed, book, key in [
+            (1991,title,'ML','conf/icml/1991w'),
+            (1991,title,'ML (Workshops)','conf/icml/1991'),
+            (1991,title.replace('Eighth','Ninth'),'ML','conf/icml/1991'),
+            (1991,title + ', Affiliated Workshop','ML','conf/icml/1991'),
+            (1987,'Machine Learning Workshop 1987, Irvine','ML','conf/icml/1987'),
+        ]:
+            with self.subTest(year=year, book=book, key=key, title=changed):
+                row, volume = self.example('icml',year,changed,book,key)
+                record, reason = normalize_row(row,{volume['key']:volume},{'ICML'})
+                self.assertIsNone(record)
+                self.assertEqual(reason,'separate_workshop')
+                self.assertNotIn('track_scope_evidence_url',scope(row,volume)[0])
+        title = "IEEE International Workshop on Intelligent Robots and Systems '88, Proceedings. IROS 1988, Tokyo, Japan, October 31 - November 2, 1988"
+        for changed, book, key in [
+            (title,'IROS','conf/iros/1988w'),
+            (title,'IROS (Workshops)','conf/iros/1988'),
+            (title.replace('Tokyo','Another city'),'IROS','conf/iros/1988'),
+        ]:
+            row, volume = self.example('iros',1988,changed,book,key)
+            self.assertEqual(normalize_row(row,{volume['key']:volume},{'IROS'})[1],'separate_workshop')
+
+    def test_primary_scope_proof_survives_native_file_ingestion(self):
+        title = 'Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA'
+        row, volume = self.example('icml',1991,title)
+        record, reason = normalize_row(row,{volume['key']:volume},{'ICML'})
+        self.assertIsNone(reason)
+        with tempfile.TemporaryDirectory() as temporary:
+            root=Path(temporary);path=root/'records.jsonl';path.write_text(json.dumps(record)+'\n')
+            path.with_suffix('.manifest.json').write_text(json.dumps({'source':'dblp','count':1,'list_url':ICML_HISTORY,'file_sha256':hashlib.sha256(path.read_bytes()).hexdigest()}))
+            ingest_file(root/'test.sqlite',path)
+            with closing(sqlite3.connect(root/'test.sqlite')) as connection:
+                self.assertEqual(connection.execute('SELECT track FROM editions').fetchone()[0],'main')
+                self.assertEqual(connection.execute('SELECT status FROM decisions').fetchone()[0],'accepted')
+                metadata=json.loads(connection.execute('SELECT metadata_json FROM source_records').fetchone()[0])
+                self.assertEqual(metadata['proceedings_title'],title)
+                self.assertEqual(metadata['track_scope_evidence_url'],ICML_HISTORY)
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_www_archive.py

@@ -0,0 +1,155 @@
+import hashlib
+import json
+from pathlib import Path
+import tempfile
+import unittest
+
+from paper_atlas.collectors.www_archive import (ROOT, SPRING, FALL, Sources, collect, decode,
+                                                early_review, spring_rows)
+
+
+class ArchiveTests(unittest.TestCase):
+    def setUp(self):
+        t=tempfile.TemporaryDirectory();self.addCleanup(t.cleanup);self.root=Path(t.name);self.cache=self.root/'cache';self.cache.mkdir()
+        self.pages={ROOT+'www1/PrelimProcs.html':'''<h1>Formal presentations</h1><ol><li><em>Keywords:</em> BUSINESS TRACK?<br><em>Title:</em><b> First contribution</b><br><em>Author:</em>Jane Doe &lt;jane@example.org&gt;<br><li><em>Title:</em><b>Unlinked contribution</b><br><em>Author:</em>John Roe<br></ol>''',
+                    FALL:'<h1>Conference Tracks</h1><ol><li><a href="Agents.html">Agents</a><li><a href="Tutor.html">Tutorials</a></ol>',
+                    ROOT+'www2/Proceedings/Agents.html':'<h3>Panel</h3><dl><dt><a href="a.html">Linked talk</a><dd>Jane Doe<dt>Unlinked talk</a><dd>John Roe</dl>',
+                    ROOT+'www2/Proceedings/Tutor.html':'<h1>Tutorials</h1><dl><dt>Training session<dd>Teacher</dl>'}
+        for track in ['papers','posters']:
+            self.pages[SPRING+track+'/author.html']='<b>Doe, Jane</b><ul><li><a href="01/paper.html">Same contribution</a></li></ul><b>Roe, John</b><a href="01/paper.html">Same contribution</a>'
+            self.pages[SPRING+track+'/abstract.html']='''<h1>Abstracts</h1><a href="#one">Same contribution</a><h3><a href="01/paper.html">Same contribution</a></h3><i>Author: Jane Doe, John Roe<br>Institute</i><p>Actual abstract with <i>inline emphasis</i> and a separator\u2028 preserved.</p><hr><h2>Another session</h2>'''
+        self.pages[SPRING+'papers/session.html']='<a href="01/paper.html">Same contribution</a>'
+
+    def sources(self):
+        rows=[]
+        for i,(url,html) in enumerate(sorted(self.pages.items())):
+            raw=html.encode()if isinstance(html,str)else html;name='source-'+str(i)+'.html';(self.cache/name).write_bytes(raw)
+            rows.append(dict(url=url,status=200,file=name,bytes=len(raw),sha256=hashlib.sha256(raw).hexdigest()))
+        p=self.cache/'manifest.json';p.write_text(json.dumps({'sources':rows}));sha=hashlib.sha256(p.read_bytes()).hexdigest()
+        return Sources(self.cache,sha),sha
+
+    def test_offline_batch_separates_tracks_review_and_importable_rows(self):
+        _,sha=self.sources();out=self.root/'batch';report=collect(self.cache,sha,out)
+        self.assertEqual(report['ready'],2);self.assertEqual(report['held'],5)
+        queue=json.loads((out/'import-queue.json').read_text())['files'];self.assertEqual(len(queue),2)
+        for name in queue:
+            p=Path(name);rows=[json.loads(l)for l in p.read_text().split('\n') if l];self.assertEqual(len(rows),1)
+            row=rows[0];self.assertEqual(row['event_key'],'www-1995-spring');self.assertEqual(row['authors'],['Jane Doe','John Roe'])
+            self.assertIn(row['track'],['main','poster']);self.assertIn('inline emphasis',row['abstract']);self.assertNotIn('Another session',row['abstract'])
+            self.assertIn('\u2028',row['abstract']);manifest=json.loads(p.with_suffix('.manifest.json').read_text());self.assertEqual(manifest['file_sha256'],hashlib.sha256(p.read_bytes()).hexdigest());self.assertEqual(manifest['status'],'partial')
+        held=[json.loads(l)for l in(out/'review-candidates.jsonl').read_text().split('\n') if l]
+        self.assertTrue(all('status'not in r and 'source_evidence'in r for r in held))
+        self.assertFalse(any('review-candidates'in p for p in queue))
+
+    def test_partial_manifest_preserves_full_listing_size_separately_from_ready_count(self):
+        self.pages[SPRING+'posters/abstract.html']+='<h3><a href="02/other.html">Unlinked poster</a></h3><i>Author: Other Person</i><p>Actual abstract</p>'
+        _,sha=self.sources();out=self.root/'batch';collect(self.cache,sha,out)
+        manifest=json.loads((out/'www-1995-spring-poster.manifest.json').read_text())
+        self.assertEqual((manifest['count'],manifest['source_listing_count'],manifest['held_entry_count']),(1,2,1))
+        self.assertNotIn('coverage_previous_count',manifest)  # Live parent must be independently preflighted.
+
+    def test_malformed_href_requires_exact_id_and_two_clean_index_anchors(self):
+        self.pages[SPRING+'papers/abstract.html']=self.pages[SPRING+'papers/abstract.html'].replace('01/paper.html','01/        paper.html')
+        sources,_=self.sources();rows,held,_=spring_rows(sources,'papers');self.assertEqual(held,[])
+        self.assertEqual(rows[0]['url'],SPRING+'papers/01/paper.html');self.assertTrue(rows[0]['metadata']['url_corroborated_by_author_and_session'])
+        self.assertIn('        ',rows[0]['metadata']['abstract_href_raw'])
+
+    def test_main_index_disagreement_fails_without_publishing_batch(self):
+        self.pages[SPRING+'papers/session.html']='<a href="02/other.html">Other work</a>';_,sha=self.sources();out=self.root/'batch'
+        with self.assertRaisesRegex(ValueError,'ID sets disagree'):collect(self.cache,sha,out)
+        self.assertFalse(out.exists())
+
+    def test_same_entry_id_with_conflicting_clean_url_is_rejected(self):
+        self.pages[SPRING+'papers/session.html']='<a href="01/other.html">Same contribution</a>';sources,_=self.sources()
+        with self.assertRaisesRegex(ValueError,'URLs disagree'):spring_rows(sources,'papers')
+
+    def test_poster_without_linked_author_entry_is_held_even_with_abstract(self):
+        self.pages[SPRING+'posters/author.html']='<b>Doe, Jane</b><a href="#">Same contribution</a>';sources,_=self.sources();rows,held,counts=spring_rows(sources,'posters')
+        self.assertEqual(rows,[]);self.assertEqual(counts['abstract_entries'],1);self.assertEqual(held[0]['reason'],'abstract_entry_missing_linked_author_index')
+
+    def test_unlinked_and_panel_entries_never_become_acceptance_or_rejection(self):
+        sources,_=self.sources();held,counts=early_review(sources);self.assertEqual(counts,{'first_conference_fields':2,'fall_categories':2,'fall_program_entries':3})
+        self.assertTrue(any(r['candidate']['title']=='Unlinked talk'for r in held));self.assertTrue(all('status'not in r for r in held))
+        self.assertEqual(held[0]['candidate']['authors_raw'],['Jane Doe'])
+
+    def test_title_disagreement_goes_to_review_with_source_claims(self):
+        self.pages[SPRING+'papers/abstract.html']=self.pages[SPRING+'papers/abstract.html'].replace('Same contribution','Different contribution');sources,_=self.sources();rows,held,_=spring_rows(sources,'papers')
+        self.assertEqual(rows,[]);self.assertEqual(held[0]['reason'],'abstract_and_author_index_titles_disagree')
+
+    def test_cache_tampering_and_manifest_hash_mismatch_fail_closed(self):
+        _,sha=self.sources()
+        with self.assertRaisesRegex(ValueError,'manifest checksum'):Sources(self.cache,'0'*64)
+        p=next(self.cache.glob('source-*.html'));p.write_bytes(p.read_bytes()+b'x')
+        with self.assertRaisesRegex(ValueError,'checksum or size'):Sources(self.cache,sha)
+
+    def test_cache_cannot_read_paths_outside_snapshot(self):
+        _,sha=self.sources();p=self.cache/'manifest.json';d=json.loads(p.read_text());d['sources'][0]['file']='../outside.html';p.write_text(json.dumps(d))
+        with self.assertRaisesRegex(ValueError,'plain files'):Sources(self.cache,hashlib.sha256(p.read_bytes()).hexdigest())
+
+    def test_keywords_only_is_missing_abstract_not_invented_summary(self):
+        self.pages[SPRING+'papers/abstract.html']='<h3><a href="01/paper.html">Same contribution</a></h3><i>Author: Jane Doe</i><p>Keywords: networking, hypertext</p>';sources,_=self.sources();rows,held,_=spring_rows(sources,'papers')
+        self.assertEqual(rows,[]);self.assertEqual(held[0]['reason'],'missing_abstract')
+
+    def test_old_encoding_and_duplicate_abstract_identity(self):
+        self.assertEqual(decode(b'M\xfcller')[0],'Müller')
+        self.pages[SPRING+'papers/abstract.html']*=2;sources,_=self.sources()
+        with self.assertRaisesRegex(ValueError,'Duplicate abstract'):spring_rows(sources,'papers')
+
+    def test_existing_output_is_never_overwritten(self):
+        _,sha=self.sources();out=self.root/'batch';out.mkdir();(out/'keep').write_text('original')
+        with self.assertRaisesRegex(ValueError,'overwrite'):collect(self.cache,sha,out)
+        self.assertEqual((out/'keep').read_text(),'original')
+
+    def review(self,*,authors=None,author_source='canonical_byline',missing='',page=None):
+        url=SPRING+'posters/01/paper.html'
+        self.pages[url]=page or '<h1>Canonical full title</h1><p>Jane Doe, John Roe</p><p>Keywords: web</p><h2>Abstract</h2><p>Body</p>'
+        _,sha=self.sources();raw=self.pages[url].encode()if isinstance(self.pages[url],str)else self.pages[url]
+        row=dict(entry_id='01',individual_url=url,individual_sha256=hashlib.sha256(raw).hexdigest(),
+                 title='Canonical full title',authors=authors or ['Jane Doe','John Roe'],author_source=author_source,
+                 reason='Reviewed exact canonical entry and its explicit authorship; retain original title variants',missing_abstract_reason=missing)
+        p=self.root/'review.json';p.write_text(json.dumps({'format':1,'source_cache_sha256':sha,'records':[row]}));return sha,p,hashlib.sha256(p.read_bytes()).hexdigest()
+
+    def test_reviewed_variant_keeps_primary_title_and_all_old_title_claims(self):
+        sha,p,h=self.review();out=self.root/'reviewed';r=collect(self.cache,sha,out,reviewed=p,reviewed_sha256=h)
+        row=json.loads((out/'www-1995-spring-poster.jsonl').read_text());self.assertEqual(row['title'],'Canonical full title')
+        self.assertEqual(row['metadata']['abstract_heading_title_raw'],'Same contribution');self.assertEqual(row['metadata']['author_index_titles_raw'],['Same contribution'])
+        self.assertEqual(len(row['metadata']['source_evidence']),3);self.assertEqual(r['reviewed_posters_sha256'],h)
+        self.assertEqual(row['external_id'],'www3:posters:01');self.assertEqual(row['status'],'accepted')
+
+    def test_canonical_byline_can_resolve_unlinked_author_index_without_guessing(self):
+        self.pages[SPRING+'posters/author.html']='<b>Doe, Jane</b><a href="#">Broken index link</a>'
+        sha,p,h=self.review();out=self.root/'reviewed';collect(self.cache,sha,out,reviewed=p,reviewed_sha256=h)
+        row=json.loads((out/'www-1995-spring-poster.jsonl').read_text());self.assertEqual(row['authors'],['Jane Doe','John Roe'])
+        self.assertEqual(row['metadata']['author_index_raw'],[]);self.assertEqual(row['metadata']['author_order'],'canonical_byline')
+
+    def test_reviewed_missing_abstract_is_explicit_partial_not_body_summary(self):
+        self.pages[SPRING+'posters/abstract.html']='<h3><a href="01/paper.html">Same contribution</a></h3><i>Author: Jane Doe</i><p>Keywords: web</p>'
+        sha,p,h=self.review(missing='Neither listing nor canonical page supplies an explicit abstract')
+        out=self.root/'reviewed';collect(self.cache,sha,out,reviewed=p,reviewed_sha256=h)
+        row=json.loads((out/'www-1995-spring-poster.jsonl').read_text());m=json.loads((out/'www-1995-spring-poster.manifest.json').read_text())
+        self.assertIsNone(row['abstract']);self.assertEqual((m['missing_abstract_count'],m['status']),(1,'partial'))
+        sha,p,h=self.review()
+        with self.assertRaisesRegex(ValueError,'Missing abstract'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
+        self.assertFalse((self.root/'bad').exists())
+
+    def test_reviewed_names_must_be_in_byline_region_not_citations_or_contact_guess(self):
+        sha,p,h=self.review(authors=['Unrelated Person'],page='<h1>Canonical full title</h1><p>Jane Doe</p><h2>Abstract</h2><p>Unrelated Person cited here</p>')
+        with self.assertRaisesRegex(ValueError,'byline region'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
+        sha,p,h=self.review(author_source='author_index',authors=['Contact Person'])
+        with self.assertRaisesRegex(ValueError,'author-index'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
+
+    def test_review_hash_cache_binding_and_canonical_h1_are_all_required(self):
+        sha,p,h=self.review()
+        with self.assertRaisesRegex(ValueError,'checksum'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256='0'*64)
+        d=json.loads(p.read_text());d['source_cache_sha256']='0'*64;p.write_text(json.dumps(d));h=hashlib.sha256(p.read_bytes()).hexdigest()
+        with self.assertRaisesRegex(ValueError,'binding'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
+        d['source_cache_sha256']=sha;d['records'][0]['title']='Wrong title';p.write_text(json.dumps(d));h=hashlib.sha256(p.read_bytes()).hexdigest()
+        with self.assertRaisesRegex(ValueError,'canonical H1'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
+
+    def test_unlisted_review_entry_and_changed_primary_digest_fail_closed(self):
+        sha,p,h=self.review();d=json.loads(p.read_text());d['records'][0]['individual_sha256']='0'*64;p.write_text(json.dumps(d));h=hashlib.sha256(p.read_bytes()).hexdigest()
+        with self.assertRaisesRegex(ValueError,'page checksum'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)
+        self.pages[SPRING+'posters/abstract.html']=self.pages[SPRING+'posters/abstract.html'].replace('01/paper.html','02/other.html')
+        self.pages[SPRING+'posters/author.html']=self.pages[SPRING+'posters/author.html'].replace('01/paper.html','02/other.html')
+        sha,p,h=self.review()
+        with self.assertRaisesRegex(ValueError,'outside abstract'):collect(self.cache,sha,self.root/'bad',reviewed=p,reviewed_sha256=h)