Compare commits

..

1 Commits

Author SHA1 Message Date
codex-bot 52f9740d99 Record ECCV 2026 publisher and programme reconciliation
Metadata integrity checks / metadata (push) Successful in 7m41s
2026-11-18 18:12:49 +08:00
17 changed files with 1336 additions and 1115 deletions
-17
View File
@@ -273,23 +273,6 @@ jobs:
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','iclr-archive','--help'],cwd=outside,check=True)
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','acl','--help'],cwd=outside,check=True)
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','springer-selected','--help'],cwd=outside,check=True)
publisher_fixture = "\n".join((
"import pathlib, runpy, sys",
"fixture = runpy.run_path(sys.argv[1])",
"root = pathlib.Path(sys.argv[2])",
"root.mkdir()",
"fixture['snapshots'](root)",
))
publisher_input = root/'eccv-publisher-snapshots'
subprocess.run([python,'-I','-B','-c',publisher_fixture,
str(src/'tests/test_eccv2026_publications.py'),str(publisher_input)],cwd=outside,check=True)
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','eccv2026-publications',
'--toc-snapshots',str(publisher_input/'toc'),
'--chapter-snapshots',str(publisher_input/'chapters'),
'--out',str(root/'installed-eccv-publisher')],cwd=outside,check=True)
publisher_report = json.loads((root/'installed-eccv-publisher/collection-report.json').read_bytes())
assert publisher_report['records'] == 1 and publisher_report['toc_chapters'] == 2835
assert not publisher_report['database_written']
springer = json.loads(gzip.decompress((src/'tests/fixtures/springer-aistats1993-fixture.json.gz').read_bytes()))
for group in ('toc','chapters'):
folder = root/'springer-snapshots'/group
-2
View File
@@ -148,5 +148,3 @@ ECCV 2026 的[暂定节目采集器](docs/eccv2026-program.md)合并重复节目
`collect mm2026-program` 从已冻结官方快照恢复全部 28 个来源 track 的 2,143 个节目 ID,保留 4,326 次展示的来源位置与场次上下文,单列 216 个摘要缺失项;输出供贡献类型、出版及作品身份核查使用,不是可直接导入的论文/决定队列。见 [MM 2026 使用、来源和恢复说明](docs/mm2026-source-status.md)。
`collect mm2026-publications` 核验完整论文集的 1,918 个注册出版 DOI,并单独关联节目分轨、展示上下文和摘要。1,743 份节目摘要明确标注来源(其中 26 份来自已复核的标题排版对应),166 项标题/作者版本差异保留为待核;所有节目及未匹配项均保留。注册文章类型不等于研究类型,输出尚需数据库身份/历史审查。见 [MM 2026 出版条目采集与恢复](docs/mm2026-publications.md)。
ECCV 2026 已归档 Springer 目录和章节可通过 [正式出版采集流程](docs/eccv2026-publications.md)整理为带 DOI 和可复核证据的 JSONL。
+2
View File
@@ -204,3 +204,5 @@ IJCAI 联办届次的逐条后续核查见 [2018/2022 分区与目录差集](ijc
- [MM 2026 官方节目来源状态](../mm2026-source-status.md):完整官方日程 JSON、重复展示盘点、28 个来源 track、缺失作者/摘要和四条匿名决定路径的 403 证据;这些不是已导入或公开拒稿全集。
- [2026-11-18 公开负面来源核查](public-negative-source-audit-2026-11-18.md):ResearchArcade 930 条非 ICLR 声明的当前库比对、5 条未绑定声明及原始接口访问证据;不把镜像差集算作新增拒稿。
- [ECCV 2026 出版目录与节目单对账](eccv2026-publisher-crosswalk-2026-11-18.md)记录 Springer 78 卷的 DOI、节目单差异及 PostgreSQL 只读绑定核查。
@@ -1,35 +0,0 @@
# 2020 起补漏状态:2026-11-18
总体仍为 **13/17,第 14 步未完成**。主动补漏从 2020 年起,2000–2019 年已发现线索仍保留,43 个早期历史缺口仍为延期,没有据此算作补齐。更早状态页保留为历史快照。
## 当前生产结果
ISMIR 两批累计31项已关联到现有论文,新增论文实体为0。第二批新增来源记录、独立投稿、录用观察各21条及173条来源标签证据;两批累计267条标签证据。覆盖 `partial31/140`,109项仍待核验。详见[来源说明](../ismir2026-program.md)、[首批验收](ismir2026-import10-2026-11-18.json)和[第二批逐条验收](ismir2026-import21-2026-11-18.json)。
最新精确计数:论文实体 **3,568,022**,来源记录 **3,815,534**,投稿 **532,223**,决定来源观察 **571,775**。实体数量不是已经完成最终科学工作去重的数量;同一次投稿可能有多条决定来源观察。
paper `2757497` 保留 ICLR2026 一次拒稿投稿下的两条来源观察,并新增 ISMIR2026 录用投稿;实际 CLI 查询显示已知录用,旧拒稿历史仍可查询。不将两条来源观察计作两次拒稿,也不将来源采集时间当作决定时间。
## 固定范围存在性盘点
只读快照 `2026-11-18 06:46:44.099675+00`,范围仍为 43 会、278 个目标年度槽位:**274 个至少有一条投稿,4 个完全为空**,均为 2026 年的 **CIKM、ECCV、ICDM、IROS**。首批ISMIR使空槽从5降至4;第二批只增加同届来源覆盖,空槽数不变。
这只统计“该年度至少有数据”,不是逐篇、逐 track、摘要或公开拒稿覆盖完整。2020–2025 没有整年空槽,仍有来源完整性和公开负面历史缺口。
|证据(相对服务器 `/data/paper-index/`)|SHA256|
|---|---|
|`collected/active-scope-census-20261118-v22/summary.json`|`ee4ec2e6d6eaed55ce3475e735bfba9bd94e512893cd3c3024035b48b744f475`|
|`collected/active-scope-census-20261118-v22/snapshot.json`|`5aabc91217ad787946ba50d6a670296b1e3152a54f5ecaa0e3a9b3a952b63fa7`|
|`validation/active-scope-fixed278-v1.json`|`a835bef5b9c3a0f76069d840cecdb9a6bcf65d3a38796d70dbfdfd4b49242264`|
本机使用固定分母、会议别名和逐 edition 快照独立重算 274/278;没有使用 `max(id)` 代替计数。
## 尚待完成
- ISMIR 仍有109项;62项无精确标题候选不等于新作品,6项作者部分匹配仍待审查。31项摘要逐字一致候选已分两批审查并消费;其余候选不能仅凭标题、作者或摘要检索结果算作已批准。
- MM 已验收572项来源观察,注册清单1918项中仍有1346项待处理。已有134个暂建索引实体仍须后续身份审计;H2AL 来源及轨道纠正、版本化来源覆盖方案尚未用于生产,不能把不同版本轨道预期数相加当完整分母。
- ECCV 的初步节目、ICDM 的研究/应用来源和缺失摘要仍需作品及出版核查;CIKM 缺实际逐篇目录;IROS 官方访问403仍保留限制,不重试绕过。各来源的采集、批准和导入状态必须分开。
- 公开 rejected、desk-rejected、withdrawn 继续按会议、年份、track核查。ISMIR 的101组自愿公开评审来自录用节目,不是101条拒稿;UAI accepted-only 镜像与 EMNLP/NeurIPS 原始接口403均不能关闭拒稿缺口。近期镜像差集核查见 [PR #72](https://gitea.entroia.com/Zayden-Jung/paper-atlas/pulls/72)。
- 后续仍需完成跨会作品与历史、独立类型和标签关系(第15步);全部导入结束后的全库去重、标识冲突、近似标题作者、摘要/约束/查询及覆盖审计(第16步);长期运行与恢复文档最终整理(第17步)。
已消费输入不得重放。下次从[ISMIR 当前增量父状态和恢复说明](../ismir2026-program.md)及 `work-state.json` 的本批已接受回执继续;保留旧 SQLite 只读备份,不因查询超时重新执行导入。
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,20 @@
# ECCV 2026 出版目录与节目单对账
2026-11-18 核验了 Springer LNCS 17001–17078 的全部 78 卷。会议索引显示 **2,835 条**,逐卷目录合计也有 **2,835 条带 DOI 的章节**,另有 156 项卷首或卷末资料。出版方卷册简介及大会节目单均为 **2,834 条**。两个来源仍有逐条差异,汇总数量相差一条的原因待核。逐卷数字、目录页哈希和只读数据库回执见[机器可读清单](eccv2026-publisher-crosswalk-2026-11-18.json)。[Springer 会议索引](https://link.springer.com/conference/eccv) · [第一卷](https://link.springer.com/book/10.1007/978-3-032-36984-0) · [大会节目单](https://eccv.ecva.net/Conferences/2026/AcceptedPapers)
## 逐篇对账
|核对对象|唯一标题且完整作者顺序一致|唯一标题且目录所显示作者顺序一致|标题对得上、作者有差异|标题待核|
|---|---:|---:|---:|---:|
|全部 2,834 条节目记录|1,332|709|535|258|
|首批已入库的 30 条节目记录|23|3|4|0|
|本次只读核验通过的 248 条既有论文候选|173|72|2|1|
|20 篇有既往拒稿、桌拒或撤回的优先论文|12|7|1|0|
目录作者栏有时只显示前六人。作者差异包括缩写、姓名文字变化与排列变化。原始名单、顺序和来源保留,前缀匹配与作者差异需查章节作者页。节目单与出版目录的摘要、分轨及公开负面决定覆盖仍待分别核查。
## 当前库核对
只读核对了全部 2,835 个出版 DOI,68 个已有论文绑定。对账后另检查 271 个已知论文关联,涵盖 245 个本次候选和 26 个已入库节目记录。其中 14 个 DOI 已指向同一论文,257 个 DOI 尚未绑定。245 个候选内有 10 篇带既往负面决定。
下一批准备以出版 DOI、正式分轨和完整作者证据复核关联。首批 30 条节目记录保留原投稿、决定和来源。确认同届同一投稿后,再补充出版证据,避免重复创建 ECCV 投稿。本次核查未写入数据库。
@@ -1,247 +0,0 @@
{
"as_of": "2026-11-18",
"goal_progress": "13/17; stage14 incomplete",
"source": "ismir_program",
"scope": "First10 reviewed existing-work source links, partial10/140, not public-rejection completeness or final deduplication",
"database_acceptance": {
"status": "passed",
"at": "2026-11-18 05:15:27.565655+00:00",
"writer_commit": "12f95cff91813a2143bc8fab5bc3e89df1bd0fdb",
"before_counts": {
"papers": {
"rows": 3568022,
"max_id": 3568595
},
"source_records": {
"rows": 3815503,
"max_id": 3815504
},
"submissions": {
"rows": 532192,
"max_id": 532193
},
"decisions": {
"rows": 571744,
"max_id": 571744
}
},
"after_counts": {
"papers": {
"rows": 3568022,
"max_id": 3568595
},
"source_records": {
"rows": 3815513,
"max_id": 3815514
},
"submissions": {
"rows": 532202,
"max_id": 532203
},
"decisions": {
"rows": 571754,
"max_id": 571754
}
},
"new_papers": 0,
"new_sources": 1,
"new_editions": 1,
"new_source_records": 10,
"new_attempts": 10,
"new_decisions": 10,
"source_id": 123160,
"source_tag_evidence_links": 94,
"protected_papers": 6394,
"all_protected_old_profiles_sources_tags_identifiers_preserved": true,
"source_observations": 10,
"source_expected": 140,
"source_remaining": 130,
"coverage": "partial",
"foundry_readyz": [
200
],
"sqlite_frozen": true,
"imported": [
{
"external_id": "2026/3",
"paper_id": 2813483,
"record_id": 3815505,
"submission_id": 532194,
"decision_id": 571745,
"edition_id": 104044
},
{
"external_id": "2026/12",
"paper_id": 2785459,
"record_id": 3815506,
"submission_id": 532195,
"decision_id": 571746,
"edition_id": 104044
},
{
"external_id": "2026/35",
"paper_id": 2757859,
"record_id": 3815507,
"submission_id": 532196,
"decision_id": 571747,
"edition_id": 104044
},
{
"external_id": "2026/51",
"paper_id": 2752643,
"record_id": 3815508,
"submission_id": 532197,
"decision_id": 571748,
"edition_id": 104044
},
{
"external_id": "2026/74",
"paper_id": 2762340,
"record_id": 3815509,
"submission_id": 532198,
"decision_id": 571749,
"edition_id": 104044
},
{
"external_id": "2026/75",
"paper_id": 2757480,
"record_id": 3815510,
"submission_id": 532199,
"decision_id": 571750,
"edition_id": 104044
},
{
"external_id": "2026/78",
"paper_id": 2768260,
"record_id": 3815511,
"submission_id": 532200,
"decision_id": 571751,
"edition_id": 104044
},
{
"external_id": "2026/84",
"paper_id": 2740614,
"record_id": 3815512,
"submission_id": 532201,
"decision_id": 571752,
"edition_id": 104044
},
{
"external_id": "2026/87",
"paper_id": 2752271,
"record_id": 3815513,
"submission_id": 532202,
"decision_id": 571753,
"edition_id": 104044
},
{
"external_id": "2026/363",
"paper_id": 2757497,
"record_id": 3815514,
"submission_id": 532203,
"decision_id": 571754,
"edition_id": 104044
}
]
},
"history_protection": {
"submissions": {
"rows": 532192,
"sha256": "34e2fe27cccf89e6e80228318212a846369fdd315c4cfe67d7d535013eac8298"
},
"decisions": {
"rows": 571744,
"sha256": "efbda4e91950c5c7298f536bc5967f7fee57d8b20a7843e00dc69871b1553bd1"
},
"decision_sources": {
"rows": 571744,
"sha256": "1d8899b16b40e9f37ac84dc601efc3e7906af5c06bca49946100697f22c96174"
},
"submission_sources": {
"rows": 581976,
"sha256": "ab3ab200c4abbfef39bc5a2a757550191441c703ae505372d38d9afa9a4daf65"
},
"submission_identifiers": {
"rows": 383956,
"sha256": "0e8f514e186e08bd00a65f8e14fc7f63db6add76722271e0111e14f5d27b9dbe"
}
},
"evidence": {
"accepted_report": "/data/paper-index/collected/ismir2026-import10-v1/postgresql-postcheck-v1.json",
"accepted_report_sha256": "8a08015bbd725b400de53477837e850ca92a50479d7f86f2cfed56d50bb22577",
"baseline": "/data/paper-index/collected/ismir2026-baseline10-v1/baseline.json",
"baseline_sha256": "a0b5b8c2c21ee567cf80152b16f66643309bbe33ee9cd67d19da33d6b6337656",
"baseline_archive_sha256": "eb94f99cb53c9a30f63d34ea11b23c11759e45b571b7efce2dcbe8cb4fe52fb8",
"preparation_sha256": "0d9fad74640ab57fc8bea8aaf72ba07202a40b174d05253db7348b14455a0462",
"acceptance_archive": "/data/paper-index/validation/ismir2026-import10-acceptance-recovery-v1.tar.gz",
"acceptance_archive_sha256": "138e1fb986f5aa6a42794eed982fdc91ae994b8be82f8bf25360db2f7ba6c70c",
"fresh_packets_sha256": "88e423aede0ea83b9f736891c09b12d0cb24f2f8b70508e4863cd07760de5e0d",
"local_acceptance_check_sha256": "42d31337c76b96f93505011db7c797123b67ce2eec7defd0175ea0e9c1df2089",
"census_summary_sha256": "2157035b72d351e2fe91f2bc0c6ebb305f91d95f8772ab9a088076ce4b86e1c2"
},
"native_query_example": {
"paper_id": 2757497,
"publication_outcome": "accepted",
"prior_venue": "ICLR",
"prior_year": 2026,
"prior_submission_id": 131721,
"prior_negative_attempts": 1,
"prior_negative_observations": 2,
"prior_decision_ids": [
133086,
538172
],
"later_observed_venue": "ISMIR",
"later_observed_year": 2026,
"original_ismir_signed_decision_verified": false,
"ismir_decision_date": null,
"observation_time_is_not_decision_time": true
},
"presence_census": {
"snapshot_at": "2026-11-18 05:20:48.641315+00",
"scope": "Fixed278 venue-year presence only, not paper/track/abstract/public-negative completeness or final deduplication. Legacy43 gaps deferred, not filled.",
"target_venues": 43,
"expected_slots": 278,
"populated_slots": 274,
"missing_slots": 4,
"missing": [
{
"venue": "CIKM",
"year": 2026
},
{
"venue": "ECCV",
"year": 2026
},
{
"venue": "ICDM",
"year": 2026
},
{
"venue": "IROS",
"year": 2026
}
],
"delta_from_v20": {
"newly_present": [
{
"venue": "ISMIR",
"year": 2026
}
],
"previous_missing": 5,
"current_missing": 4
},
"snapshot_sha256": "c14bab883c5fcab33b450fcf240dec7225f2f17bc04c3e728b5583657b4ad23b",
"fixed_scope_sha256": "a835bef5b9c3a0f76069d840cecdb9a6bcf65d3a38796d70dbfdfd4b49242264",
"accepted_import_sha256": "8a08015bbd725b400de53477837e850ca92a50479d7f86f2cfed56d50bb22577",
"database_written": false
},
"consumed_queue": "/data/paper-index/collected/ismir2026-prepared10-v1: NEVER REPLAY",
"next": [
"Review130 remaining source observations, including31 total verbatim-abstract candidates of which10 consumed; do not count a title/author screen as identity approval.",
"Fresh incremental parent must be partial10/140; old preparation0/140 is consumed.",
"Continue all venue/year/track public negative availability and formal source coverage.",
"Finish cross-venue identities/types, then final whole-DB dedup/identifier/abstract/constraints/query audit after imports."
]
}
@@ -1,330 +0,0 @@
{
"as_of": "2026-11-18",
"goal_progress": "13/17; stage14 incomplete",
"source": "ismir_program",
"scope": "21 additional reviewed existing-work source links; cumulativepartial31/140, not public-rejection completeness or final deduplication",
"database_acceptance": {
"status": "passed",
"at": "2026-11-18 06:38:55.148539+00:00",
"writer_commit": "3754545003e1f96b88c6533cc3a69dc557f07305",
"before_counts": {
"papers": {
"rows": 3568022,
"max_id": 3568595
},
"source_records": {
"rows": 3815513,
"max_id": 3815514
},
"submissions": {
"rows": 532202,
"max_id": 532203
},
"decisions": {
"rows": 571754,
"max_id": 571754
}
},
"after_counts": {
"papers": {
"rows": 3568022,
"max_id": 3568595
},
"source_records": {
"rows": 3815534,
"max_id": 3815535
},
"submissions": {
"rows": 532223,
"max_id": 532224
},
"decisions": {
"rows": 571775,
"max_id": 571775
}
},
"new_papers": 0,
"new_sources": 0,
"new_editions": 0,
"new_source_records": 21,
"new_attempts": 21,
"new_decisions": 21,
"source_id": 123160,
"source_tag_evidence_links": 173,
"protected_papers": 6415,
"all_protected_old_profiles_sources_tags_identifiers_preserved": true,
"source_observations": 31,
"source_expected": 140,
"source_remaining": 109,
"coverage": "partial",
"foundry_readyz": [
200
],
"sqlite_frozen": true,
"imported": [
{
"external_id": "2026/148",
"paper_id": 2755057,
"record_id": 3815515,
"submission_id": 532204,
"decision_id": 571755,
"edition_id": 104044
},
{
"external_id": "2026/184",
"paper_id": 2752723,
"record_id": 3815516,
"submission_id": 532205,
"decision_id": 571756,
"edition_id": 104044
},
{
"external_id": "2026/196",
"paper_id": 2784571,
"record_id": 3815517,
"submission_id": 532206,
"decision_id": 571757,
"edition_id": 104044
},
{
"external_id": "2026/240",
"paper_id": 2738896,
"record_id": 3815518,
"submission_id": 532207,
"decision_id": 571758,
"edition_id": 104044
},
{
"external_id": "2026/251",
"paper_id": 2753889,
"record_id": 3815519,
"submission_id": 532208,
"decision_id": 571759,
"edition_id": 104044
},
{
"external_id": "2026/258",
"paper_id": 2662030,
"record_id": 3815520,
"submission_id": 532209,
"decision_id": 571760,
"edition_id": 104044
},
{
"external_id": "2026/270",
"paper_id": 2769109,
"record_id": 3815521,
"submission_id": 532210,
"decision_id": 571761,
"edition_id": 104044
},
{
"external_id": "2026/298",
"paper_id": 2752114,
"record_id": 3815522,
"submission_id": 532211,
"decision_id": 571762,
"edition_id": 104044
},
{
"external_id": "2026/317",
"paper_id": 2765378,
"record_id": 3815523,
"submission_id": 532212,
"decision_id": 571763,
"edition_id": 104044
},
{
"external_id": "2026/318",
"paper_id": 2764935,
"record_id": 3815524,
"submission_id": 532213,
"decision_id": 571764,
"edition_id": 104044
},
{
"external_id": "2026/342",
"paper_id": 2738585,
"record_id": 3815525,
"submission_id": 532214,
"decision_id": 571765,
"edition_id": 104044
},
{
"external_id": "2026/370",
"paper_id": 2830871,
"record_id": 3815526,
"submission_id": 532215,
"decision_id": 571766,
"edition_id": 104044
},
{
"external_id": "2026/371",
"paper_id": 2858445,
"record_id": 3815527,
"submission_id": 532216,
"decision_id": 571767,
"edition_id": 104044
},
{
"external_id": "2026/373",
"paper_id": 2820562,
"record_id": 3815528,
"submission_id": 532217,
"decision_id": 571768,
"edition_id": 104044
},
{
"external_id": "2026/409",
"paper_id": 2522863,
"record_id": 3815529,
"submission_id": 532218,
"decision_id": 571769,
"edition_id": 104044
},
{
"external_id": "2026/418",
"paper_id": 2750749,
"record_id": 3815530,
"submission_id": 532219,
"decision_id": 571770,
"edition_id": 104044
},
{
"external_id": "2026/468",
"paper_id": 2846824,
"record_id": 3815531,
"submission_id": 532220,
"decision_id": 571771,
"edition_id": 104044
},
{
"external_id": "2026/518",
"paper_id": 2761686,
"record_id": 3815532,
"submission_id": 532221,
"decision_id": 571772,
"edition_id": 104044
},
{
"external_id": "2026/524",
"paper_id": 2784543,
"record_id": 3815533,
"submission_id": 532222,
"decision_id": 571773,
"edition_id": 104044
},
{
"external_id": "2026/539",
"paper_id": 2761760,
"record_id": 3815534,
"submission_id": 532223,
"decision_id": 571774,
"edition_id": 104044
},
{
"external_id": "2026/552",
"paper_id": 2758575,
"record_id": 3815535,
"submission_id": 532224,
"decision_id": 571775,
"edition_id": 104044
}
]
},
"history_protection": {
"submissions": {
"rows": 532202,
"sha256": "9d41955c80fc91dae4f65d65e0f69198ce2d79508ceebfe943bdc19825983df8"
},
"decisions": {
"rows": 571754,
"sha256": "c8dc762d089d1656ee1308dacec9109605e1a4be7e5fcfa8f95013e77a0cb55c"
},
"decision_sources": {
"rows": 571754,
"sha256": "231f15b4c70435977d6ab784112ff5c748541eeb4d54c28bb861f8835f1eba10"
},
"submission_sources": {
"rows": 581986,
"sha256": "b9d8c39e945df340daf1c695a6359cfb549b8bd2f391cb139dabba2c16afa242"
},
"submission_identifiers": {
"rows": 383956,
"sha256": "0e8f514e186e08bd00a65f8e14fc7f63db6add76722271e0111e14f5d27b9dbe"
}
},
"evidence": {
"parent_acceptance_sha256": "8a08015bbd725b400de53477837e850ca92a50479d7f86f2cfed56d50bb22577",
"accepted_report": "/data/paper-index/collected/ismir2026-import21-v1/postgresql-postcheck-v1.json",
"accepted_report_sha256": "153ef13dbac9f4243c81bdd2a16d3ade74372d3e45ca5e84acc2b7258bca7bfc",
"baseline": "/data/paper-index/collected/ismir2026-baseline21-v1/baseline.json",
"baseline_sha256": "204b571b47e164d68930158c2578f609814969940628d421a2b64c3a6ce1b018",
"baseline_archive_sha256": "ecac32e1d3890786a5670ff06f88fa7bd92155306cf8133f3164de0cb9f48e6b",
"preparation_sha256": "6781edfaa67798d76329fb5c7948d83693607c96352a38eccce0067faa95deaa",
"acceptance_archive": "/data/paper-index/validation/ismir2026-import21-acceptance-recovery-v1.tar.gz",
"acceptance_archive_sha256": "55e1cc8301548ff0abc195add5f40d7082a06e4f087f6da5838ffd2f6b33339c",
"fresh_packets_sha256": "79e0e8187a522478c443d036fdbf2f8d5e6a22c949d46fb0592a664ec1bc22fe",
"local_acceptance_check_sha256": "04f4ec6969a71655d70f446347646f63efc56e5a5a8e974b931703482e5e7ceb",
"census_summary_sha256": "ee4ec2e6d6eaed55ce3475e735bfba9bd94e512893cd3c3024035b48b744f475",
"ci_run": 1257,
"ci_tests": 556,
"ci_proof_sha256": "5ad08aa907caec5a1ca63df5e7345db4802dab4e3a91293a0303bf481a903ff8"
},
"consumed_queue": "/data/paper-index/collected/ismir2026-prepared21-v2: NEVER REPLAY",
"previous_consumed_queue": "/data/paper-index/collected/ismir2026-prepared10-v1: NEVER REPLAY",
"query_example_rechecked": {
"paper_id": 2757497,
"publication_outcome": "accepted",
"rejected_attempts": 1,
"rejection_source_observations": 2,
"distinct_ismir_accepted_attempts": 1,
"original_ismir_signed_decision_verified": false,
"ismir_decision_date": null
},
"new_batch_prior_negative_attempts": 0,
"new_batch_prior_negative_observations": 0,
"source_tag_links_cumulative": 267,
"presence_census": {
"snapshot_at": "2026-11-18 06:46:44.099675+00",
"scope": "Fixed278 venue-year presence only, not paper/track/abstract/public-negative completeness or final deduplication. Legacy43 gaps deferred, not filled.",
"target_venues": 43,
"expected_slots": 278,
"populated_slots": 274,
"missing_slots": 4,
"missing": [
{
"venue": "CIKM",
"year": 2026
},
{
"venue": "ECCV",
"year": 2026
},
{
"venue": "ICDM",
"year": 2026
},
{
"venue": "IROS",
"year": 2026
}
],
"delta_from_v21": {
"newly_present": [],
"previous_missing": 4,
"current_missing": 4,
"ISMIR_source_observations": 31
},
"snapshot_sha256": "5aabc91217ad787946ba50d6a670296b1e3152a54f5ecaa0e3a9b3a952b63fa7",
"fixed_scope_sha256": "a835bef5b9c3a0f76069d840cecdb9a6bcf65d3a38796d70dbfdfd4b49242264",
"accepted_import_sha256": "153ef13dbac9f4243c81bdd2a16d3ade74372d3e45ca5e84acc2b7258bca7bfc",
"database_written": false
},
"next": [
"Remaining109 observations require review;31 exact-source-abstract candidates are now consumed, not reusable queues.",
"Future increments require a fresh partial31/140 parent and new preservation baseline.",
"Keep raw authors/title/abstract/keyword versions; no numeric miniconf-to-OpenReview-ID assumption, global person aliases or inferred research types.",
"Continue all43 venue/year/track formal-source and public-negative availability audits, then final whole-DB dedup after imports."
]
}
-16
View File
@@ -1,16 +0,0 @@
# ECCV 2026 正式出版章节采集
`collect eccv2026-publications` 读取已归档的 Springer LNCS 17001–17078 目录及所选章节的公开 HTML,输出可复核的 Paper Atlas 记录。目录覆盖 2,835 个章节 DOI。会议节目单与出版方卷册简介均为 2,834 条,逐条差异仍待核。Workshop 论文集另行处理。
```sh
paper-atlas collect eccv2026-publications \
--toc-snapshots /data/paper-index/collected/eccv2026-springer-toc-v1 \
--chapter-snapshots /data/paper-index/validation/eccv-verified-chapter-pages \
--out /data/paper-index/collected/eccv-publisher-metadata
```
目录归档含 `receipts.jsonl`、`volume-checkpoint.json`、`summary.json` 及 156 页 HTML。摘要在章节详情页读取,章节归档沿用 `springer-selected` 的 `receipts.json` 格式。两类归档都按响应大小和 SHA-256 校验。目录卷号、分页、章节 DOI 归属、标题及 2026 年出处需要一致。已有输出目录需另查回执或换新空目录后运行。
输出含规范化 JSONL、manifest、采集回执以及所选章节的内容寻址存证。每条保留出版 DOI、卷号、页码、按顺序的完整作者及公开摘要。目录证据用 URL 和哈希连接到输入归档,保存归档以便复核。主会出版观察依据 Springer 正式论文集,原始投稿号、决定日期及签名仍未知。工具仅处理已归档的公开 HTML。
生成的文件待做论文身份复核后方可导入。准备生产批次时核实每个 DOI 与现有 arXiv 标识的绑定,保持两者的来源证据。对已存在的同届节目投稿需先核对投稿身份及轨道,再附加出版来源,避免因直接导入创建重复投稿。具有 ICLR 等既往决定的论文须保留原记录。新的增量导入 manifest 需以当前实际来源覆盖为父版本,留存待核差集和独立验收回执。
-33
View File
@@ -48,36 +48,3 @@ paper-atlas collect ismir2026-program \
正式名单与政策恢复包:同目录 `remaining2026-listings-research-v2-recovery.tar.gz`,SHA256 为 `f5ef7c2e73d46712146e4d6711ad5ad562e16b8b55c0cd75d1041cd0b81142cb`。安全恢复三个源目录、核验文件后可重跑上述命令。
完整来源和独立 140 项清单在测试夹具中冻结。测试、构建、打包、安装及实际隔离目录 CLI 验证只在用户指定 `ces-control:host` runner 执行。生产规范化后还需独立核验全部记录,并用 PostgreSQL 只读预检检查候选作品和既往投稿;不自动合并来源差异或人物身份。
## 2026-11-18:首批 10 条验收(历史阶段)
本段保留首批历史状态;当前累计结果和增量父状态见下方最新验收段。
首批 10 项已逐篇核查研究内容,并通过当前数据库实体、作者、摘要及完整历史的只读复核,随后使用已通过远程 CI 的包内 `import records` 导入。新增论文实体 **0**,新增来源记录、投稿、录用观察各 **10**,来源标签证据 **94** 条;规范标题、作者、摘要及旧历史保留。原始来源摘要另存,不覆盖规范摘要。覆盖为 `partial10/140`,还有 **130** 项待核验。数值节目 ID 仍只作为节目定位符,未冒充 OpenReview 投稿 ID,未新增全局人名别名或推断研究类型。
[逐条验收与恢复记录](coverage/ismir2026-import10-2026-11-18.json)包含生产行 ID、保护基线、归档及证据哈希。服务器基线保护此前 6,384 篇与本批 10 篇的并集 **6,394** 篇,全库旧投稿、决定及来源关系哈希未变;独立新快照的 10 个完整实体包、来源字段、标签和标识也逐项核验。Foundry `/readyz` 返回200,17,987,751,936字节的 SQLite 备份保持只读。该验收不是最终全库科学工作去重。
例如 paper `2757497`(Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping)保留一次 ICLR2026 拒稿投稿和两条拒稿来源观察,并新增独立 ISMIR2026 录用投稿。实际执行以下查询,返回 `publication_outcome=accepted`,同时保留原拒稿历史:
```sh
paper-atlas query --pg-service foundry-research --pg-schema research \
--paper 2757497 --outcome accepted
```
ISMIR 的 accepted 来自正式名单和节目,尚无原始签名决定或具体决定日期;不能把采集时间当作决定时间。决定的规范时间使用包内既有毫秒格式,来源回执的原始微秒时间仍保留于元数据。
**禁止重放** `/data/paper-index/collected/ismir2026-prepared10-v1` 的文件。导入单元 `paper-atlas-ismir2026-import10-v1.service`、invocation `538ce9c9324a4f59b5bb49d3094d5265` 已成功结束;该阶段建立了 `partial10/140` 父状态;后续累计结果见下方最新段,不得沿用已消费的 `0/140` 父状态。保护基线恢复包为 `/data/paper-index/validation/ismir2026-baseline10-recovery-v1.tar.gz`,验收恢复包为 `/data/paper-index/validation/ismir2026-import10-acceptance-recovery-v1.tar.gz`;本机 paper-index 工作目录有同名副本及独立检查记录。
## 2026-11-18:新增 21 条,累计 31/140
第二批21项已通过逐篇研究内容审查、新鲜只读父状态检查、保护基线及独立入库验收。新增论文实体、来源和edition均为0;新增来源记录、投稿、录用观察各21条,新增173条来源标签证据。累计31条录用来源观察、267条标签证据,覆盖仍为 `partial31/140`,**109项待核验**。三十一项摘要都保留在原始来源记录中,既有规范资料没有被覆盖。
本批明确保留不同研究工作,例如UOT-IR音乐压缩与SymphonyGen交响生成,以及AI混合音轨占比回归与BAMM真实广播检测;不因作者或主题相近合并。源关键词和展示方式继续使用各自namespace,没有推断全局人名别名、原始OpenReview投稿ID或研究类型。
[第二批逐条验收及恢复记录](coverage/ismir2026-import21-2026-11-18.json)包含21组生产行ID、源码提交、全部证据哈希和新计数。保护并核验了6,415篇作品的规范资料、旧来源与标签,以及所有旧全局投稿/决定/来源关系指纹;新快照另导出21个完整实体包逐字段核验。首批10项保留;paper2757497的实际查询仍显示accepted及原ICLR一次拒稿下的两条来源观察。
本批使用固定源码 `3754545003e1f96b88c6533cc3a69dc557f07305`,已在远程 `ces-control` CI1257通过556项测试,含真实PostgreSQL只读增量门禁回归。该修复解决预检对Mapping行对象切片的错误,未弱化覆盖校验;旧的只读准备v1失败时没有写库,修复后v2重新检查通过。默认运行时未因本批操作切换。
**禁止重放** `/data/paper-index/collected/ismir2026-prepared21-v2`。服务 `paper-atlas-ismir2026-import21-v1.service`、invocation `78077b901a274426bb161ef8d69963ad` 已成功结束。新批次必须从当前 `partial31/140` 准备,不能再用 `10/140` 或 `0/140`。保护基线恢复包为 `/data/paper-index/validation/ismir2026-baseline21-recovery-v1.tar.gz`,验收恢复包为 `/data/paper-index/validation/ismir2026-import21-acceptance-recovery-v1.tar.gz`;本机paper-index目录有同名副本与独立检查结果。
新的固定范围盘点仍为274/278个年度有数据;本批提高ISMIR来源覆盖,没有填补其他空年度,也没有关闭公开拒稿或最终全库去重任务。Foundry就绪检查200,SQLite备份继续只读。
-1
View File
@@ -27,7 +27,6 @@ COMMANDS = {
('collect','ismir2026-program'): ('paper_atlas.collectors.ismir2026', {}, '核验 ISMIR 2026 公开节目与录用名单并保留原始摘要、来源差异和自愿公开评审'),
('collect','icaps2026-registry'): ('paper_atlas.collectors.icaps_registry2026', {}, '核验 ICAPS 2026 Crossref 注册元数据并保留独立版本'),
('collect','eccv2026-program'): ('paper_atlas.collectors.eccv2026', {}, '核验ECCV2026暂定节目并断点采集公开元数据'),
('collect','eccv2026-publications'): ('paper_atlas.collectors.eccv2026_publications', {}, '核验ECCV2026 Springer 正式出版章节'),
('collect','icassp2026-registry'): ('paper_atlas.collectors.icassp2026', {}, '核验 ICASSP 2026 冻结注册论文元数据'),
('collect','icdm2026-program'): ('paper_atlas.collectors.icdm2026', {}, '核验 ICDM2026 冻结日程及明确的研究/应用轨道'),
('collect','sdm2026-proceedings'): ('paper_atlas.collectors.sdm2026', {}, '核验 SDM 2026 注册论文并单独保留公开日程版本'),
@@ -1,184 +0,0 @@
"""Normalize frozen ECCV 2026 Springer proceedings and preserve publisher evidence."""
import argparse
import fcntl
import hashlib
import json
from pathlib import Path
import re
import tempfile
from paper_atlas.io.html import Document, clean
from paper_atlas.io.springer import parse_chapter, read_page, snapshot_index
SOURCE = 'eccv2026_springer'
ORIGIN = 'https://link.springer.com'
BOOK_TITLE = 'Computer Vision – ECCV 2026'
LIST_URL = ORIGIN + '/conference/eccv'
BOOK = re.compile(r'10\.1007/978-3-032-\d{5}-[\dXx]')
CHAPTER = re.compile(BOOK.pattern + r'_[1-9]\d*')
VOLUMES = set(range(17001, 17079))
PUBLICATIONS = 2835
def fingerprint(raw):
return hashlib.sha256(raw).hexdigest()
def toc_page(html, book_doi, volume):
tree = Document(html).root
text = clean(tree.text())
if BOOK_TITLE not in text or 'Malmö' not in text or f'(LNCS, volume {volume})' not in text:
raise ValueError('Unexpected Springer ECCV volume scope')
entries = {}
for node in tree.walk():
if node.tag != 'li' or node.attrs.get('data-test') != 'chapter':
continue
headings = [n for n in node.walk() if n.tag == 'h3']
if len(headings) != 1:
raise ValueError('Ambiguous Springer contents heading')
heading, = headings
title = clean(heading.text())
links = [n.attrs.get('href', '') for n in heading.walk() if n.tag == 'a']
if title in ('Front Matter', 'Back Matter') and not links:
continue
if len(links) != 1 or not title:
raise ValueError('Authored contents entry lacks one chapter link')
link, = links
doi = link.removeprefix('/chapter/')
if (not link.startswith('/chapter/') or not CHAPTER.fullmatch(doi)
or not doi.startswith(book_doi + '_') or doi in entries):
raise ValueError('Unexpected or duplicate Springer chapter DOI')
entries[doi] = title
if not entries:
raise ValueError('Empty Springer contents page')
return entries
def publisher_inventory(root):
root = Path(root).resolve()
receipt_bytes = (root / 'receipts.jsonl').read_bytes()
inventory_bytes = (root / 'volume-checkpoint.json').read_bytes()
summary = json.loads((root / 'summary.json').read_bytes())
volumes = json.loads(inventory_bytes)
receipt_rows = [json.loads(line) for line in receipt_bytes.splitlines()]
receipts = {row['file']: row for row in receipt_rows}
if (summary['raw_receipts_sha256'] != fingerprint(receipt_bytes)
or summary['volume_checkpoint_sha256'] != fingerprint(inventory_bytes)
or set(volumes) != {str(n) for n in VOLUMES}
or len(receipts) != len(receipt_rows)):
raise ValueError('Incomplete or inconsistent Springer contents snapshot')
entries, used = {}, set()
for number, volume in sorted(volumes.items()):
book_url = volume['book']
book_doi = book_url.removeprefix(ORIGIN + '/book/')
if not BOOK.fullmatch(book_doi) or book_url != ORIGIN + '/book/' + book_doi:
raise ValueError('Unexpected Springer volume URL')
if set(volume['files']) != {'1', '2'}:
raise ValueError('Missing reviewed Springer contents page')
volume_dois = set()
for index, file in volume['files'].items():
receipt = receipts[file]
path = root / file
page_url = book_url + ('?page=2' if index == '2' else '')
if (file in used or Path(file).name != file or not path.resolve().is_relative_to(root)
or receipt['url'] != page_url or receipt['status'] != 200
or not 0 < receipt['bytes'] <= 3_000_000):
raise ValueError('Unsafe, reused or mismatched Springer contents receipt')
raw = path.read_bytes()
if len(raw) != receipt['bytes'] or fingerprint(raw) != receipt['sha256']:
raise ValueError('Springer contents bytes disagree with receipt')
used.add(file)
page_entries = toc_page(raw.decode(), book_doi, int(number))
if entries.keys() & page_entries.keys():
raise ValueError('Springer chapter repeats across contents pages')
volume_dois.update(page_entries)
entries.update({doi: {'title': title, 'lncs_volume': int(number), 'book_doi': book_doi,
'evidence_url': page_url, 'evidence_sha256': receipt['sha256']}
for doi, title in page_entries.items()})
if volume_dois != set(volume['chapter_dois']) or len(volume_dois) != len(volume['chapter_dois']):
raise ValueError('Springer volume chapter set differs from checkpoint')
if used != receipts.keys() or len(entries) != PUBLICATIONS or summary['chapter_link_count'] != PUBLICATIONS:
raise ValueError('Springer contents chapter total differs from the reviewed archive')
return entries
def publisher_row(chapter, toc, evidence):
doi = chapter['doi']
if clean(chapter['title']) != toc['title']:
raise ValueError('Chapter title differs from Springer contents, review the version')
url = ORIGIN + '/chapter/' + doi
return {
'source': SOURCE, 'external_id': doi, 'url': url,
'venue': 'ECCV', 'year': 2026, 'track': 'main',
'title': chapter['title'], 'authors': chapter['authors'], 'abstract': chapter['abstract'],
'status': 'accepted', 'identifiers': {'doi': doi},
'submission_identifiers': [{'scheme': 'publication_doi', 'value': doi}],
'evidence_url': url, 'observed_at': evidence['observed_at'],
'raw_decision': 'Chapter published in the refereed ECCV 2026 conference proceedings.',
'tags': [{'namespace': SOURCE + ':publication_format', 'value': 'proceedings_chapter'}],
'metadata': {
'book_title': BOOK_TITLE, 'book_doi': toc['book_doi'], 'lncs_volume': toc['lncs_volume'],
'authors_citation_raw': chapter['authors_citation_raw'], 'first_page': chapter['first_page'],
'last_page': chapter['last_page'], 'publisher_chapter_evidence': evidence,
'publisher_contents_evidence': {'url': toc['evidence_url'], 'sha256': toc['evidence_sha256']},
'scope': 'Refereed conference proceedings, LNCS 17001–17078.',
'original_decision_signature_verified': False,
},
}
def collect(out, *, toc_snapshots, chapter_snapshots):
contents = publisher_inventory(toc_snapshots)
chapters = snapshot_index(chapter_snapshots)
urls = {ORIGIN + '/chapter/' + doi: doi for doi in contents}
if not chapters or not chapters.keys() <= urls.keys():
raise ValueError('No chapter snapshots or chapter outside reviewed ECCV proceedings')
rows, raw = [], {}
for url in sorted(chapters):
doi = urls[url]
body, evidence = read_page(chapters, url)
chapter = parse_chapter(body.decode(), doi, book_title=BOOK_TITLE, publication_year=2026)
rows.append(publisher_row(chapter, contents[doi], evidence))
raw[evidence['file']] = body
out = Path(out)
out.mkdir(parents=True, exist_ok=True)
lock_path = out / 'eccv2026-publications.collect.lock'
filename = 'eccv2026-springer-main.jsonl'
with lock_path.open('a') as lock:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
if any(p != lock_path for p in out.iterdir()):
raise FileExistsError('Existing output must be inspected, not overwritten')
with tempfile.TemporaryDirectory(prefix='.eccv2026-publications-', dir=out) as temporary:
stage = Path(temporary)
(stage / 'evidence').mkdir()
for path, body in raw.items():
(stage / path).write_bytes(body)
data = ''.join(json.dumps(row, ensure_ascii=False) + '\n' for row in rows).encode()
(stage / filename).write_bytes(data)
manifest = {
'source': SOURCE, 'venue': 'ECCV', 'year': 2026, 'track': 'main', 'parser_version': 1,
'file': str(out / filename), 'file_sha256': fingerprint(data), 'count': len(rows),
'list_url': LIST_URL, 'source_expected_count': PUBLICATIONS, 'status': 'partial',
'missing_abstract_count': sum(not r['abstract'] for r in rows),
}
report = {
'status': 'collected_pending_identity_review', 'records': len(rows), 'files': [manifest],
'toc_chapters': len(contents), 'chapter_details_missing': len(contents) - len(rows),
'database_written': False,
}
metadata = Path(filename).with_suffix('.manifest.json')
(stage / metadata).write_text(json.dumps(manifest, indent=2) + '\n')
(stage / 'collection-report.json').write_text(json.dumps(report, indent=2) + '\n')
for item in ('evidence', filename, metadata, 'collection-report.json'):
(stage / item).replace(out / item)
return report
def main(argv=None, *, prog=None):
parser = argparse.ArgumentParser(prog=prog)
parser.add_argument('--toc-snapshots', required=True)
parser.add_argument('--chapter-snapshots', required=True)
parser.add_argument('--out', required=True)
args = parser.parse_args(argv)
print(json.dumps(collect(args.out, toc_snapshots=args.toc_snapshots,
chapter_snapshots=args.chapter_snapshots), ensure_ascii=False))
+81 -3
View File
@@ -1,14 +1,16 @@
"""Reviewed postconference selected volumes from frozen public HTML metadata."""
import argparse
from collections import defaultdict
import fcntl
import hashlib
import json
from pathlib import Path
import re
import tempfile
from urllib.parse import parse_qs, urlsplit
from paper_atlas.db.submissions import timestamp
from paper_atlas.io.html import Document, clean
from paper_atlas.io.springer import parse_chapter as parse_citation_chapter, read_page, snapshot_index
SOURCE = 'springer_selected_volume'
BOOK = '10.1007/978-1-4612-2660-4'
@@ -24,6 +26,50 @@ SCOPE = {
}
def snapshot_index(root):
root = Path(root).resolve()
receipts = json.loads((root/'receipts.json').read_bytes())['responses']
index = {}
for receipt in receipts:
url = receipt['url']
if url in index:
raise ValueError('Duplicate snapshot URL')
index[url] = (root, receipt)
return index
def read_page(index, url):
if url not in index:
raise ValueError('Missing required public HTML snapshot: '+url)
root, receipt = index[url]
filename = Path(receipt['file'])
path = root/filename
content_type = {k.lower():v for k,v in receipt.get('headers', {}).items()}.get('content-type', '')
requested, final = urlsplit(url), urlsplit(receipt.get('final_url', ''))
query = parse_qs(final.query, keep_blank_values=True)
original_query = parse_qs(requested.query, keep_blank_values=True)
# Springer returns the public metadata with an explicit cookie diagnostic.
# Retain that final URL verbatim; accept no other redirect or changed page.
if query != original_query and requested.netloc == 'link.springer.com':
if (query.pop('error', None) != ['cookies_not_supported']
or not re.fullmatch(r'[0-9a-f-]{36}', ''.join(query.pop('code', [])))):
raise ValueError('Unreviewed Springer redirect query')
if (filename.is_absolute() or '..' in filename.parts or not path.resolve().is_relative_to(root)
or receipt.get('http_status') != 200
or (final.scheme, final.netloc, final.path) != (requested.scheme, requested.netloc, requested.path)
or final.fragment or query != original_query
or 'text/html' not in content_type.lower()):
raise ValueError('Unsafe, redirected or unsuccessful public HTML snapshot')
raw = path.read_bytes()
digest = hashlib.sha256(raw).hexdigest()
if len(raw) > 2_000_000 or len(raw) != receipt['bytes'] or digest != receipt['sha256']:
raise ValueError('Snapshot checksum or size differs')
evidence = {'url':url, 'sha256':digest, 'bytes':len(raw),
'file':'evidence/'+digest+'.html', 'http_status':200,
'observed_at':timestamp(receipt['at']), 'response_receipt':receipt}
return raw, evidence
def parse_toc(html, *, excluded_materials=None):
entries = {}
for node in Document(html).root.walk():
@@ -51,8 +97,40 @@ def parse_toc(html, *, excluded_materials=None):
def parse_chapter(html, doi):
return parse_citation_chapter(html, doi, book_title=SCOPE['book_title'],
publication_year=SCOPE['publication_year'])
nodes = list(Document(html).root.walk())
meta = defaultdict(list)
for node in nodes:
if node.tag == 'meta' and node.attrs.get('name', '').startswith('citation_'):
meta[node.attrs['name']].append(node.attrs.get('content', ''))
def one(key):
values = meta[key]
if len(values) != 1 or not values[0]:
raise ValueError('Missing or ambiguous citation field: '+key)
return values[0]
if (one('citation_doi') != doi or one('citation_inbook_title') != SCOPE['book_title']
or one('citation_publication_date') != str(SCOPE['publication_year'])):
raise ValueError('Chapter DOI, book or publication year differs from reviewed scope')
raw_authors = meta['citation_author']
if not raw_authors:
raise ValueError('Missing named chapter authors')
authors = []
for raw in raw_authors:
if raw.count(',') != 1:
raise ValueError('Unreviewed citation author format')
family, given = [clean(s) for s in raw.split(',', 1)]
if not family or not given:
raise ValueError('Incomplete citation author')
authors.append({'given':given, 'family':family})
abstract_nodes = [n for n in nodes if n.attrs.get('id') == 'Abs1-content']
if len(abstract_nodes) > 1:
raise ValueError('Ambiguous abstract section')
abstract = clean(abstract_nodes[0].text()) if abstract_nodes else None
first, last = one('citation_firstpage'), one('citation_lastpage')
if not first.isdigit() or not last.isdigit() or int(first) > int(last):
raise ValueError('Invalid chapter pages')
return {'doi':doi, 'title':one('citation_title'), 'authors':authors,
'authors_citation_raw':list(raw_authors), 'abstract':abstract or None,
'first_page':first, 'last_page':last}
def record_for(chapter, evidence, scope_evidence):
+1 -1
View File
@@ -66,7 +66,7 @@ def require_incremental_parent(c, source, manifest, options):
if entries:
if len(entries) != 1 or entries[0][0] != manifest['list_url']:
raise ValueError('Incremental coverage requires one unambiguous source listing')
if tuple(entries[0][i] for i in (1, 2, 3)) != ('partial', prior_expected, previous):
if tuple(entries[0][1:]) != ('partial', prior_expected, previous):
raise ValueError('Incremental coverage parent changed or is not partial')
elif previous != 0 or prior_expected is not None:
raise ValueError('Incremental coverage parent is absent')
-91
View File
@@ -1,91 +0,0 @@
"""Frozen Springer public HTML receipts and chapter citation metadata."""
from collections import defaultdict
import hashlib
from pathlib import Path
import json
import re
from urllib.parse import parse_qs, urlsplit
from paper_atlas.db.submissions import timestamp
from paper_atlas.io.html import Document, clean
def snapshot_index(root):
root = Path(root).resolve()
receipts = json.loads((root/'receipts.json').read_bytes())['responses']
index = {}
for receipt in receipts:
url = receipt['url']
if url in index:
raise ValueError('Duplicate snapshot URL')
index[url] = (root, receipt)
return index
def read_page(index, url):
if url not in index:
raise ValueError('Missing required public HTML snapshot: '+url)
root, receipt = index[url]
filename = Path(receipt['file'])
path = root/filename
content_type = {k.lower():v for k,v in receipt.get('headers', {}).items()}.get('content-type', '')
requested, final = urlsplit(url), urlsplit(receipt.get('final_url', ''))
query = parse_qs(final.query, keep_blank_values=True)
original_query = parse_qs(requested.query, keep_blank_values=True)
# Springer returns the public metadata with an explicit cookie diagnostic.
# Retain that final URL verbatim; accept no other redirect or changed page.
if query != original_query and requested.netloc == 'link.springer.com':
if (query.pop('error', None) != ['cookies_not_supported']
or not re.fullmatch(r'[0-9a-f-]{36}', ''.join(query.pop('code', [])))):
raise ValueError('Unreviewed Springer redirect query')
if (filename.is_absolute() or '..' in filename.parts or not path.resolve().is_relative_to(root)
or receipt.get('http_status') != 200
or (final.scheme, final.netloc, final.path) != (requested.scheme, requested.netloc, requested.path)
or final.fragment or query != original_query
or 'text/html' not in content_type.lower()):
raise ValueError('Unsafe, redirected or unsuccessful public HTML snapshot')
raw = path.read_bytes()
digest = hashlib.sha256(raw).hexdigest()
if len(raw) > 2_000_000 or len(raw) != receipt['bytes'] or digest != receipt['sha256']:
raise ValueError('Snapshot checksum or size differs')
evidence = {'url':url, 'sha256':digest, 'bytes':len(raw),
'file':'evidence/'+digest+'.html', 'http_status':200,
'observed_at':timestamp(receipt['at']), 'response_receipt':receipt}
return raw, evidence
def parse_chapter(html, doi, *, book_title, publication_year):
nodes = list(Document(html).root.walk())
meta = defaultdict(list)
for node in nodes:
if node.tag == 'meta' and node.attrs.get('name', '').startswith('citation_'):
meta[node.attrs['name']].append(node.attrs.get('content', ''))
def one(key):
values = meta[key]
if len(values) != 1 or not values[0]:
raise ValueError('Missing or ambiguous citation field: '+key)
return values[0]
if (one('citation_doi') != doi or one('citation_inbook_title') != book_title
or one('citation_publication_date') != str(publication_year)):
raise ValueError('Chapter DOI, book or publication year differs from reviewed scope')
raw_authors = meta['citation_author']
if not raw_authors:
raise ValueError('Missing named chapter authors')
authors = []
for raw in raw_authors:
if raw.count(',') != 1:
raise ValueError('Unreviewed citation author format')
family, given = [clean(s) for s in raw.split(',', 1)]
if not family or not given:
raise ValueError('Incomplete citation author')
authors.append({'given':given, 'family':family})
abstract_nodes = [n for n in nodes if n.attrs.get('id') == 'Abs1-content']
if len(abstract_nodes) > 1:
raise ValueError('Ambiguous abstract section')
abstract = clean(abstract_nodes[0].text()) if abstract_nodes else None
first, last = one('citation_firstpage'), one('citation_lastpage')
if not first.isdigit() or not last.isdigit() or int(first) > int(last):
raise ValueError('Invalid chapter pages')
return {'doi':doi, 'title':one('citation_title'), 'authors':authors,
'authors_citation_raw':list(raw_authors), 'abstract':abstract or None,
'first_page':first, 'last_page':last}
-124
View File
@@ -1,124 +0,0 @@
"""Checks for published ECCV metadata and preserved submission histories."""
from hashlib import sha256
import json
from pathlib import Path
import sqlite3
import tempfile
import unittest
from paper_atlas.collectors.eccv2026_publications import BOOK_TITLE, SOURCE, collect
from paper_atlas.db.store import Store
from paper_atlas.ingestion.records import ingest_file
FIRST_BOOK = '10.1007/978-3-032-10000-0'
FIRST_DOI = FIRST_BOOK + '_1'
def snapshots(root):
toc, chapters = root / 'toc', root / 'chapters'
toc.mkdir()
chapters.mkdir()
receipts, volumes = [], {}
for offset in range(78):
volume = 17001 + offset
book = f'10.1007/978-3-032-{10000 + offset}-0'
book_url = 'https://link.springer.com/book/' + book
count = 37 if offset < 27 else 36
files, chapter_dois = {}, []
for page, indices in ((1, range(1, 20)), (2, range(20, count + 1))):
filename = f'{volume}-{page}.html'
html = f'<h1>{BOOK_TITLE}</h1><p>Malmö (LNCS, volume {volume})</p>'
html += '<li data-test="chapter"><h3>Front Matter</h3></li>'
for i in indices:
doi = f'{book}_{i}'
html += f'<li data-test="chapter"><h3><a href="/chapter/{doi}">Title {volume}-{i}</a></h3></li>'
chapter_dois.append(doi)
raw = html.encode()
(toc / filename).write_bytes(raw)
files[str(page)] = filename
receipts.append({'file': filename, 'status': 200, 'bytes': len(raw), 'sha256': sha256(raw).hexdigest(),
'url': book_url + ('?page=2' if page == 2 else '')})
volumes[str(volume)] = {'book': book_url, 'files': files, 'chapter_dois': chapter_dois}
data = ''.join(json.dumps(row) + '\n' for row in receipts).encode()
(toc / 'receipts.jsonl').write_bytes(data)
checkpoint = json.dumps(volumes).encode()
(toc / 'volume-checkpoint.json').write_bytes(checkpoint)
(toc / 'summary.json').write_text(json.dumps({
'raw_receipts_sha256': sha256(data).hexdigest(),
'volume_checkpoint_sha256': sha256(checkpoint).hexdigest(),
'chapter_link_count': 2835,
}))
url = 'https://link.springer.com/chapter/' + FIRST_DOI
citation = {
'citation_doi': FIRST_DOI, 'citation_title': 'Title 17001-1',
'citation_inbook_title': BOOK_TITLE, 'citation_publication_date': '2026',
'citation_firstpage': '1', 'citation_lastpage': '10',
'citation_author': 'Chen, Lei',
}
body = ''.join(f'<meta name="{k}" content="{v}">' for k, v in citation.items())
body += '<div id="Abs1-content"><p>Published abstract.</p></div>'
raw = body.encode()
(chapters / 'first.html').write_bytes(raw)
(chapters / 'receipts.json').write_text(json.dumps({'responses': [{
'url': url, 'final_url': url, 'file': 'first.html', 'at': '2026-11-18T00:00:00Z',
'http_status': 200, 'bytes': len(raw), 'sha256': sha256(raw).hexdigest(),
'headers': {'Content-Type': 'text/html'},
}]}))
return {'toc_snapshots': toc, 'chapter_snapshots': chapters}
class ECCV2026PublicationsTests(unittest.TestCase):
def test_frozen_snapshot_and_existing_history(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
source = snapshots(root)
report = collect(root / 'out', **source)
self.assertEqual((report['toc_chapters'], report['chapter_details_missing']), (2835, 2834))
manifest, = report['files']
self.assertEqual((manifest['track'], manifest['status']), ('main', 'partial'))
row = json.loads(Path(manifest['file']).read_text())
self.assertEqual(row['identifiers'], {'doi': FIRST_DOI})
self.assertEqual(row['submission_identifiers'], [{'scheme': 'publication_doi', 'value': FIRST_DOI}])
self.assertEqual(row['authors'], [{'family': 'Chen', 'given': 'Lei'}])
self.assertEqual(row['abstract'], 'Published abstract.')
self.assertEqual(row['metadata']['lncs_volume'], 17001)
self.assertFalse(row['metadata']['original_decision_signature_verified'])
detail = row['metadata']['publisher_chapter_evidence']
self.assertEqual(sha256((root / 'out' / detail['file']).read_bytes()).hexdigest(), detail['sha256'])
db = root / 'example.sqlite'
store = Store(db)
pid, rid = store.record('prior', 'work', 'https://example.org/work',
row['title'], 'Original preprint abstract.', row['authors'],
{'arxiv': '2601.00001'})
store.decision(pid, rid, 'ICLR', 2026, 'main', 'work', 'https://example.org/decision',
'rejected', 'Original ICLR result.')
store.conn.commit()
store.conn.close()
row['identifiers']['arxiv'] = '2601.00001'
path = Path(manifest['file'])
raw = (json.dumps(row) + '\n').encode()
path.write_bytes(raw)
manifest['file_sha256'] = sha256(raw).hexdigest()
path.with_suffix('.manifest.json').write_text(json.dumps(manifest))
ingest_file(db, path)
ingest_file(db, path)
with sqlite3.connect(db) as conn:
self.assertEqual(conn.execute('SELECT id,abstract FROM papers').fetchall(),
[(pid, 'Original preprint abstract.')])
self.assertEqual(conn.execute('SELECT status FROM decisions ORDER BY id').fetchall(),
[('rejected',), ('accepted',)])
self.assertEqual(conn.execute('SELECT scheme,value FROM submission_identifiers').fetchall(),
[('publication_doi', FIRST_DOI)])
self.assertEqual(conn.execute('SELECT name FROM sources ORDER BY id').fetchall(), [('prior',), (SOURCE,)])
with self.assertRaises(FileExistsError):
collect(root / 'out', **source)
def test_changed_contents_receipt_stops_before_output(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
source = snapshots(root)
(source['toc_snapshots'] / '17001-1.html').write_text('unverified bytes')
with self.assertRaisesRegex(ValueError, 'contents bytes disagree'):
collect(root / 'out', **source)
self.assertFalse((root / 'out').exists())
+1 -31
View File
@@ -15,7 +15,7 @@ from paper_atlas.db.backend import open_store, open_read_connection
from paper_atlas.db.store import Store
from paper_atlas.ingestion import arxiv
from paper_atlas.ingestion.backfill import backfill
from paper_atlas.ingestion.records import coverage_options, ingest_file, require_incremental_parent
from paper_atlas.ingestion.records import ingest_file
from paper_atlas.postgres.client import connect, ReadConnection
from paper_atlas.postgres.migration import migrate
from paper_atlas.postgres.schema import quote
@@ -114,36 +114,6 @@ class PostgreSQLIngestionTests(unittest.TestCase):
self.assertEqual(run[0],1)
self.assertEqual(json.loads(run[1])['coverage']['imported_count'],3)
def test_incremental_parent_preflight_uses_readonly_postgres_rows_and_keeps_guards(self):
rows = [self.row(source='dblp', external=key, status='accepted', year=2001, venue='UAI')
for key in ('preflight1', 'preflight2')]
path = self.file('preflight-parent', rows)
target = path.with_suffix('.manifest.json')
manifest = json.loads(target.read_text())
manifest['status'] = 'partial'
target.write_text(json.dumps(manifest))
self.ingest(path)
proposed = dict(manifest, coverage_mode='incremental', count=1,
coverage_previous_count=2, coverage_previous_expected_count=2,
coverage_expected_count=4)
with closing(ReadConnection(self.service, self.schema)) as c:
self.assertEqual(c.execute('SHOW transaction_read_only').fetchone()[0], 'on')
source = c.execute("SELECT id FROM sources WHERE name='dblp'").fetchone()[0]
before = {table: c.execute('SELECT count(*) FROM '+table).fetchone()[0]
for table in ('papers', 'source_records', 'submissions', 'decisions', 'ingest_runs')}
coverage = [dict(row) for row in c.execute('SELECT * FROM edition_sources ORDER BY edition_id,source_id,list_url')]
require_incremental_parent(c, source, proposed, coverage_options(proposed))
for changes, error in (({'coverage_previous_count': 1}, 'parent changed'),
({'coverage_previous_expected_count': 3}, 'parent changed'),
({'list_url': 'https://example.org/other-list'}, 'unambiguous source listing')):
with self.subTest(changes=changes):
stale = dict(proposed, **changes)
with self.assertRaisesRegex(ValueError, error):
require_incremental_parent(c, source, stale, coverage_options(stale))
self.assertEqual({table: c.execute('SELECT count(*) FROM '+table).fetchone()[0]
for table in before}, before)
self.assertEqual([dict(row) for row in c.execute('SELECT * FROM edition_sources ORDER BY edition_id,source_id,list_url')], coverage)
def test_incremental_postgres_bound_failure_rolls_back_rows_and_summary(self):
listing='https://example.org/uai/2001'
rows=[self.row(source='dblp',external=key,status='accepted',year=2001,venue='UAI')