Compare commits

..

2 Commits

Author SHA1 Message Date
codex-bot a980b1a1df Record verified ISMIR increment to31 existing-work source links
Metadata integrity checks / metadata (push) Successful in 6m45s
2026-11-18 14:51:27 +08:00
codex-bot e38b2f1ca8 Record verified ISMIR source links and preserved rejection history
Metadata integrity checks / metadata (push) Successful in 7m36s
2026-11-18 13:25:06 +08:00
9 changed files with 645 additions and 209 deletions
-20
View File
@@ -47,26 +47,6 @@ jobs:
subprocess.run([python,'-B','-m','pip','install','--no-deps',str(wheel)],check=True)
outside = root / 'outside-checkout'
outside.mkdir()
cikm_snapshots = pathlib.Path('/data/paper-index/validation/cikm2026-source-snapshots-v1')
if cikm_snapshots.is_dir():
cikm_output = root / 'installed-cikm2026'
command = [python, '-I', '-B', '-m', 'paper_atlas', 'collect', 'cikm2026-sources']
subprocess.run([*command, '--snapshots', str(cikm_snapshots), '--out', str(cikm_output)], cwd=outside, check=True)
report = json.loads((cikm_output/'collection-report.json').read_text())
assert {name: entry['records'] for name, entry in report['files'].items()} == {
'programme.jsonl': 1008, 'alternate-programme.jsonl': 1005,
'supplemental-programme.jsonl': 73, 'crossref.jsonl': 1081}
assert len(report['registry_outside_doi_programme']) == 73 and not report['database_written']
programme = [json.loads(line) for line in (cikm_output/'programme.jsonl').read_text().splitlines()]
assert all(row['abstract'] and row['author_display_names'] for row in programme)
assert len({row['doi'] for row in programme}) == 1008
broken = root/'tampered-cikm2026'
broken.mkdir()
(broken/'program.html').write_text('tampered')
rejected = subprocess.run([*command, '--snapshots', str(broken), '--out', str(root/'rejected-cikm2026')], cwd=outside, capture_output=True, text=True)
assert rejected.returncode != 0 and 'Snapshot hash mismatch' in rejected.stderr
assert not (root/'rejected-cikm2026').exists()
print('CIKM 2026 full frozen-source CLI check passed', flush=True)
mm_fixture = json.loads(gzip.decompress((src/'tests/fixtures/mm2026-source.json.gz').read_bytes()))
mm_snapshots = root/'mm2026-snapshots'
for name, body in mm_fixture['files'].items():
-2
View File
@@ -137,8 +137,6 @@ ECCV 2026 的[暂定节目采集器](docs/eccv2026-program.md)合并重复节目
`collect sdm2026-proceedings` 核验 53 个 Crossref 论文章节,并单独保留 53 项日程观察及 52 页摘要。作者差异、取消展示和摘要缺失分别保留,不自动拼接来源或绑定作品;见 [SDM 2026 使用与恢复说明](docs/sdm2026-proceedings.md)。
`collect cikm2026-sources` 核验并分开导出两份官方节目、附属议程和出版登记。见 [CIKM 2026 来源采集与恢复](docs/cikm2026-sources.md)。
`collect icdm2026-program` 离线核验 324 条 ICDM2026 官方线下日程,按明确的研究/应用标签输出 241/83 条;展示长短、缺失摘要及未核实出版身份分别保留,不自动入库。见 [ICDM2026 使用与恢复说明](docs/icdm2026-program.md)。
`collect recsys2026-program` 核验九个 track 的 263 项官方录用贡献观察(260 个不同 DOI),保留摘要、原始作者和重复 DOI 的来源版本。见 [RecSys 2026 使用与恢复说明](docs/recsys2026-program.md)。
-22
View File
@@ -1,22 +0,0 @@
# CIKM 2026 来源采集与恢复
使用固定节目和登记快照复现 CIKM 2026 来源盘点。采集命令生成四份独立 JSONL,输入逐文件校验 SHA-256。
```sh
paper-atlas collect cikm2026-sources \
--snapshots /data/paper-index/validation/cikm2026-source-snapshots-v1 \
--out /data/paper-index/collected/cikm2026-sources-v1
```
输入包括两版官方节目 HTML、Crossref 卷登记、五页逐篇登记及分页回执。服务器验证目录保留输入原文。输出目录必须是新目录。先检查 `collection-report.json`,报告在四份 JSONL 写完后发布。
| 文件 | 内容 |
| --- | --- |
| `programme.jsonl` | 带 DOI 的 1,008 条节目记录,包括标题、作者显示名、摘要和五类节目标签 |
| `alternate-programme.jsonl` | 另一版节目 1,005 条记录及各自的字段 |
| `supplemental-programme.jsonl` | 73 条演讲、教程、工作坊及其他附属活动观察 |
| `crossref.jsonl` | 1,081 条 ACM 卷内登记,每条保留登记字段及分页文件位置 |
节目 DOI 指向 Crossref 的出版登记。确认作者和标题差异以及历史投稿绑定后,再为来源观察准备入库队列。报告另列 73 个未出现在带 DOI 节目区的登记,附属活动仍保留各自的活动类型。公开拒稿与撤回决定仍需独立来源核查。
官方原始来源见[带 DOI 的节目](https://cikm2026.diag.uniroma1.it/cikm-program/program.html)、[另一版节目](https://cikm2026.diag.uniroma1.it/cikm-program/CIKM2026_program.html)和 [Crossref 卷登记](https://api.crossref.org/works/10.1145/3799682)。
@@ -0,0 +1,35 @@
# 2020 起补漏状态:2026-11-18
总体仍为 **13/17,第 14 步未完成**。主动补漏从 2020 年起,2000–2019 年已发现线索仍保留,43 个早期历史缺口仍为延期,没有据此算作补齐。更早状态页保留为历史快照。
## 当前生产结果
ISMIR 两批累计31项已关联到现有论文,新增论文实体为0。第二批新增来源记录、独立投稿、录用观察各21条及173条来源标签证据;两批累计267条标签证据。覆盖 `partial31/140`,109项仍待核验。详见[来源说明](../ismir2026-program.md)、[首批验收](ismir2026-import10-2026-11-18.json)和[第二批逐条验收](ismir2026-import21-2026-11-18.json)。
最新精确计数:论文实体 **3,568,022**,来源记录 **3,815,534**,投稿 **532,223**,决定来源观察 **571,775**。实体数量不是已经完成最终科学工作去重的数量;同一次投稿可能有多条决定来源观察。
paper `2757497` 保留 ICLR2026 一次拒稿投稿下的两条来源观察,并新增 ISMIR2026 录用投稿;实际 CLI 查询显示已知录用,旧拒稿历史仍可查询。不将两条来源观察计作两次拒稿,也不将来源采集时间当作决定时间。
## 固定范围存在性盘点
只读快照 `2026-11-18 06:46:44.099675+00`,范围仍为 43 会、278 个目标年度槽位:**274 个至少有一条投稿,4 个完全为空**,均为 2026 年的 **CIKM、ECCV、ICDM、IROS**。首批ISMIR使空槽从5降至4;第二批只增加同届来源覆盖,空槽数不变。
这只统计“该年度至少有数据”,不是逐篇、逐 track、摘要或公开拒稿覆盖完整。2020–2025 没有整年空槽,仍有来源完整性和公开负面历史缺口。
|证据(相对服务器 `/data/paper-index/`)|SHA256|
|---|---|
|`collected/active-scope-census-20261118-v22/summary.json`|`ee4ec2e6d6eaed55ce3475e735bfba9bd94e512893cd3c3024035b48b744f475`|
|`collected/active-scope-census-20261118-v22/snapshot.json`|`5aabc91217ad787946ba50d6a670296b1e3152a54f5ecaa0e3a9b3a952b63fa7`|
|`validation/active-scope-fixed278-v1.json`|`a835bef5b9c3a0f76069d840cecdb9a6bcf65d3a38796d70dbfdfd4b49242264`|
本机使用固定分母、会议别名和逐 edition 快照独立重算 274/278;没有使用 `max(id)` 代替计数。
## 尚待完成
- ISMIR 仍有109项;62项无精确标题候选不等于新作品,6项作者部分匹配仍待审查。31项摘要逐字一致候选已分两批审查并消费;其余候选不能仅凭标题、作者或摘要检索结果算作已批准。
- MM 已验收572项来源观察,注册清单1918项中仍有1346项待处理。已有134个暂建索引实体仍须后续身份审计;H2AL 来源及轨道纠正、版本化来源覆盖方案尚未用于生产,不能把不同版本轨道预期数相加当完整分母。
- ECCV 的初步节目、ICDM 的研究/应用来源和缺失摘要仍需作品及出版核查;CIKM 缺实际逐篇目录;IROS 官方访问403仍保留限制,不重试绕过。各来源的采集、批准和导入状态必须分开。
- 公开 rejected、desk-rejected、withdrawn 继续按会议、年份、track核查。ISMIR 的101组自愿公开评审来自录用节目,不是101条拒稿;UAI accepted-only 镜像与 EMNLP/NeurIPS 原始接口403均不能关闭拒稿缺口。近期镜像差集核查见 [PR #72](https://gitea.entroia.com/Zayden-Jung/paper-atlas/pulls/72)。
- 后续仍需完成跨会作品与历史、独立类型和标签关系(第15步);全部导入结束后的全库去重、标识冲突、近似标题作者、摘要/约束/查询及覆盖审计(第16步);长期运行与恢复文档最终整理(第17步)。
已消费输入不得重放。下次从[ISMIR 当前增量父状态和恢复说明](../ismir2026-program.md)及 `work-state.json` 的本批已接受回执继续;保留旧 SQLite 只读备份,不因查询超时重新执行导入。
@@ -0,0 +1,247 @@
{
"as_of": "2026-11-18",
"goal_progress": "13/17; stage14 incomplete",
"source": "ismir_program",
"scope": "First10 reviewed existing-work source links, partial10/140, not public-rejection completeness or final deduplication",
"database_acceptance": {
"status": "passed",
"at": "2026-11-18 05:15:27.565655+00:00",
"writer_commit": "12f95cff91813a2143bc8fab5bc3e89df1bd0fdb",
"before_counts": {
"papers": {
"rows": 3568022,
"max_id": 3568595
},
"source_records": {
"rows": 3815503,
"max_id": 3815504
},
"submissions": {
"rows": 532192,
"max_id": 532193
},
"decisions": {
"rows": 571744,
"max_id": 571744
}
},
"after_counts": {
"papers": {
"rows": 3568022,
"max_id": 3568595
},
"source_records": {
"rows": 3815513,
"max_id": 3815514
},
"submissions": {
"rows": 532202,
"max_id": 532203
},
"decisions": {
"rows": 571754,
"max_id": 571754
}
},
"new_papers": 0,
"new_sources": 1,
"new_editions": 1,
"new_source_records": 10,
"new_attempts": 10,
"new_decisions": 10,
"source_id": 123160,
"source_tag_evidence_links": 94,
"protected_papers": 6394,
"all_protected_old_profiles_sources_tags_identifiers_preserved": true,
"source_observations": 10,
"source_expected": 140,
"source_remaining": 130,
"coverage": "partial",
"foundry_readyz": [
200
],
"sqlite_frozen": true,
"imported": [
{
"external_id": "2026/3",
"paper_id": 2813483,
"record_id": 3815505,
"submission_id": 532194,
"decision_id": 571745,
"edition_id": 104044
},
{
"external_id": "2026/12",
"paper_id": 2785459,
"record_id": 3815506,
"submission_id": 532195,
"decision_id": 571746,
"edition_id": 104044
},
{
"external_id": "2026/35",
"paper_id": 2757859,
"record_id": 3815507,
"submission_id": 532196,
"decision_id": 571747,
"edition_id": 104044
},
{
"external_id": "2026/51",
"paper_id": 2752643,
"record_id": 3815508,
"submission_id": 532197,
"decision_id": 571748,
"edition_id": 104044
},
{
"external_id": "2026/74",
"paper_id": 2762340,
"record_id": 3815509,
"submission_id": 532198,
"decision_id": 571749,
"edition_id": 104044
},
{
"external_id": "2026/75",
"paper_id": 2757480,
"record_id": 3815510,
"submission_id": 532199,
"decision_id": 571750,
"edition_id": 104044
},
{
"external_id": "2026/78",
"paper_id": 2768260,
"record_id": 3815511,
"submission_id": 532200,
"decision_id": 571751,
"edition_id": 104044
},
{
"external_id": "2026/84",
"paper_id": 2740614,
"record_id": 3815512,
"submission_id": 532201,
"decision_id": 571752,
"edition_id": 104044
},
{
"external_id": "2026/87",
"paper_id": 2752271,
"record_id": 3815513,
"submission_id": 532202,
"decision_id": 571753,
"edition_id": 104044
},
{
"external_id": "2026/363",
"paper_id": 2757497,
"record_id": 3815514,
"submission_id": 532203,
"decision_id": 571754,
"edition_id": 104044
}
]
},
"history_protection": {
"submissions": {
"rows": 532192,
"sha256": "34e2fe27cccf89e6e80228318212a846369fdd315c4cfe67d7d535013eac8298"
},
"decisions": {
"rows": 571744,
"sha256": "efbda4e91950c5c7298f536bc5967f7fee57d8b20a7843e00dc69871b1553bd1"
},
"decision_sources": {
"rows": 571744,
"sha256": "1d8899b16b40e9f37ac84dc601efc3e7906af5c06bca49946100697f22c96174"
},
"submission_sources": {
"rows": 581976,
"sha256": "ab3ab200c4abbfef39bc5a2a757550191441c703ae505372d38d9afa9a4daf65"
},
"submission_identifiers": {
"rows": 383956,
"sha256": "0e8f514e186e08bd00a65f8e14fc7f63db6add76722271e0111e14f5d27b9dbe"
}
},
"evidence": {
"accepted_report": "/data/paper-index/collected/ismir2026-import10-v1/postgresql-postcheck-v1.json",
"accepted_report_sha256": "8a08015bbd725b400de53477837e850ca92a50479d7f86f2cfed56d50bb22577",
"baseline": "/data/paper-index/collected/ismir2026-baseline10-v1/baseline.json",
"baseline_sha256": "a0b5b8c2c21ee567cf80152b16f66643309bbe33ee9cd67d19da33d6b6337656",
"baseline_archive_sha256": "eb94f99cb53c9a30f63d34ea11b23c11759e45b571b7efce2dcbe8cb4fe52fb8",
"preparation_sha256": "0d9fad74640ab57fc8bea8aaf72ba07202a40b174d05253db7348b14455a0462",
"acceptance_archive": "/data/paper-index/validation/ismir2026-import10-acceptance-recovery-v1.tar.gz",
"acceptance_archive_sha256": "138e1fb986f5aa6a42794eed982fdc91ae994b8be82f8bf25360db2f7ba6c70c",
"fresh_packets_sha256": "88e423aede0ea83b9f736891c09b12d0cb24f2f8b70508e4863cd07760de5e0d",
"local_acceptance_check_sha256": "42d31337c76b96f93505011db7c797123b67ce2eec7defd0175ea0e9c1df2089",
"census_summary_sha256": "2157035b72d351e2fe91f2bc0c6ebb305f91d95f8772ab9a088076ce4b86e1c2"
},
"native_query_example": {
"paper_id": 2757497,
"publication_outcome": "accepted",
"prior_venue": "ICLR",
"prior_year": 2026,
"prior_submission_id": 131721,
"prior_negative_attempts": 1,
"prior_negative_observations": 2,
"prior_decision_ids": [
133086,
538172
],
"later_observed_venue": "ISMIR",
"later_observed_year": 2026,
"original_ismir_signed_decision_verified": false,
"ismir_decision_date": null,
"observation_time_is_not_decision_time": true
},
"presence_census": {
"snapshot_at": "2026-11-18 05:20:48.641315+00",
"scope": "Fixed278 venue-year presence only, not paper/track/abstract/public-negative completeness or final deduplication. Legacy43 gaps deferred, not filled.",
"target_venues": 43,
"expected_slots": 278,
"populated_slots": 274,
"missing_slots": 4,
"missing": [
{
"venue": "CIKM",
"year": 2026
},
{
"venue": "ECCV",
"year": 2026
},
{
"venue": "ICDM",
"year": 2026
},
{
"venue": "IROS",
"year": 2026
}
],
"delta_from_v20": {
"newly_present": [
{
"venue": "ISMIR",
"year": 2026
}
],
"previous_missing": 5,
"current_missing": 4
},
"snapshot_sha256": "c14bab883c5fcab33b450fcf240dec7225f2f17bc04c3e728b5583657b4ad23b",
"fixed_scope_sha256": "a835bef5b9c3a0f76069d840cecdb9a6bcf65d3a38796d70dbfdfd4b49242264",
"accepted_import_sha256": "8a08015bbd725b400de53477837e850ca92a50479d7f86f2cfed56d50bb22577",
"database_written": false
},
"consumed_queue": "/data/paper-index/collected/ismir2026-prepared10-v1: NEVER REPLAY",
"next": [
"Review130 remaining source observations, including31 total verbatim-abstract candidates of which10 consumed; do not count a title/author screen as identity approval.",
"Fresh incremental parent must be partial10/140; old preparation0/140 is consumed.",
"Continue all venue/year/track public negative availability and formal source coverage.",
"Finish cross-venue identities/types, then final whole-DB dedup/identifier/abstract/constraints/query audit after imports."
]
}
@@ -0,0 +1,330 @@
{
"as_of": "2026-11-18",
"goal_progress": "13/17; stage14 incomplete",
"source": "ismir_program",
"scope": "21 additional reviewed existing-work source links; cumulativepartial31/140, not public-rejection completeness or final deduplication",
"database_acceptance": {
"status": "passed",
"at": "2026-11-18 06:38:55.148539+00:00",
"writer_commit": "3754545003e1f96b88c6533cc3a69dc557f07305",
"before_counts": {
"papers": {
"rows": 3568022,
"max_id": 3568595
},
"source_records": {
"rows": 3815513,
"max_id": 3815514
},
"submissions": {
"rows": 532202,
"max_id": 532203
},
"decisions": {
"rows": 571754,
"max_id": 571754
}
},
"after_counts": {
"papers": {
"rows": 3568022,
"max_id": 3568595
},
"source_records": {
"rows": 3815534,
"max_id": 3815535
},
"submissions": {
"rows": 532223,
"max_id": 532224
},
"decisions": {
"rows": 571775,
"max_id": 571775
}
},
"new_papers": 0,
"new_sources": 0,
"new_editions": 0,
"new_source_records": 21,
"new_attempts": 21,
"new_decisions": 21,
"source_id": 123160,
"source_tag_evidence_links": 173,
"protected_papers": 6415,
"all_protected_old_profiles_sources_tags_identifiers_preserved": true,
"source_observations": 31,
"source_expected": 140,
"source_remaining": 109,
"coverage": "partial",
"foundry_readyz": [
200
],
"sqlite_frozen": true,
"imported": [
{
"external_id": "2026/148",
"paper_id": 2755057,
"record_id": 3815515,
"submission_id": 532204,
"decision_id": 571755,
"edition_id": 104044
},
{
"external_id": "2026/184",
"paper_id": 2752723,
"record_id": 3815516,
"submission_id": 532205,
"decision_id": 571756,
"edition_id": 104044
},
{
"external_id": "2026/196",
"paper_id": 2784571,
"record_id": 3815517,
"submission_id": 532206,
"decision_id": 571757,
"edition_id": 104044
},
{
"external_id": "2026/240",
"paper_id": 2738896,
"record_id": 3815518,
"submission_id": 532207,
"decision_id": 571758,
"edition_id": 104044
},
{
"external_id": "2026/251",
"paper_id": 2753889,
"record_id": 3815519,
"submission_id": 532208,
"decision_id": 571759,
"edition_id": 104044
},
{
"external_id": "2026/258",
"paper_id": 2662030,
"record_id": 3815520,
"submission_id": 532209,
"decision_id": 571760,
"edition_id": 104044
},
{
"external_id": "2026/270",
"paper_id": 2769109,
"record_id": 3815521,
"submission_id": 532210,
"decision_id": 571761,
"edition_id": 104044
},
{
"external_id": "2026/298",
"paper_id": 2752114,
"record_id": 3815522,
"submission_id": 532211,
"decision_id": 571762,
"edition_id": 104044
},
{
"external_id": "2026/317",
"paper_id": 2765378,
"record_id": 3815523,
"submission_id": 532212,
"decision_id": 571763,
"edition_id": 104044
},
{
"external_id": "2026/318",
"paper_id": 2764935,
"record_id": 3815524,
"submission_id": 532213,
"decision_id": 571764,
"edition_id": 104044
},
{
"external_id": "2026/342",
"paper_id": 2738585,
"record_id": 3815525,
"submission_id": 532214,
"decision_id": 571765,
"edition_id": 104044
},
{
"external_id": "2026/370",
"paper_id": 2830871,
"record_id": 3815526,
"submission_id": 532215,
"decision_id": 571766,
"edition_id": 104044
},
{
"external_id": "2026/371",
"paper_id": 2858445,
"record_id": 3815527,
"submission_id": 532216,
"decision_id": 571767,
"edition_id": 104044
},
{
"external_id": "2026/373",
"paper_id": 2820562,
"record_id": 3815528,
"submission_id": 532217,
"decision_id": 571768,
"edition_id": 104044
},
{
"external_id": "2026/409",
"paper_id": 2522863,
"record_id": 3815529,
"submission_id": 532218,
"decision_id": 571769,
"edition_id": 104044
},
{
"external_id": "2026/418",
"paper_id": 2750749,
"record_id": 3815530,
"submission_id": 532219,
"decision_id": 571770,
"edition_id": 104044
},
{
"external_id": "2026/468",
"paper_id": 2846824,
"record_id": 3815531,
"submission_id": 532220,
"decision_id": 571771,
"edition_id": 104044
},
{
"external_id": "2026/518",
"paper_id": 2761686,
"record_id": 3815532,
"submission_id": 532221,
"decision_id": 571772,
"edition_id": 104044
},
{
"external_id": "2026/524",
"paper_id": 2784543,
"record_id": 3815533,
"submission_id": 532222,
"decision_id": 571773,
"edition_id": 104044
},
{
"external_id": "2026/539",
"paper_id": 2761760,
"record_id": 3815534,
"submission_id": 532223,
"decision_id": 571774,
"edition_id": 104044
},
{
"external_id": "2026/552",
"paper_id": 2758575,
"record_id": 3815535,
"submission_id": 532224,
"decision_id": 571775,
"edition_id": 104044
}
]
},
"history_protection": {
"submissions": {
"rows": 532202,
"sha256": "9d41955c80fc91dae4f65d65e0f69198ce2d79508ceebfe943bdc19825983df8"
},
"decisions": {
"rows": 571754,
"sha256": "c8dc762d089d1656ee1308dacec9109605e1a4be7e5fcfa8f95013e77a0cb55c"
},
"decision_sources": {
"rows": 571754,
"sha256": "231f15b4c70435977d6ab784112ff5c748541eeb4d54c28bb861f8835f1eba10"
},
"submission_sources": {
"rows": 581986,
"sha256": "b9d8c39e945df340daf1c695a6359cfb549b8bd2f391cb139dabba2c16afa242"
},
"submission_identifiers": {
"rows": 383956,
"sha256": "0e8f514e186e08bd00a65f8e14fc7f63db6add76722271e0111e14f5d27b9dbe"
}
},
"evidence": {
"parent_acceptance_sha256": "8a08015bbd725b400de53477837e850ca92a50479d7f86f2cfed56d50bb22577",
"accepted_report": "/data/paper-index/collected/ismir2026-import21-v1/postgresql-postcheck-v1.json",
"accepted_report_sha256": "153ef13dbac9f4243c81bdd2a16d3ade74372d3e45ca5e84acc2b7258bca7bfc",
"baseline": "/data/paper-index/collected/ismir2026-baseline21-v1/baseline.json",
"baseline_sha256": "204b571b47e164d68930158c2578f609814969940628d421a2b64c3a6ce1b018",
"baseline_archive_sha256": "ecac32e1d3890786a5670ff06f88fa7bd92155306cf8133f3164de0cb9f48e6b",
"preparation_sha256": "6781edfaa67798d76329fb5c7948d83693607c96352a38eccce0067faa95deaa",
"acceptance_archive": "/data/paper-index/validation/ismir2026-import21-acceptance-recovery-v1.tar.gz",
"acceptance_archive_sha256": "55e1cc8301548ff0abc195add5f40d7082a06e4f087f6da5838ffd2f6b33339c",
"fresh_packets_sha256": "79e0e8187a522478c443d036fdbf2f8d5e6a22c949d46fb0592a664ec1bc22fe",
"local_acceptance_check_sha256": "04f4ec6969a71655d70f446347646f63efc56e5a5a8e974b931703482e5e7ceb",
"census_summary_sha256": "ee4ec2e6d6eaed55ce3475e735bfba9bd94e512893cd3c3024035b48b744f475",
"ci_run": 1257,
"ci_tests": 556,
"ci_proof_sha256": "5ad08aa907caec5a1ca63df5e7345db4802dab4e3a91293a0303bf481a903ff8"
},
"consumed_queue": "/data/paper-index/collected/ismir2026-prepared21-v2: NEVER REPLAY",
"previous_consumed_queue": "/data/paper-index/collected/ismir2026-prepared10-v1: NEVER REPLAY",
"query_example_rechecked": {
"paper_id": 2757497,
"publication_outcome": "accepted",
"rejected_attempts": 1,
"rejection_source_observations": 2,
"distinct_ismir_accepted_attempts": 1,
"original_ismir_signed_decision_verified": false,
"ismir_decision_date": null
},
"new_batch_prior_negative_attempts": 0,
"new_batch_prior_negative_observations": 0,
"source_tag_links_cumulative": 267,
"presence_census": {
"snapshot_at": "2026-11-18 06:46:44.099675+00",
"scope": "Fixed278 venue-year presence only, not paper/track/abstract/public-negative completeness or final deduplication. Legacy43 gaps deferred, not filled.",
"target_venues": 43,
"expected_slots": 278,
"populated_slots": 274,
"missing_slots": 4,
"missing": [
{
"venue": "CIKM",
"year": 2026
},
{
"venue": "ECCV",
"year": 2026
},
{
"venue": "ICDM",
"year": 2026
},
{
"venue": "IROS",
"year": 2026
}
],
"delta_from_v21": {
"newly_present": [],
"previous_missing": 4,
"current_missing": 4,
"ISMIR_source_observations": 31
},
"snapshot_sha256": "5aabc91217ad787946ba50d6a670296b1e3152a54f5ecaa0e3a9b3a952b63fa7",
"fixed_scope_sha256": "a835bef5b9c3a0f76069d840cecdb9a6bcf65d3a38796d70dbfdfd4b49242264",
"accepted_import_sha256": "153ef13dbac9f4243c81bdd2a16d3ade74372d3e45ca5e84acc2b7258bca7bfc",
"database_written": false
},
"next": [
"Remaining109 observations require review;31 exact-source-abstract candidates are now consumed, not reusable queues.",
"Future increments require a fresh partial31/140 parent and new preservation baseline.",
"Keep raw authors/title/abstract/keyword versions; no numeric miniconf-to-OpenReview-ID assumption, global person aliases or inferred research types.",
"Continue all43 venue/year/track formal-source and public-negative availability audits, then final whole-DB dedup after imports."
]
}
+33
View File
@@ -48,3 +48,36 @@ paper-atlas collect ismir2026-program \
正式名单与政策恢复包:同目录 `remaining2026-listings-research-v2-recovery.tar.gz`,SHA256 为 `f5ef7c2e73d46712146e4d6711ad5ad562e16b8b55c0cd75d1041cd0b81142cb`。安全恢复三个源目录、核验文件后可重跑上述命令。
完整来源和独立 140 项清单在测试夹具中冻结。测试、构建、打包、安装及实际隔离目录 CLI 验证只在用户指定 `ces-control:host` runner 执行。生产规范化后还需独立核验全部记录,并用 PostgreSQL 只读预检检查候选作品和既往投稿;不自动合并来源差异或人物身份。
## 2026-11-18:首批 10 条验收(历史阶段)
本段保留首批历史状态;当前累计结果和增量父状态见下方最新验收段。
首批 10 项已逐篇核查研究内容,并通过当前数据库实体、作者、摘要及完整历史的只读复核,随后使用已通过远程 CI 的包内 `import records` 导入。新增论文实体 **0**,新增来源记录、投稿、录用观察各 **10**,来源标签证据 **94** 条;规范标题、作者、摘要及旧历史保留。原始来源摘要另存,不覆盖规范摘要。覆盖为 `partial10/140`,还有 **130** 项待核验。数值节目 ID 仍只作为节目定位符,未冒充 OpenReview 投稿 ID,未新增全局人名别名或推断研究类型。
[逐条验收与恢复记录](coverage/ismir2026-import10-2026-11-18.json)包含生产行 ID、保护基线、归档及证据哈希。服务器基线保护此前 6,384 篇与本批 10 篇的并集 **6,394** 篇,全库旧投稿、决定及来源关系哈希未变;独立新快照的 10 个完整实体包、来源字段、标签和标识也逐项核验。Foundry `/readyz` 返回200,17,987,751,936字节的 SQLite 备份保持只读。该验收不是最终全库科学工作去重。
例如 paper `2757497`(Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping)保留一次 ICLR2026 拒稿投稿和两条拒稿来源观察,并新增独立 ISMIR2026 录用投稿。实际执行以下查询,返回 `publication_outcome=accepted`,同时保留原拒稿历史:
```sh
paper-atlas query --pg-service foundry-research --pg-schema research \
--paper 2757497 --outcome accepted
```
ISMIR 的 accepted 来自正式名单和节目,尚无原始签名决定或具体决定日期;不能把采集时间当作决定时间。决定的规范时间使用包内既有毫秒格式,来源回执的原始微秒时间仍保留于元数据。
**禁止重放** `/data/paper-index/collected/ismir2026-prepared10-v1` 的文件。导入单元 `paper-atlas-ismir2026-import10-v1.service`、invocation `538ce9c9324a4f59b5bb49d3094d5265` 已成功结束;该阶段建立了 `partial10/140` 父状态;后续累计结果见下方最新段,不得沿用已消费的 `0/140` 父状态。保护基线恢复包为 `/data/paper-index/validation/ismir2026-baseline10-recovery-v1.tar.gz`,验收恢复包为 `/data/paper-index/validation/ismir2026-import10-acceptance-recovery-v1.tar.gz`;本机 paper-index 工作目录有同名副本及独立检查记录。
## 2026-11-18:新增 21 条,累计 31/140
第二批21项已通过逐篇研究内容审查、新鲜只读父状态检查、保护基线及独立入库验收。新增论文实体、来源和edition均为0;新增来源记录、投稿、录用观察各21条,新增173条来源标签证据。累计31条录用来源观察、267条标签证据,覆盖仍为 `partial31/140`,**109项待核验**。三十一项摘要都保留在原始来源记录中,既有规范资料没有被覆盖。
本批明确保留不同研究工作,例如UOT-IR音乐压缩与SymphonyGen交响生成,以及AI混合音轨占比回归与BAMM真实广播检测;不因作者或主题相近合并。源关键词和展示方式继续使用各自namespace,没有推断全局人名别名、原始OpenReview投稿ID或研究类型。
[第二批逐条验收及恢复记录](coverage/ismir2026-import21-2026-11-18.json)包含21组生产行ID、源码提交、全部证据哈希和新计数。保护并核验了6,415篇作品的规范资料、旧来源与标签,以及所有旧全局投稿/决定/来源关系指纹;新快照另导出21个完整实体包逐字段核验。首批10项保留;paper2757497的实际查询仍显示accepted及原ICLR一次拒稿下的两条来源观察。
本批使用固定源码 `3754545003e1f96b88c6533cc3a69dc557f07305`,已在远程 `ces-control` CI1257通过556项测试,含真实PostgreSQL只读增量门禁回归。该修复解决预检对Mapping行对象切片的错误,未弱化覆盖校验;旧的只读准备v1失败时没有写库,修复后v2重新检查通过。默认运行时未因本批操作切换。
**禁止重放** `/data/paper-index/collected/ismir2026-prepared21-v2`。服务 `paper-atlas-ismir2026-import21-v1.service`、invocation `78077b901a274426bb161ef8d69963ad` 已成功结束。新批次必须从当前 `partial31/140` 准备,不能再用 `10/140` 或 `0/140`。保护基线恢复包为 `/data/paper-index/validation/ismir2026-baseline21-recovery-v1.tar.gz`,验收恢复包为 `/data/paper-index/validation/ismir2026-import21-acceptance-recovery-v1.tar.gz`;本机paper-index目录有同名副本与独立检查结果。
新的固定范围盘点仍为274/278个年度有数据;本批提高ISMIR来源覆盖,没有填补其他空年度,也没有关闭公开拒稿或最终全库去重任务。Foundry就绪检查200,SQLite备份继续只读。
-1
View File
@@ -28,7 +28,6 @@ COMMANDS = {
('collect','eccv2026-program'): ('paper_atlas.collectors.eccv2026', {}, '核验ECCV2026暂定节目并断点采集公开元数据'),
('collect','icassp2026-registry'): ('paper_atlas.collectors.icassp2026', {}, '核验 ICASSP 2026 冻结注册论文元数据'),
('collect','icdm2026-program'): ('paper_atlas.collectors.icdm2026', {}, '核验 ICDM2026 冻结日程及明确的研究/应用轨道'),
('collect','cikm2026-sources'): ('paper_atlas.collectors.cikm2026', {}, '核对 CIKM 2026 节目、附属议程与 Crossref 出版登记'),
('collect','sdm2026-proceedings'): ('paper_atlas.collectors.sdm2026', {}, '核验 SDM 2026 注册论文并单独保留公开日程版本'),
('collect','icaps2026-program'): ('paper_atlas.collectors.icaps2026', {}, '核验 ICAPS 2026 官方录用名单并分开既有论文展示'),
('collect','rss2026-proceedings'): ('paper_atlas.collectors.rss2026', {}, '核验 RSS 2026 官方论文集与独立录用表'),
-164
View File
@@ -1,164 +0,0 @@
"""Collect CIKM 2026 programme and Crossref metadata from frozen snapshots."""
import argparse
from collections import Counter
from hashlib import sha256
import json
from pathlib import Path
import re
from paper_atlas.io.html import Document, clean
PROGRAM = 'https://cikm2026.diag.uniroma1.it/cikm-program/program.html'
VOLUME = '10.1145/3799682'
CONTAINER = 'Proceedings of the 35th ACM International Conference on Information and Knowledge Management'
TRACKS = {'FULL': 596, 'SHORT': 234, 'APPLIED': 86, 'RESOURCE': 46, 'DEMO': 46}
SUPPLEMENTAL = {'KEYNOTE', 'TUTORIAL', 'INDUSTRY', 'POSTER', 'PHD', 'PHD · REMOTE', 'WORKSHOP', 'ANALYTICUP'}
PAGE_COUNTS = (250, 250, 250, 250, 81)
HASHES = {
'program.html': '3eaa668b6aaf09a671d8a3e19a2889b9ab21afaa7ded013794afa94bf35281d6',
'CIKM2026_program.html': 'e3630bd1247df713b8b618f1d3cd9709e7020b2524bb677f82a10f83badd1732',
'crossref-parent.json': 'f3370418d86ccfcfaeb972073bc5fcd1c1c737caed96570fc83f99db0f7d204a',
'crossref-page-1.json': '16bed29f8ee82d668b124c369b632c5c1de0753bdbec7eb72ecb22ffdda5c710',
'crossref-page-2.json': '54424ca31d99462394c9fde5fecfd7dd16337351d21ede3169f19af54d4f49c5',
'crossref-page-3.json': '2dc31d89f41966614ed66a11d1ebaa4b2a6904dcf92945b7185e72e7e65382dd',
'crossref-page-4.json': 'fbbec2b52650dc99a0251c743f035a6fd95be263b04c55c12134cf8473bdde76',
'crossref-page-5.json': '06151c5f770c1e37e8ea9dabe5210be9c969a14fa3bf9925a2d7730ced47d6e5',
'crossref-pages.json': '8be1f8f05139f7a1231b73ad6430f28d2f625809f1530cd9de654f8b3c68c1ae',
}
def fields(node, css_class):
return [child for child in node.walk() if css_class in child.attrs.get('class', '').split()]
def parse_program(raw, *, require_doi):
document = Document(raw.decode('utf-8'))
papers = {}
for card in fields(document.root, 'paper'):
pid = card.attrs.get('data-pid')
if not pid:
continue
titles = fields(card, 'ptitle')
labels = fields(card, 'badge')
abstracts = fields(card, 'abstxt')
if pid in papers or len(titles) != 1 or len(labels) != 1 or len(abstracts) > 1:
raise ValueError(f'Invalid or repeated programme card {pid}')
title = clean(titles[0].text())
label = clean(labels[0].text())
authors = [clean(author.text()) for author in fields(card, 'au')]
abstract = clean(abstracts[0].text()) if abstracts else None
doi_url = titles[0].attrs.get('href')
doi = doi_url.removeprefix('https://doi.org/') if doi_url else None
if label not in TRACKS or not title or not authors or not all(authors):
raise ValueError(f'Incomplete programme metadata {pid}')
if require_doi and (not doi or not re.fullmatch(re.escape(VOLUME) + r'\.\d+', doi) or not abstract):
raise ValueError(f'Missing programme DOI or abstract {pid}')
papers[pid] = {'programme_id': pid, 'title': title, 'author_display_names': authors,
'abstract': abstract, 'programme_label': label, 'programme_doi_link': doi_url,
'doi': doi if require_doi else None}
return papers
def parse_supplemental(raw):
document = Document(raw.decode('utf-8'))
events = {}
for session in fields(document.root, 'sessblock'):
section = session.attrs.get('id', '')
if not section.startswith(('pre-', 'kn-')):
continue
for card in fields(session, 'paper'):
labels = fields(card, 'badge')
if len(labels) != 1:
raise ValueError(f'Ambiguous supplemental label in {section}')
label = clean(labels[0].text())
if label not in SUPPLEMENTAL:
continue
titles = fields(card, 'ptitle')
abstracts = fields(card, 'abstxt')
locator = card.attrs.get('id') or section
if locator in events or len(titles) != 1 or len(abstracts) > 1:
raise ValueError(f'Ambiguous supplemental entry {locator}')
events[locator] = {'programme_locator': locator, 'programme_section': section,
'programme_label': label, 'title': clean(titles[0].text()),
'author_display_names': [clean(a.text()) for a in fields(card, 'au')],
'abstract': clean(abstracts[0].text()) if abstracts else None}
return events
def parse_registry(snapshots):
volume = json.loads(snapshots['crossref-parent.json'])['message']
if (volume['DOI'], volume['title'], volume['ISBN']) != (VOLUME, [CONTAINER], ['9798400725395']):
raise ValueError('Unexpected Crossref volume')
records = {}
for number, count in enumerate(PAGE_COUNTS, 1):
name = f'crossref-page-{number}.json'
message = json.loads(snapshots[name])['message']
if message['total-results'] != 1081 or len(message['items']) != count:
raise ValueError(f'Unexpected Crossref pagination in {name}')
for item in message['items']:
doi = item['DOI'].lower()
if (doi in records or not re.fullmatch(re.escape(VOLUME) + r'\.\d+', doi)
or item.get('container-title') != [CONTAINER] or item.get('type') != 'proceedings-article'
or item.get('published', {}).get('date-parts') != [[2026, 11, 6]]):
raise ValueError(f'Unexpected Crossref record {doi}')
records[doi] = {'registry': item, 'source_file': name}
return records
def collect(snapshots, output):
snapshots = Path(snapshots)
output = Path(output)
if output.exists() or output.is_symlink():
raise FileExistsError(output)
bodies = {}
for name, expected in HASHES.items():
path = snapshots / name
if path.is_symlink() or not path.is_file() or path.stat().st_size > 8 * 1024**2:
raise ValueError(f'Invalid snapshot file {name}')
raw = path.read_bytes()
if sha256(raw).hexdigest() != expected:
raise ValueError(f'Snapshot hash mismatch for {name}')
bodies[name] = raw
programme = parse_program(bodies['program.html'], require_doi=True)
alternate = parse_program(bodies['CIKM2026_program.html'], require_doi=False)
supplemental = parse_supplemental(bodies['program.html'])
registry = parse_registry(bodies)
programme_dois = {r['doi'] for r in programme.values()}
tracks = Counter(r['programme_label'] for r in programme.values())
if (tracks != TRACKS or len(programme_dois) != len(programme) or programme_dois - registry.keys()
or len(alternate) != 1005 or alternate.keys() - programme.keys()
or len(supplemental) != 73 or len(registry) - len(programme_dois) != 73):
raise ValueError('Unexpected programme or registry coverage')
groups = {
'programme': programme,
'alternate-programme': alternate,
'supplemental-programme': supplemental,
'crossref': registry,
}
output.mkdir(parents=True, exist_ok=False)
files = {}
for name, rows in groups.items():
filename = name + '.jsonl'
payload = ''.join(json.dumps({'source_key': key, **row}, ensure_ascii=False) + '\n'
for key, row in sorted(rows.items())).encode()
(output / filename).write_bytes(payload)
files[filename] = {'records': len(rows), 'sha256': sha256(payload).hexdigest()}
report = {'status': 'collected_pending_identity_review', 'files': files, 'input_sha256': HASHES,
'programme_by_label': dict(tracks), 'programme_dois_in_registry': len(programme_dois),
'registry_outside_doi_programme': sorted(registry.keys() - programme_dois),
'programme_ids_missing_from_alternate': sorted(programme.keys() - alternate.keys()),
'database_written': False, 'public_negative_decisions_verified': False}
partial = output / 'collection-report.json.part'
partial.write_text(json.dumps(report, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
partial.replace(output / 'collection-report.json')
return report
def main(argv=None, *, prog=None):
parser = argparse.ArgumentParser(prog=prog, description=__doc__)
parser.add_argument('--snapshots', type=Path, required=True)
parser.add_argument('--out', type=Path, required=True)
options = parser.parse_args(argv)
print(json.dumps(collect(options.snapshots, options.out), ensure_ascii=False))