Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a980b1a1df | |||
| e38b2f1ca8 |
@@ -117,8 +117,6 @@ CI 在已授权的远程 runner 上运行测试、构建 wheel,并从源码目
|
|||||||
|
|
||||||
已导入整卷的逐条轨道修正见 [edition-tracks 维护流程](docs/edition-tracks.md):按完整证据生成只读计划,保留原投稿和决定 ID,事务性修正 edition、标识与覆盖,并防止旧整卷刷新覆盖分类。
|
已导入整卷的逐条轨道修正见 [edition-tracks 维护流程](docs/edition-tracks.md):按完整证据生成只读计划,保留原投稿和决定 ID,事务性修正 edition、标识与覆盖,并防止旧整卷刷新覆盖分类。
|
||||||
|
|
||||||
完整出版来源的新旧版本导致分轨或摘要变化时,使用 [source-revisions 流程](docs/source-revisions.md):核对同一 DOI 全集,保存完整修正前数据与版本链,支持已有记录的目标 track,按新版本重算分区覆盖数,并阻止旧版元数据刷新。普通增量导入仍禁止缩小预期数量。
|
|
||||||
|
|
||||||
入库前可使用通用的[来源身份只读预检](docs/source-identity-audit.md),导出候选论文、稳定标识和既有投稿决定证据;候选结果需要审核,不会自动批准导入。
|
入库前可使用通用的[来源身份只读预检](docs/source-identity-audit.md),导出候选论文、稳定标识和既有投稿决定证据;候选结果需要审核,不会自动批准导入。
|
||||||
|
|
||||||
RSS 2026 已独立适配官方论文集与录用表;使用方法和与 2025 demo 分类的区别见 [RSS 2026 来源说明](docs/rss2026-proceedings.md)。
|
RSS 2026 已独立适配官方论文集与录用表;使用方法和与 2025 demo 分类的区别见 [RSS 2026 来源说明](docs/rss2026-proceedings.md)。
|
||||||
|
|||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# 2020 起补漏状态:2026-11-18
|
||||||
|
|
||||||
|
总体仍为 **13/17,第 14 步未完成**。主动补漏从 2020 年起,2000–2019 年已发现线索仍保留,43 个早期历史缺口仍为延期,没有据此算作补齐。更早状态页保留为历史快照。
|
||||||
|
|
||||||
|
## 当前生产结果
|
||||||
|
|
||||||
|
ISMIR 两批累计31项已关联到现有论文,新增论文实体为0。第二批新增来源记录、独立投稿、录用观察各21条及173条来源标签证据;两批累计267条标签证据。覆盖 `partial31/140`,109项仍待核验。详见[来源说明](../ismir2026-program.md)、[首批验收](ismir2026-import10-2026-11-18.json)和[第二批逐条验收](ismir2026-import21-2026-11-18.json)。
|
||||||
|
|
||||||
|
最新精确计数:论文实体 **3,568,022**,来源记录 **3,815,534**,投稿 **532,223**,决定来源观察 **571,775**。实体数量不是已经完成最终科学工作去重的数量;同一次投稿可能有多条决定来源观察。
|
||||||
|
|
||||||
|
paper `2757497` 保留 ICLR2026 一次拒稿投稿下的两条来源观察,并新增 ISMIR2026 录用投稿;实际 CLI 查询显示已知录用,旧拒稿历史仍可查询。不将两条来源观察计作两次拒稿,也不将来源采集时间当作决定时间。
|
||||||
|
|
||||||
|
## 固定范围存在性盘点
|
||||||
|
|
||||||
|
只读快照 `2026-11-18 06:46:44.099675+00`,范围仍为 43 会、278 个目标年度槽位:**274 个至少有一条投稿,4 个完全为空**,均为 2026 年的 **CIKM、ECCV、ICDM、IROS**。首批ISMIR使空槽从5降至4;第二批只增加同届来源覆盖,空槽数不变。
|
||||||
|
|
||||||
|
这只统计“该年度至少有数据”,不是逐篇、逐 track、摘要或公开拒稿覆盖完整。2020–2025 没有整年空槽,仍有来源完整性和公开负面历史缺口。
|
||||||
|
|
||||||
|
|证据(相对服务器 `/data/paper-index/`)|SHA256|
|
||||||
|
|---|---|
|
||||||
|
|`collected/active-scope-census-20261118-v22/summary.json`|`ee4ec2e6d6eaed55ce3475e735bfba9bd94e512893cd3c3024035b48b744f475`|
|
||||||
|
|`collected/active-scope-census-20261118-v22/snapshot.json`|`5aabc91217ad787946ba50d6a670296b1e3152a54f5ecaa0e3a9b3a952b63fa7`|
|
||||||
|
|`validation/active-scope-fixed278-v1.json`|`a835bef5b9c3a0f76069d840cecdb9a6bcf65d3a38796d70dbfdfd4b49242264`|
|
||||||
|
|
||||||
|
本机使用固定分母、会议别名和逐 edition 快照独立重算 274/278;没有使用 `max(id)` 代替计数。
|
||||||
|
|
||||||
|
## 尚待完成
|
||||||
|
|
||||||
|
- ISMIR 仍有109项;62项无精确标题候选不等于新作品,6项作者部分匹配仍待审查。31项摘要逐字一致候选已分两批审查并消费;其余候选不能仅凭标题、作者或摘要检索结果算作已批准。
|
||||||
|
- MM 已验收572项来源观察,注册清单1918项中仍有1346项待处理。已有134个暂建索引实体仍须后续身份审计;H2AL 来源及轨道纠正、版本化来源覆盖方案尚未用于生产,不能把不同版本轨道预期数相加当完整分母。
|
||||||
|
- ECCV 的初步节目、ICDM 的研究/应用来源和缺失摘要仍需作品及出版核查;CIKM 缺实际逐篇目录;IROS 官方访问403仍保留限制,不重试绕过。各来源的采集、批准和导入状态必须分开。
|
||||||
|
- 公开 rejected、desk-rejected、withdrawn 继续按会议、年份、track核查。ISMIR 的101组自愿公开评审来自录用节目,不是101条拒稿;UAI accepted-only 镜像与 EMNLP/NeurIPS 原始接口403均不能关闭拒稿缺口。近期镜像差集核查见 [PR #72](https://gitea.entroia.com/Zayden-Jung/paper-atlas/pulls/72)。
|
||||||
|
- 后续仍需完成跨会作品与历史、独立类型和标签关系(第15步);全部导入结束后的全库去重、标识冲突、近似标题作者、摘要/约束/查询及覆盖审计(第16步);长期运行与恢复文档最终整理(第17步)。
|
||||||
|
|
||||||
|
已消费输入不得重放。下次从[ISMIR 当前增量父状态和恢复说明](../ismir2026-program.md)及 `work-state.json` 的本批已接受回执继续;保留旧 SQLite 只读备份,不因查询超时重新执行导入。
|
||||||
@@ -0,0 +1,247 @@
|
|||||||
|
{
|
||||||
|
"as_of": "2026-11-18",
|
||||||
|
"goal_progress": "13/17; stage14 incomplete",
|
||||||
|
"source": "ismir_program",
|
||||||
|
"scope": "First10 reviewed existing-work source links, partial10/140, not public-rejection completeness or final deduplication",
|
||||||
|
"database_acceptance": {
|
||||||
|
"status": "passed",
|
||||||
|
"at": "2026-11-18 05:15:27.565655+00:00",
|
||||||
|
"writer_commit": "12f95cff91813a2143bc8fab5bc3e89df1bd0fdb",
|
||||||
|
"before_counts": {
|
||||||
|
"papers": {
|
||||||
|
"rows": 3568022,
|
||||||
|
"max_id": 3568595
|
||||||
|
},
|
||||||
|
"source_records": {
|
||||||
|
"rows": 3815503,
|
||||||
|
"max_id": 3815504
|
||||||
|
},
|
||||||
|
"submissions": {
|
||||||
|
"rows": 532192,
|
||||||
|
"max_id": 532193
|
||||||
|
},
|
||||||
|
"decisions": {
|
||||||
|
"rows": 571744,
|
||||||
|
"max_id": 571744
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"after_counts": {
|
||||||
|
"papers": {
|
||||||
|
"rows": 3568022,
|
||||||
|
"max_id": 3568595
|
||||||
|
},
|
||||||
|
"source_records": {
|
||||||
|
"rows": 3815513,
|
||||||
|
"max_id": 3815514
|
||||||
|
},
|
||||||
|
"submissions": {
|
||||||
|
"rows": 532202,
|
||||||
|
"max_id": 532203
|
||||||
|
},
|
||||||
|
"decisions": {
|
||||||
|
"rows": 571754,
|
||||||
|
"max_id": 571754
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"new_papers": 0,
|
||||||
|
"new_sources": 1,
|
||||||
|
"new_editions": 1,
|
||||||
|
"new_source_records": 10,
|
||||||
|
"new_attempts": 10,
|
||||||
|
"new_decisions": 10,
|
||||||
|
"source_id": 123160,
|
||||||
|
"source_tag_evidence_links": 94,
|
||||||
|
"protected_papers": 6394,
|
||||||
|
"all_protected_old_profiles_sources_tags_identifiers_preserved": true,
|
||||||
|
"source_observations": 10,
|
||||||
|
"source_expected": 140,
|
||||||
|
"source_remaining": 130,
|
||||||
|
"coverage": "partial",
|
||||||
|
"foundry_readyz": [
|
||||||
|
200
|
||||||
|
],
|
||||||
|
"sqlite_frozen": true,
|
||||||
|
"imported": [
|
||||||
|
{
|
||||||
|
"external_id": "2026/3",
|
||||||
|
"paper_id": 2813483,
|
||||||
|
"record_id": 3815505,
|
||||||
|
"submission_id": 532194,
|
||||||
|
"decision_id": 571745,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/12",
|
||||||
|
"paper_id": 2785459,
|
||||||
|
"record_id": 3815506,
|
||||||
|
"submission_id": 532195,
|
||||||
|
"decision_id": 571746,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/35",
|
||||||
|
"paper_id": 2757859,
|
||||||
|
"record_id": 3815507,
|
||||||
|
"submission_id": 532196,
|
||||||
|
"decision_id": 571747,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/51",
|
||||||
|
"paper_id": 2752643,
|
||||||
|
"record_id": 3815508,
|
||||||
|
"submission_id": 532197,
|
||||||
|
"decision_id": 571748,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/74",
|
||||||
|
"paper_id": 2762340,
|
||||||
|
"record_id": 3815509,
|
||||||
|
"submission_id": 532198,
|
||||||
|
"decision_id": 571749,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/75",
|
||||||
|
"paper_id": 2757480,
|
||||||
|
"record_id": 3815510,
|
||||||
|
"submission_id": 532199,
|
||||||
|
"decision_id": 571750,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/78",
|
||||||
|
"paper_id": 2768260,
|
||||||
|
"record_id": 3815511,
|
||||||
|
"submission_id": 532200,
|
||||||
|
"decision_id": 571751,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/84",
|
||||||
|
"paper_id": 2740614,
|
||||||
|
"record_id": 3815512,
|
||||||
|
"submission_id": 532201,
|
||||||
|
"decision_id": 571752,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/87",
|
||||||
|
"paper_id": 2752271,
|
||||||
|
"record_id": 3815513,
|
||||||
|
"submission_id": 532202,
|
||||||
|
"decision_id": 571753,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/363",
|
||||||
|
"paper_id": 2757497,
|
||||||
|
"record_id": 3815514,
|
||||||
|
"submission_id": 532203,
|
||||||
|
"decision_id": 571754,
|
||||||
|
"edition_id": 104044
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"history_protection": {
|
||||||
|
"submissions": {
|
||||||
|
"rows": 532192,
|
||||||
|
"sha256": "34e2fe27cccf89e6e80228318212a846369fdd315c4cfe67d7d535013eac8298"
|
||||||
|
},
|
||||||
|
"decisions": {
|
||||||
|
"rows": 571744,
|
||||||
|
"sha256": "efbda4e91950c5c7298f536bc5967f7fee57d8b20a7843e00dc69871b1553bd1"
|
||||||
|
},
|
||||||
|
"decision_sources": {
|
||||||
|
"rows": 571744,
|
||||||
|
"sha256": "1d8899b16b40e9f37ac84dc601efc3e7906af5c06bca49946100697f22c96174"
|
||||||
|
},
|
||||||
|
"submission_sources": {
|
||||||
|
"rows": 581976,
|
||||||
|
"sha256": "ab3ab200c4abbfef39bc5a2a757550191441c703ae505372d38d9afa9a4daf65"
|
||||||
|
},
|
||||||
|
"submission_identifiers": {
|
||||||
|
"rows": 383956,
|
||||||
|
"sha256": "0e8f514e186e08bd00a65f8e14fc7f63db6add76722271e0111e14f5d27b9dbe"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"evidence": {
|
||||||
|
"accepted_report": "/data/paper-index/collected/ismir2026-import10-v1/postgresql-postcheck-v1.json",
|
||||||
|
"accepted_report_sha256": "8a08015bbd725b400de53477837e850ca92a50479d7f86f2cfed56d50bb22577",
|
||||||
|
"baseline": "/data/paper-index/collected/ismir2026-baseline10-v1/baseline.json",
|
||||||
|
"baseline_sha256": "a0b5b8c2c21ee567cf80152b16f66643309bbe33ee9cd67d19da33d6b6337656",
|
||||||
|
"baseline_archive_sha256": "eb94f99cb53c9a30f63d34ea11b23c11759e45b571b7efce2dcbe8cb4fe52fb8",
|
||||||
|
"preparation_sha256": "0d9fad74640ab57fc8bea8aaf72ba07202a40b174d05253db7348b14455a0462",
|
||||||
|
"acceptance_archive": "/data/paper-index/validation/ismir2026-import10-acceptance-recovery-v1.tar.gz",
|
||||||
|
"acceptance_archive_sha256": "138e1fb986f5aa6a42794eed982fdc91ae994b8be82f8bf25360db2f7ba6c70c",
|
||||||
|
"fresh_packets_sha256": "88e423aede0ea83b9f736891c09b12d0cb24f2f8b70508e4863cd07760de5e0d",
|
||||||
|
"local_acceptance_check_sha256": "42d31337c76b96f93505011db7c797123b67ce2eec7defd0175ea0e9c1df2089",
|
||||||
|
"census_summary_sha256": "2157035b72d351e2fe91f2bc0c6ebb305f91d95f8772ab9a088076ce4b86e1c2"
|
||||||
|
},
|
||||||
|
"native_query_example": {
|
||||||
|
"paper_id": 2757497,
|
||||||
|
"publication_outcome": "accepted",
|
||||||
|
"prior_venue": "ICLR",
|
||||||
|
"prior_year": 2026,
|
||||||
|
"prior_submission_id": 131721,
|
||||||
|
"prior_negative_attempts": 1,
|
||||||
|
"prior_negative_observations": 2,
|
||||||
|
"prior_decision_ids": [
|
||||||
|
133086,
|
||||||
|
538172
|
||||||
|
],
|
||||||
|
"later_observed_venue": "ISMIR",
|
||||||
|
"later_observed_year": 2026,
|
||||||
|
"original_ismir_signed_decision_verified": false,
|
||||||
|
"ismir_decision_date": null,
|
||||||
|
"observation_time_is_not_decision_time": true
|
||||||
|
},
|
||||||
|
"presence_census": {
|
||||||
|
"snapshot_at": "2026-11-18 05:20:48.641315+00",
|
||||||
|
"scope": "Fixed278 venue-year presence only, not paper/track/abstract/public-negative completeness or final deduplication. Legacy43 gaps deferred, not filled.",
|
||||||
|
"target_venues": 43,
|
||||||
|
"expected_slots": 278,
|
||||||
|
"populated_slots": 274,
|
||||||
|
"missing_slots": 4,
|
||||||
|
"missing": [
|
||||||
|
{
|
||||||
|
"venue": "CIKM",
|
||||||
|
"year": 2026
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"venue": "ECCV",
|
||||||
|
"year": 2026
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"venue": "ICDM",
|
||||||
|
"year": 2026
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"venue": "IROS",
|
||||||
|
"year": 2026
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"delta_from_v20": {
|
||||||
|
"newly_present": [
|
||||||
|
{
|
||||||
|
"venue": "ISMIR",
|
||||||
|
"year": 2026
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"previous_missing": 5,
|
||||||
|
"current_missing": 4
|
||||||
|
},
|
||||||
|
"snapshot_sha256": "c14bab883c5fcab33b450fcf240dec7225f2f17bc04c3e728b5583657b4ad23b",
|
||||||
|
"fixed_scope_sha256": "a835bef5b9c3a0f76069d840cecdb9a6bcf65d3a38796d70dbfdfd4b49242264",
|
||||||
|
"accepted_import_sha256": "8a08015bbd725b400de53477837e850ca92a50479d7f86f2cfed56d50bb22577",
|
||||||
|
"database_written": false
|
||||||
|
},
|
||||||
|
"consumed_queue": "/data/paper-index/collected/ismir2026-prepared10-v1: NEVER REPLAY",
|
||||||
|
"next": [
|
||||||
|
"Review130 remaining source observations, including31 total verbatim-abstract candidates of which10 consumed; do not count a title/author screen as identity approval.",
|
||||||
|
"Fresh incremental parent must be partial10/140; old preparation0/140 is consumed.",
|
||||||
|
"Continue all venue/year/track public negative availability and formal source coverage.",
|
||||||
|
"Finish cross-venue identities/types, then final whole-DB dedup/identifier/abstract/constraints/query audit after imports."
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,330 @@
|
|||||||
|
{
|
||||||
|
"as_of": "2026-11-18",
|
||||||
|
"goal_progress": "13/17; stage14 incomplete",
|
||||||
|
"source": "ismir_program",
|
||||||
|
"scope": "21 additional reviewed existing-work source links; cumulativepartial31/140, not public-rejection completeness or final deduplication",
|
||||||
|
"database_acceptance": {
|
||||||
|
"status": "passed",
|
||||||
|
"at": "2026-11-18 06:38:55.148539+00:00",
|
||||||
|
"writer_commit": "3754545003e1f96b88c6533cc3a69dc557f07305",
|
||||||
|
"before_counts": {
|
||||||
|
"papers": {
|
||||||
|
"rows": 3568022,
|
||||||
|
"max_id": 3568595
|
||||||
|
},
|
||||||
|
"source_records": {
|
||||||
|
"rows": 3815513,
|
||||||
|
"max_id": 3815514
|
||||||
|
},
|
||||||
|
"submissions": {
|
||||||
|
"rows": 532202,
|
||||||
|
"max_id": 532203
|
||||||
|
},
|
||||||
|
"decisions": {
|
||||||
|
"rows": 571754,
|
||||||
|
"max_id": 571754
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"after_counts": {
|
||||||
|
"papers": {
|
||||||
|
"rows": 3568022,
|
||||||
|
"max_id": 3568595
|
||||||
|
},
|
||||||
|
"source_records": {
|
||||||
|
"rows": 3815534,
|
||||||
|
"max_id": 3815535
|
||||||
|
},
|
||||||
|
"submissions": {
|
||||||
|
"rows": 532223,
|
||||||
|
"max_id": 532224
|
||||||
|
},
|
||||||
|
"decisions": {
|
||||||
|
"rows": 571775,
|
||||||
|
"max_id": 571775
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"new_papers": 0,
|
||||||
|
"new_sources": 0,
|
||||||
|
"new_editions": 0,
|
||||||
|
"new_source_records": 21,
|
||||||
|
"new_attempts": 21,
|
||||||
|
"new_decisions": 21,
|
||||||
|
"source_id": 123160,
|
||||||
|
"source_tag_evidence_links": 173,
|
||||||
|
"protected_papers": 6415,
|
||||||
|
"all_protected_old_profiles_sources_tags_identifiers_preserved": true,
|
||||||
|
"source_observations": 31,
|
||||||
|
"source_expected": 140,
|
||||||
|
"source_remaining": 109,
|
||||||
|
"coverage": "partial",
|
||||||
|
"foundry_readyz": [
|
||||||
|
200
|
||||||
|
],
|
||||||
|
"sqlite_frozen": true,
|
||||||
|
"imported": [
|
||||||
|
{
|
||||||
|
"external_id": "2026/148",
|
||||||
|
"paper_id": 2755057,
|
||||||
|
"record_id": 3815515,
|
||||||
|
"submission_id": 532204,
|
||||||
|
"decision_id": 571755,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/184",
|
||||||
|
"paper_id": 2752723,
|
||||||
|
"record_id": 3815516,
|
||||||
|
"submission_id": 532205,
|
||||||
|
"decision_id": 571756,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/196",
|
||||||
|
"paper_id": 2784571,
|
||||||
|
"record_id": 3815517,
|
||||||
|
"submission_id": 532206,
|
||||||
|
"decision_id": 571757,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/240",
|
||||||
|
"paper_id": 2738896,
|
||||||
|
"record_id": 3815518,
|
||||||
|
"submission_id": 532207,
|
||||||
|
"decision_id": 571758,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/251",
|
||||||
|
"paper_id": 2753889,
|
||||||
|
"record_id": 3815519,
|
||||||
|
"submission_id": 532208,
|
||||||
|
"decision_id": 571759,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/258",
|
||||||
|
"paper_id": 2662030,
|
||||||
|
"record_id": 3815520,
|
||||||
|
"submission_id": 532209,
|
||||||
|
"decision_id": 571760,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/270",
|
||||||
|
"paper_id": 2769109,
|
||||||
|
"record_id": 3815521,
|
||||||
|
"submission_id": 532210,
|
||||||
|
"decision_id": 571761,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/298",
|
||||||
|
"paper_id": 2752114,
|
||||||
|
"record_id": 3815522,
|
||||||
|
"submission_id": 532211,
|
||||||
|
"decision_id": 571762,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/317",
|
||||||
|
"paper_id": 2765378,
|
||||||
|
"record_id": 3815523,
|
||||||
|
"submission_id": 532212,
|
||||||
|
"decision_id": 571763,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/318",
|
||||||
|
"paper_id": 2764935,
|
||||||
|
"record_id": 3815524,
|
||||||
|
"submission_id": 532213,
|
||||||
|
"decision_id": 571764,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/342",
|
||||||
|
"paper_id": 2738585,
|
||||||
|
"record_id": 3815525,
|
||||||
|
"submission_id": 532214,
|
||||||
|
"decision_id": 571765,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/370",
|
||||||
|
"paper_id": 2830871,
|
||||||
|
"record_id": 3815526,
|
||||||
|
"submission_id": 532215,
|
||||||
|
"decision_id": 571766,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/371",
|
||||||
|
"paper_id": 2858445,
|
||||||
|
"record_id": 3815527,
|
||||||
|
"submission_id": 532216,
|
||||||
|
"decision_id": 571767,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/373",
|
||||||
|
"paper_id": 2820562,
|
||||||
|
"record_id": 3815528,
|
||||||
|
"submission_id": 532217,
|
||||||
|
"decision_id": 571768,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/409",
|
||||||
|
"paper_id": 2522863,
|
||||||
|
"record_id": 3815529,
|
||||||
|
"submission_id": 532218,
|
||||||
|
"decision_id": 571769,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/418",
|
||||||
|
"paper_id": 2750749,
|
||||||
|
"record_id": 3815530,
|
||||||
|
"submission_id": 532219,
|
||||||
|
"decision_id": 571770,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/468",
|
||||||
|
"paper_id": 2846824,
|
||||||
|
"record_id": 3815531,
|
||||||
|
"submission_id": 532220,
|
||||||
|
"decision_id": 571771,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/518",
|
||||||
|
"paper_id": 2761686,
|
||||||
|
"record_id": 3815532,
|
||||||
|
"submission_id": 532221,
|
||||||
|
"decision_id": 571772,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/524",
|
||||||
|
"paper_id": 2784543,
|
||||||
|
"record_id": 3815533,
|
||||||
|
"submission_id": 532222,
|
||||||
|
"decision_id": 571773,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/539",
|
||||||
|
"paper_id": 2761760,
|
||||||
|
"record_id": 3815534,
|
||||||
|
"submission_id": 532223,
|
||||||
|
"decision_id": 571774,
|
||||||
|
"edition_id": 104044
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"external_id": "2026/552",
|
||||||
|
"paper_id": 2758575,
|
||||||
|
"record_id": 3815535,
|
||||||
|
"submission_id": 532224,
|
||||||
|
"decision_id": 571775,
|
||||||
|
"edition_id": 104044
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"history_protection": {
|
||||||
|
"submissions": {
|
||||||
|
"rows": 532202,
|
||||||
|
"sha256": "9d41955c80fc91dae4f65d65e0f69198ce2d79508ceebfe943bdc19825983df8"
|
||||||
|
},
|
||||||
|
"decisions": {
|
||||||
|
"rows": 571754,
|
||||||
|
"sha256": "c8dc762d089d1656ee1308dacec9109605e1a4be7e5fcfa8f95013e77a0cb55c"
|
||||||
|
},
|
||||||
|
"decision_sources": {
|
||||||
|
"rows": 571754,
|
||||||
|
"sha256": "231f15b4c70435977d6ab784112ff5c748541eeb4d54c28bb861f8835f1eba10"
|
||||||
|
},
|
||||||
|
"submission_sources": {
|
||||||
|
"rows": 581986,
|
||||||
|
"sha256": "b9d8c39e945df340daf1c695a6359cfb549b8bd2f391cb139dabba2c16afa242"
|
||||||
|
},
|
||||||
|
"submission_identifiers": {
|
||||||
|
"rows": 383956,
|
||||||
|
"sha256": "0e8f514e186e08bd00a65f8e14fc7f63db6add76722271e0111e14f5d27b9dbe"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"evidence": {
|
||||||
|
"parent_acceptance_sha256": "8a08015bbd725b400de53477837e850ca92a50479d7f86f2cfed56d50bb22577",
|
||||||
|
"accepted_report": "/data/paper-index/collected/ismir2026-import21-v1/postgresql-postcheck-v1.json",
|
||||||
|
"accepted_report_sha256": "153ef13dbac9f4243c81bdd2a16d3ade74372d3e45ca5e84acc2b7258bca7bfc",
|
||||||
|
"baseline": "/data/paper-index/collected/ismir2026-baseline21-v1/baseline.json",
|
||||||
|
"baseline_sha256": "204b571b47e164d68930158c2578f609814969940628d421a2b64c3a6ce1b018",
|
||||||
|
"baseline_archive_sha256": "ecac32e1d3890786a5670ff06f88fa7bd92155306cf8133f3164de0cb9f48e6b",
|
||||||
|
"preparation_sha256": "6781edfaa67798d76329fb5c7948d83693607c96352a38eccce0067faa95deaa",
|
||||||
|
"acceptance_archive": "/data/paper-index/validation/ismir2026-import21-acceptance-recovery-v1.tar.gz",
|
||||||
|
"acceptance_archive_sha256": "55e1cc8301548ff0abc195add5f40d7082a06e4f087f6da5838ffd2f6b33339c",
|
||||||
|
"fresh_packets_sha256": "79e0e8187a522478c443d036fdbf2f8d5e6a22c949d46fb0592a664ec1bc22fe",
|
||||||
|
"local_acceptance_check_sha256": "04f4ec6969a71655d70f446347646f63efc56e5a5a8e974b931703482e5e7ceb",
|
||||||
|
"census_summary_sha256": "ee4ec2e6d6eaed55ce3475e735bfba9bd94e512893cd3c3024035b48b744f475",
|
||||||
|
"ci_run": 1257,
|
||||||
|
"ci_tests": 556,
|
||||||
|
"ci_proof_sha256": "5ad08aa907caec5a1ca63df5e7345db4802dab4e3a91293a0303bf481a903ff8"
|
||||||
|
},
|
||||||
|
"consumed_queue": "/data/paper-index/collected/ismir2026-prepared21-v2: NEVER REPLAY",
|
||||||
|
"previous_consumed_queue": "/data/paper-index/collected/ismir2026-prepared10-v1: NEVER REPLAY",
|
||||||
|
"query_example_rechecked": {
|
||||||
|
"paper_id": 2757497,
|
||||||
|
"publication_outcome": "accepted",
|
||||||
|
"rejected_attempts": 1,
|
||||||
|
"rejection_source_observations": 2,
|
||||||
|
"distinct_ismir_accepted_attempts": 1,
|
||||||
|
"original_ismir_signed_decision_verified": false,
|
||||||
|
"ismir_decision_date": null
|
||||||
|
},
|
||||||
|
"new_batch_prior_negative_attempts": 0,
|
||||||
|
"new_batch_prior_negative_observations": 0,
|
||||||
|
"source_tag_links_cumulative": 267,
|
||||||
|
"presence_census": {
|
||||||
|
"snapshot_at": "2026-11-18 06:46:44.099675+00",
|
||||||
|
"scope": "Fixed278 venue-year presence only, not paper/track/abstract/public-negative completeness or final deduplication. Legacy43 gaps deferred, not filled.",
|
||||||
|
"target_venues": 43,
|
||||||
|
"expected_slots": 278,
|
||||||
|
"populated_slots": 274,
|
||||||
|
"missing_slots": 4,
|
||||||
|
"missing": [
|
||||||
|
{
|
||||||
|
"venue": "CIKM",
|
||||||
|
"year": 2026
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"venue": "ECCV",
|
||||||
|
"year": 2026
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"venue": "ICDM",
|
||||||
|
"year": 2026
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"venue": "IROS",
|
||||||
|
"year": 2026
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"delta_from_v21": {
|
||||||
|
"newly_present": [],
|
||||||
|
"previous_missing": 4,
|
||||||
|
"current_missing": 4,
|
||||||
|
"ISMIR_source_observations": 31
|
||||||
|
},
|
||||||
|
"snapshot_sha256": "5aabc91217ad787946ba50d6a670296b1e3152a54f5ecaa0e3a9b3a952b63fa7",
|
||||||
|
"fixed_scope_sha256": "a835bef5b9c3a0f76069d840cecdb9a6bcf65d3a38796d70dbfdfd4b49242264",
|
||||||
|
"accepted_import_sha256": "153ef13dbac9f4243c81bdd2a16d3ade74372d3e45ca5e84acc2b7258bca7bfc",
|
||||||
|
"database_written": false
|
||||||
|
},
|
||||||
|
"next": [
|
||||||
|
"Remaining109 observations require review;31 exact-source-abstract candidates are now consumed, not reusable queues.",
|
||||||
|
"Future increments require a fresh partial31/140 parent and new preservation baseline.",
|
||||||
|
"Keep raw authors/title/abstract/keyword versions; no numeric miniconf-to-OpenReview-ID assumption, global person aliases or inferred research types.",
|
||||||
|
"Continue all43 venue/year/track formal-source and public-negative availability audits, then final whole-DB dedup after imports."
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -48,3 +48,36 @@ paper-atlas collect ismir2026-program \
|
|||||||
正式名单与政策恢复包:同目录 `remaining2026-listings-research-v2-recovery.tar.gz`,SHA256 为 `f5ef7c2e73d46712146e4d6711ad5ad562e16b8b55c0cd75d1041cd0b81142cb`。安全恢复三个源目录、核验文件后可重跑上述命令。
|
正式名单与政策恢复包:同目录 `remaining2026-listings-research-v2-recovery.tar.gz`,SHA256 为 `f5ef7c2e73d46712146e4d6711ad5ad562e16b8b55c0cd75d1041cd0b81142cb`。安全恢复三个源目录、核验文件后可重跑上述命令。
|
||||||
|
|
||||||
完整来源和独立 140 项清单在测试夹具中冻结。测试、构建、打包、安装及实际隔离目录 CLI 验证只在用户指定 `ces-control:host` runner 执行。生产规范化后还需独立核验全部记录,并用 PostgreSQL 只读预检检查候选作品和既往投稿;不自动合并来源差异或人物身份。
|
完整来源和独立 140 项清单在测试夹具中冻结。测试、构建、打包、安装及实际隔离目录 CLI 验证只在用户指定 `ces-control:host` runner 执行。生产规范化后还需独立核验全部记录,并用 PostgreSQL 只读预检检查候选作品和既往投稿;不自动合并来源差异或人物身份。
|
||||||
|
|
||||||
|
## 2026-11-18:首批 10 条验收(历史阶段)
|
||||||
|
|
||||||
|
本段保留首批历史状态;当前累计结果和增量父状态见下方最新验收段。
|
||||||
|
|
||||||
|
首批 10 项已逐篇核查研究内容,并通过当前数据库实体、作者、摘要及完整历史的只读复核,随后使用已通过远程 CI 的包内 `import records` 导入。新增论文实体 **0**,新增来源记录、投稿、录用观察各 **10**,来源标签证据 **94** 条;规范标题、作者、摘要及旧历史保留。原始来源摘要另存,不覆盖规范摘要。覆盖为 `partial10/140`,还有 **130** 项待核验。数值节目 ID 仍只作为节目定位符,未冒充 OpenReview 投稿 ID,未新增全局人名别名或推断研究类型。
|
||||||
|
|
||||||
|
[逐条验收与恢复记录](coverage/ismir2026-import10-2026-11-18.json)包含生产行 ID、保护基线、归档及证据哈希。服务器基线保护此前 6,384 篇与本批 10 篇的并集 **6,394** 篇,全库旧投稿、决定及来源关系哈希未变;独立新快照的 10 个完整实体包、来源字段、标签和标识也逐项核验。Foundry `/readyz` 返回200,17,987,751,936字节的 SQLite 备份保持只读。该验收不是最终全库科学工作去重。
|
||||||
|
|
||||||
|
例如 paper `2757497`(Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping)保留一次 ICLR2026 拒稿投稿和两条拒稿来源观察,并新增独立 ISMIR2026 录用投稿。实际执行以下查询,返回 `publication_outcome=accepted`,同时保留原拒稿历史:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
paper-atlas query --pg-service foundry-research --pg-schema research \
|
||||||
|
--paper 2757497 --outcome accepted
|
||||||
|
```
|
||||||
|
|
||||||
|
ISMIR 的 accepted 来自正式名单和节目,尚无原始签名决定或具体决定日期;不能把采集时间当作决定时间。决定的规范时间使用包内既有毫秒格式,来源回执的原始微秒时间仍保留于元数据。
|
||||||
|
|
||||||
|
**禁止重放** `/data/paper-index/collected/ismir2026-prepared10-v1` 的文件。导入单元 `paper-atlas-ismir2026-import10-v1.service`、invocation `538ce9c9324a4f59b5bb49d3094d5265` 已成功结束;该阶段建立了 `partial10/140` 父状态;后续累计结果见下方最新段,不得沿用已消费的 `0/140` 父状态。保护基线恢复包为 `/data/paper-index/validation/ismir2026-baseline10-recovery-v1.tar.gz`,验收恢复包为 `/data/paper-index/validation/ismir2026-import10-acceptance-recovery-v1.tar.gz`;本机 paper-index 工作目录有同名副本及独立检查记录。
|
||||||
|
|
||||||
|
## 2026-11-18:新增 21 条,累计 31/140
|
||||||
|
|
||||||
|
第二批21项已通过逐篇研究内容审查、新鲜只读父状态检查、保护基线及独立入库验收。新增论文实体、来源和edition均为0;新增来源记录、投稿、录用观察各21条,新增173条来源标签证据。累计31条录用来源观察、267条标签证据,覆盖仍为 `partial31/140`,**109项待核验**。三十一项摘要都保留在原始来源记录中,既有规范资料没有被覆盖。
|
||||||
|
|
||||||
|
本批明确保留不同研究工作,例如UOT-IR音乐压缩与SymphonyGen交响生成,以及AI混合音轨占比回归与BAMM真实广播检测;不因作者或主题相近合并。源关键词和展示方式继续使用各自namespace,没有推断全局人名别名、原始OpenReview投稿ID或研究类型。
|
||||||
|
|
||||||
|
[第二批逐条验收及恢复记录](coverage/ismir2026-import21-2026-11-18.json)包含21组生产行ID、源码提交、全部证据哈希和新计数。保护并核验了6,415篇作品的规范资料、旧来源与标签,以及所有旧全局投稿/决定/来源关系指纹;新快照另导出21个完整实体包逐字段核验。首批10项保留;paper2757497的实际查询仍显示accepted及原ICLR一次拒稿下的两条来源观察。
|
||||||
|
|
||||||
|
本批使用固定源码 `3754545003e1f96b88c6533cc3a69dc557f07305`,已在远程 `ces-control` CI1257通过556项测试,含真实PostgreSQL只读增量门禁回归。该修复解决预检对Mapping行对象切片的错误,未弱化覆盖校验;旧的只读准备v1失败时没有写库,修复后v2重新检查通过。默认运行时未因本批操作切换。
|
||||||
|
|
||||||
|
**禁止重放** `/data/paper-index/collected/ismir2026-prepared21-v2`。服务 `paper-atlas-ismir2026-import21-v1.service`、invocation `78077b901a274426bb161ef8d69963ad` 已成功结束。新批次必须从当前 `partial31/140` 准备,不能再用 `10/140` 或 `0/140`。保护基线恢复包为 `/data/paper-index/validation/ismir2026-baseline21-recovery-v1.tar.gz`,验收恢复包为 `/data/paper-index/validation/ismir2026-import21-acceptance-recovery-v1.tar.gz`;本机paper-index目录有同名副本与独立检查结果。
|
||||||
|
|
||||||
|
新的固定范围盘点仍为274/278个年度有数据;本批提高ISMIR来源覆盖,没有填补其他空年度,也没有关闭公开拒稿或最终全库去重任务。Foundry就绪检查200,SQLite备份继续只读。
|
||||||
|
|||||||
@@ -1,57 +0,0 @@
|
|||||||
# 出版来源版本与覆盖修正
|
|
||||||
|
|
||||||
`paper-atlas maintain source-revisions` 用于同一出版清单的新版本:稳定 DOI 全集、标题、作者和决定不变,但官方分轨、摘要或对应来源证据已经过重新核对。它不导入新论文、不合并实体、不推断拒稿,不把出版分类当成研究类型。
|
|
||||||
|
|
||||||
例如,同一清单的“未分类”由 192 条减少到 166 条,而主会等分区增加。直接放松增量导入的计数保护会隐藏清单丢失。本命令先核验完整的新旧清单及数据库快照,再单独修正版本;普通 `import records` 的单调计数规则保持原样。
|
|
||||||
|
|
||||||
## 准备完整证据
|
|
||||||
|
|
||||||
在工作区外保存 inventory JSON,包含以下字段:
|
|
||||||
|
|
||||||
| 字段 | 内容 |
|
|
||||||
| --- | --- |
|
|
||||||
| `format` | `1` |
|
|
||||||
| `source`, `venue`, `year`, `event_key`, `list_url` | 明确的已有来源、会议缩写、年份、事件键和 HTTPS 出版清单地址 |
|
|
||||||
| `previous_rows`, `rows` | 完整旧版、新版规范化来源记录数组,每版最多 10,000 条;每条包含稳定 DOI、来源字段、作者、摘要、track、决定、标签、原始来源 metadata |
|
|
||||||
| `evidence` | 一个或多个 `{ "url": "https://…", "sha256": "…" }`,绑定已独立核验的原始证据文件 |
|
|
||||||
|
|
||||||
两版必须有完全相同的 DOI 集合;仅允许 `abstract`、`track`、`tags`、`metadata` 的有据变化。必须保留 collector 原始字段,不能将工作身份核查产生的额外锚点混入原始来源版本。只有明确 `accepted` 的出版观察可进入本流程。证据 URL/hash 是审核输入,本命令不联网确认它们;执行者必须另行验证文件哈希、完整性及对应关系。
|
|
||||||
|
|
||||||
数据库可以部分导入,甚至各分区的预期数暂时来自不同版本。现有覆盖必须为 `partial`、无错误,导入数与实际来源记录一致;预期数必须恰好属于旧版或新版的同一分区。已有目标 edition 和对应覆盖记录必须存在,目标中可以有其他来源及既有投稿。本版本不会新建 edition,也不会处理共享来源投稿、多次既有 scope curation 的再迁移,或成员集合变化;遇到这些情况会拒绝并要求单独核查。
|
|
||||||
|
|
||||||
## 先生成只读计划
|
|
||||||
|
|
||||||
```sh
|
|
||||||
paper-atlas maintain source-revisions \
|
|
||||||
--pg-service foundry-research --pg-schema research \
|
|
||||||
--inventory /data/paper-index/validation/reviewed-source-version.json \
|
|
||||||
--out /data/paper-index/validation/source-revision-plan.json
|
|
||||||
```
|
|
||||||
|
|
||||||
计划包含明确数据库身份、两个完整来源版本、当前版本父节点、覆盖表原值和修正值、逐条变更,以及受影响会议 scope 内所有论文的规范记录、稳定标识、全部跨会议投稿和决定、来源关系、标签与已存在的 annotation。拒稿后录用的历史也在保护范围内,不能只校验当前会议的录用观察。
|
|
||||||
|
|
||||||
计划和原始证据应保存到服务器与独立恢复位置,核对完整差异及计划文件 SHA-256。执行前重新核对生产保护基线与单写入安排。输出文件必须是不存在的新路径,禁止覆盖旧报告。
|
|
||||||
|
|
||||||
## 按文件哈希应用并验收
|
|
||||||
|
|
||||||
```sh
|
|
||||||
paper-atlas maintain source-revisions \
|
|
||||||
--pg-service foundry-research --pg-schema research \
|
|
||||||
--plan /data/paper-index/validation/source-revision-plan.json \
|
|
||||||
--plan-sha256 <已审核的计划文件SHA256> \
|
|
||||||
--out /data/paper-index/validation/source-revision-result.json
|
|
||||||
```
|
|
||||||
|
|
||||||
应用时重新生成计划并要求完全一致;PostgreSQL 使用已有 writer 锁及相关表写锁,SQLite 使用立即事务。任何保护数据、当前版本或目标标识冲突都使操作失败。操作保留投稿和决定 ID、原始决定证据、规范论文文本、全部稳定标识、来源关系和现有标签;只修改计划列出的投稿/投稿标识 edition,以及来源记录的摘要与 metadata 当前投影。旧投影完整保存在版本历史中,来源原始获取时间不会伪造为修正时间。
|
|
||||||
|
|
||||||
覆盖只调整明确分区的预期数和来源观察数,保留 `partial`、cursor、成功时间和错误字段原值。尚未导入的分区不产生虚构投稿或来源观察。覆盖数不能被解读为科学作品数量,或所有 track、摘要、负面决定已完整。
|
|
||||||
|
|
||||||
同一事务显式增加可共享的可选表 `source_scope_revisions`,保持原 21 张业务表的身份。每个版本存储新来源全集、父版本、完整计划及修正前数据;应用接口只追加版本,不更新或删除旧版本。没有新建 PostgreSQL 实例。该表纳入 SQLite→PostgreSQL 的可选表迁移、全量内容哈希、约束与计数验收。`version_sha256` 是规范化计划内容哈希;CLI 的 `--plan-sha256` 是计划文件字节哈希,二者用途不同。
|
|
||||||
|
|
||||||
提交前在同一事务比较全部保护数据与明确预期的变更;失败则全部回滚,包括新增表和版本记录。相同计划的重放还会重新验收当前状态;如后续数据已变化则拒绝,不会重复迁移或恢复旧状态。完整计划与 before-image 用于审查和恢复准备,本命令没有自动回滚历史版本的入口;不得手工删除版本来绕开刷新保护。
|
|
||||||
|
|
||||||
## 后续刷新及限制
|
|
||||||
|
|
||||||
`import records` 检查当前版本的逐条完整来源字段;`Store.record` 也检查原始文本、作者、来源 metadata 和基础标识,防止直接调用绕回旧版摘要。来源字段必须与当前版本相符,可以额外携带已经核验的工作身份/curation 证据,不能替换来源自身字段。相同来源、会议事件和清单地址下未登记的成员必须另行核验新版本。
|
|
||||||
|
|
||||||
版本表通过包内接口追加,未声明数据库管理员的任意 SQL 为安全接口。新来源或新版本在导入前仍需要独立工作身份核查;姓名、标题或无候选都不证明作品身份。实现测试和版本修正验收也不替代所有导入结束后的全库去重、标识冲突、近似标题/作者、摘要、来源覆盖与查询审计。
|
|
||||||
@@ -46,7 +46,6 @@ COMMANDS = {
|
|||||||
('import','arxiv'): ('paper_atlas.ingestion.arxiv', {}, '导入 arXiv 元数据 ZIP'),
|
('import','arxiv'): ('paper_atlas.ingestion.arxiv', {}, '导入 arXiv 元数据 ZIP'),
|
||||||
('import','records'): ('paper_atlas.ingestion.records', {}, '导入核验过的会议文件'),
|
('import','records'): ('paper_atlas.ingestion.records', {}, '导入核验过的会议文件'),
|
||||||
('maintain','edition-tracks'): ('paper_atlas.ingestion.edition_tracks', {}, '按逐条来源证据修正既有投稿轨道并保护刷新'),
|
('maintain','edition-tracks'): ('paper_atlas.ingestion.edition_tracks', {}, '按逐条来源证据修正既有投稿轨道并保护刷新'),
|
||||||
('maintain','source-revisions'): ('paper_atlas.ingestion.source_revisions', {}, '按完整来源版本核验并修正出版分轨与覆盖数,保留历史并阻止旧版刷新'),
|
|
||||||
('maintain','joint-editions'): ('paper_atlas.ingestion.joint_editions', {}, '按已核验计划修正既有联合会议届次归属'),
|
('maintain','joint-editions'): ('paper_atlas.ingestion.joint_editions', {}, '按已核验计划修正既有联合会议届次归属'),
|
||||||
('maintain','event-identity'): ('paper_atlas.ingestion.event_identity', {}, '按核验计划增加同年会议事件身份'),
|
('maintain','event-identity'): ('paper_atlas.ingestion.event_identity', {}, '按核验计划增加同年会议事件身份'),
|
||||||
('maintain','merge'): ('paper_atlas.ingestion.maintenance', {}, '协调单写入合并与恢复'),
|
('maintain','merge'): ('paper_atlas.ingestion.maintenance', {}, '协调单写入合并与恢复'),
|
||||||
|
|||||||
@@ -1,128 +0,0 @@
|
|||||||
"""Versioned publication-source scopes and guards against stale source refreshes."""
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
|
|
||||||
TABLE = 'source_scope_revisions'
|
|
||||||
MARKER = 'source_scope_revisions_v1'
|
|
||||||
COLUMNS = ('version_sha256', 'source_id', 'venue_id', 'year', 'event_key', 'list_url',
|
|
||||||
'parent_sha256', 'records_json', 'plan_json', 'applied_at')
|
|
||||||
|
|
||||||
|
|
||||||
def available(c):
|
|
||||||
if getattr(c, 'postgres', False):
|
|
||||||
return bool(c.execute("SELECT 1 FROM information_schema.tables WHERE table_schema=current_schema() AND table_name=?", (TABLE,)).fetchone())
|
|
||||||
return bool(c.execute("SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (TABLE,)).fetchone())
|
|
||||||
|
|
||||||
|
|
||||||
def initialize(c):
|
|
||||||
exists = available(c)
|
|
||||||
marker = bool(c.execute('SELECT 1 FROM schema_migrations WHERE name=?', (MARKER,)).fetchone())
|
|
||||||
if exists != marker:
|
|
||||||
raise ValueError('Source revision table/marker mismatch')
|
|
||||||
if exists:
|
|
||||||
result = c.execute('SELECT * FROM source_scope_revisions LIMIT 0')
|
|
||||||
description = result.cursor.description if getattr(c, 'postgres', False) else result.description
|
|
||||||
if tuple(x[0] for x in description) != COLUMNS:
|
|
||||||
raise ValueError('Unexpected source revision schema')
|
|
||||||
return
|
|
||||||
ddl = '''CREATE TABLE source_scope_revisions (
|
|
||||||
version_sha256 TEXT PRIMARY KEY,
|
|
||||||
source_id INTEGER NOT NULL REFERENCES sources(id),
|
|
||||||
venue_id INTEGER NOT NULL REFERENCES venues(id),
|
|
||||||
year INTEGER NOT NULL, event_key TEXT NOT NULL, list_url TEXT NOT NULL,
|
|
||||||
parent_sha256 TEXT UNIQUE REFERENCES source_scope_revisions(version_sha256),
|
|
||||||
records_json TEXT NOT NULL, plan_json TEXT NOT NULL,
|
|
||||||
applied_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP)'''
|
|
||||||
if getattr(c, 'postgres', False):
|
|
||||||
ddl = re.sub(r'\bINTEGER\b', 'BIGINT', ddl)
|
|
||||||
ddl = re.sub(r'\bTEXT\b', 'TEXT COLLATE "C"', ddl)
|
|
||||||
c.execute(ddl)
|
|
||||||
c.execute('CREATE INDEX source_scope_revisions_scope ON source_scope_revisions(source_id,venue_id,year,event_key,list_url)')
|
|
||||||
c.execute('INSERT INTO schema_migrations(name) VALUES(?)', (MARKER,))
|
|
||||||
|
|
||||||
|
|
||||||
def heads(c, source_id):
|
|
||||||
if not available(c):
|
|
||||||
return []
|
|
||||||
if not c.execute('SELECT 1 FROM schema_migrations WHERE name=?', (MARKER,)).fetchone():
|
|
||||||
raise ValueError('Source revision table/marker mismatch')
|
|
||||||
return c.execute('''SELECT v.version_sha256,v.venue_id,v.year,v.event_key,v.list_url
|
|
||||||
FROM source_scope_revisions v WHERE v.source_id=? AND NOT EXISTS
|
|
||||||
(SELECT 1 FROM source_scope_revisions child WHERE child.parent_sha256=v.version_sha256)
|
|
||||||
ORDER BY v.version_sha256''', (source_id,)).fetchall()
|
|
||||||
|
|
||||||
|
|
||||||
def current_rows(c, source_id, cache):
|
|
||||||
active = heads(c, source_id)
|
|
||||||
signatures = tuple(r[0] for r in active)
|
|
||||||
# A new head invalidates the cache, even when maintenance uses the same Store.
|
|
||||||
if cache.get('signatures') != (source_id, signatures):
|
|
||||||
records = {}; scopes = {}
|
|
||||||
for signature in signatures:
|
|
||||||
raw = c.execute('SELECT records_json FROM source_scope_revisions WHERE version_sha256=?', (signature,)).fetchone()[0]
|
|
||||||
scopes[signature] = json.loads(raw)
|
|
||||||
for row in scopes[signature]:
|
|
||||||
ext = row['external_id']
|
|
||||||
if ext in records and records[ext] != row:
|
|
||||||
raise ValueError('Conflicting active source snapshots')
|
|
||||||
records[ext] = row
|
|
||||||
cache.clear()
|
|
||||||
cache.update(signatures=(source_id, signatures), records=records, scopes=scopes)
|
|
||||||
return cache['records']
|
|
||||||
|
|
||||||
|
|
||||||
def subset(expected, actual):
|
|
||||||
"""Reviewed source fields must survive additional work-identity/curation evidence."""
|
|
||||||
return isinstance(actual, dict) and all(k in actual and actual[k] == v for k, v in expected.items())
|
|
||||||
|
|
||||||
|
|
||||||
def require_record(c, source_id, external, url, title, abstract, authors, identifiers, metadata, cache):
|
|
||||||
expected = current_rows(c, source_id, cache).get(external)
|
|
||||||
if expected is None:
|
|
||||||
return
|
|
||||||
actual = {'url': url, 'title': title, 'abstract': abstract, 'authors': authors}
|
|
||||||
if any(actual[k] != expected.get(k) for k in actual):
|
|
||||||
raise ValueError('Stale source snapshot: reconcile the source version before refresh')
|
|
||||||
if not subset(expected.get('metadata', {}), metadata or {}) or not subset(expected.get('identifiers', {}), identifiers or {}):
|
|
||||||
raise ValueError('Stale source snapshot metadata or identifiers')
|
|
||||||
|
|
||||||
|
|
||||||
def scope_head(c, source_id, manifest):
|
|
||||||
venue = c.execute('SELECT id FROM venues WHERE acronym=?', (manifest.get('venue'),)).fetchone()
|
|
||||||
if not venue:
|
|
||||||
return None
|
|
||||||
matches = [r for r in heads(c, source_id)
|
|
||||||
if [r[i] for i in range(1, 5)] == [venue[0], manifest.get('year'), manifest.get('event_key', ''), manifest.get('list_url')]]
|
|
||||||
if len(matches) > 1:
|
|
||||||
raise ValueError('Forked source revision history')
|
|
||||||
return matches[0][0] if matches else None
|
|
||||||
|
|
||||||
|
|
||||||
def require_manifest(c, source_id, manifest, cache):
|
|
||||||
current_rows(c, source_id, cache)
|
|
||||||
signature = scope_head(c, source_id, manifest)
|
|
||||||
if signature is None:
|
|
||||||
return
|
|
||||||
expected = sum(r['track'] == manifest.get('track') for r in cache['scopes'][signature])
|
|
||||||
bound = manifest.get('coverage_expected_count') if manifest.get('coverage_mode') == 'incremental' else manifest.get('count')
|
|
||||||
if expected < 1 or bound != expected or manifest.get('status') != 'partial':
|
|
||||||
raise ValueError('Source coverage manifest differs from current source snapshot')
|
|
||||||
|
|
||||||
|
|
||||||
def require_row(c, source_id, row, manifest, cache):
|
|
||||||
expected = current_rows(c, source_id, cache).get(row['external_id'])
|
|
||||||
scoped_manifest = {**manifest, 'venue': row['venue'], 'year': row['year'], 'event_key': row.get('event_key', '')}
|
|
||||||
signature = scope_head(c, source_id, scoped_manifest)
|
|
||||||
if expected is None:
|
|
||||||
if signature is not None:
|
|
||||||
raise ValueError('Publication absent from current source snapshot; reconcile source membership first')
|
|
||||||
return
|
|
||||||
if signature is None or not any(r['external_id'] == row['external_id'] for r in cache['scopes'][signature]):
|
|
||||||
raise ValueError('Source listing differs from current source snapshot')
|
|
||||||
for key, value in expected.items():
|
|
||||||
if key in ('metadata', 'identifiers'):
|
|
||||||
okay = subset(value, row.get(key))
|
|
||||||
else:
|
|
||||||
okay = row.get(key) == value
|
|
||||||
if not okay:
|
|
||||||
raise ValueError('Stale source snapshot field: ' + key)
|
|
||||||
@@ -96,10 +96,6 @@ class Store:
|
|||||||
if found:
|
if found:
|
||||||
matches.add(found[0])
|
matches.add(found[0])
|
||||||
old = c.execute('SELECT paper_id,title,abstract,retrieved_at,metadata_json FROM source_records WHERE source_id=? AND external_id=?',(source_id:=self.source(source,url),external_id)).fetchone()
|
old = c.execute('SELECT paper_id,title,abstract,retrieved_at,metadata_json FROM source_records WHERE source_id=? AND external_id=?',(source_id:=self.source(source,url),external_id)).fetchone()
|
||||||
from paper_atlas.db.source_revisions import require_record
|
|
||||||
if not hasattr(self, '_source_revision_cache'):
|
|
||||||
self._source_revision_cache = {}
|
|
||||||
require_record(c, source_id, external_id, url, title, abstract, authors, identifiers, metadata, self._source_revision_cache)
|
|
||||||
from paper_atlas.db.edition_scopes import preserve
|
from paper_atlas.db.edition_scopes import preserve
|
||||||
metadata = preserve(dict(metadata or {}), old[4] if old else None)
|
metadata = preserve(dict(metadata or {}), old[4] if old else None)
|
||||||
if old:
|
if old:
|
||||||
|
|||||||
@@ -101,9 +101,6 @@ def ingest_file(db,path, *, pg_service=None, pg_schema=None):
|
|||||||
c.commit()
|
c.commit()
|
||||||
if incremental is not None:
|
if incremental is not None:
|
||||||
require_incremental_parent(c, source_id, manifest, incremental)
|
require_incremental_parent(c, source_id, manifest, incremental)
|
||||||
from paper_atlas.db.source_revisions import require_manifest
|
|
||||||
store._source_revision_cache = {}
|
|
||||||
require_manifest(c, source_id, manifest, store._source_revision_cache)
|
|
||||||
digest = hashlib.sha256()
|
digest = hashlib.sha256()
|
||||||
with path.open('rb') as stream:
|
with path.open('rb') as stream:
|
||||||
for line in stream:
|
for line in stream:
|
||||||
@@ -116,10 +113,6 @@ def ingest_file(db,path, *, pg_service=None, pg_schema=None):
|
|||||||
require_track_record(row, manifest)
|
require_track_record(row, manifest)
|
||||||
from paper_atlas.db.events import validate_event_key
|
from paper_atlas.db.events import validate_event_key
|
||||||
validate_event_key(row.get('event_key'))
|
validate_event_key(row.get('event_key'))
|
||||||
from paper_atlas.db.source_revisions import require_row
|
|
||||||
if not hasattr(store, '_source_revision_cache'):
|
|
||||||
store._source_revision_cache = {}
|
|
||||||
require_row(c, source_id, row, manifest, store._source_revision_cache)
|
|
||||||
external = row['external_id']
|
external = row['external_id']
|
||||||
if external in seen:
|
if external in seen:
|
||||||
raise ValueError('Duplicate external ID in one collected file: '+external)
|
raise ValueError('Duplicate external ID in one collected file: '+external)
|
||||||
|
|||||||
@@ -1,327 +0,0 @@
|
|||||||
"""Plan-bound publication scope revisions with complete before-images, never reimport."""
|
|
||||||
import argparse
|
|
||||||
from collections import Counter
|
|
||||||
from contextlib import closing
|
|
||||||
import copy
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
from pathlib import Path
|
|
||||||
import re
|
|
||||||
|
|
||||||
from paper_atlas.db import source_revisions as versions
|
|
||||||
from paper_atlas.db.backend import database_arguments, open_read_connection, open_store
|
|
||||||
from paper_atlas.db.edition_scopes import KEY
|
|
||||||
from paper_atlas.ingestion.edition_tracks import curation
|
|
||||||
from paper_atlas.ingestion.event_identity import database_identity, digest, values
|
|
||||||
|
|
||||||
MAX_ROWS = 10000
|
|
||||||
|
|
||||||
|
|
||||||
def validate(inventory):
|
|
||||||
required = {'format', 'source', 'venue', 'year', 'event_key', 'list_url',
|
|
||||||
'previous_rows', 'rows', 'evidence'}
|
|
||||||
if not isinstance(inventory, dict) or set(inventory) != required or inventory['format'] != 1:
|
|
||||||
raise ValueError('Unsupported source revision inventory')
|
|
||||||
if (type(inventory['year']) is not int or inventory['year'] < 2000
|
|
||||||
or not all(isinstance(inventory[k], str) and inventory[k] for k in ('source', 'venue', 'list_url'))
|
|
||||||
or not isinstance(inventory['event_key'], str)):
|
|
||||||
raise ValueError('Explicit publication source scope required')
|
|
||||||
if not inventory['list_url'].startswith('https://'):
|
|
||||||
raise ValueError('HTTPS source listing required')
|
|
||||||
evidence = inventory['evidence']
|
|
||||||
if not isinstance(evidence, list) or not 1 <= len(evidence) <= 100:
|
|
||||||
raise ValueError('Bounded source evidence required')
|
|
||||||
for item in evidence:
|
|
||||||
if (not isinstance(item, dict) or set(item) != {'url', 'sha256'}
|
|
||||||
or not isinstance(item['url'], str) or not item['url'].startswith('https://')
|
|
||||||
or not isinstance(item['sha256'], str) or not re.fullmatch('[0-9a-f]{64}', item['sha256'])):
|
|
||||||
raise ValueError('Source evidence URL/hash required')
|
|
||||||
snapshots = []
|
|
||||||
for key in ('previous_rows', 'rows'):
|
|
||||||
items = inventory[key]
|
|
||||||
if not isinstance(items, list) or not 1 <= len(items) <= MAX_ROWS:
|
|
||||||
raise ValueError('Complete bounded source snapshots required')
|
|
||||||
records = {}
|
|
||||||
for row in items:
|
|
||||||
if not isinstance(row, dict) or any(row.get(k) != inventory[k] for k in ('source', 'venue', 'year')):
|
|
||||||
raise ValueError('Source snapshot scope mismatch')
|
|
||||||
ext = row.get('external_id')
|
|
||||||
if (not isinstance(ext, str) or ext in records or row.get('status') != 'accepted'
|
|
||||||
or not re.fullmatch(r'10\.\d{4,9}/\S+', ext)
|
|
||||||
or row.get('identifiers', {}).get('doi') != ext
|
|
||||||
or not isinstance(row.get('track'), str) or not row['track']
|
|
||||||
or row.get('event_key', '') != inventory['event_key']
|
|
||||||
or not isinstance(row.get('title'), str) or not row['title']
|
|
||||||
or not isinstance(row.get('authors'), list)
|
|
||||||
or not isinstance(row.get('metadata'), dict)):
|
|
||||||
raise ValueError('Unique DOI-bearing accepted publication records required')
|
|
||||||
records[ext] = row
|
|
||||||
snapshots.append(records)
|
|
||||||
previous, current = snapshots
|
|
||||||
if set(previous) != set(current):
|
|
||||||
raise ValueError('Source membership changed; this operation cannot add or remove publications')
|
|
||||||
for ext, row in current.items():
|
|
||||||
before = previous[ext]
|
|
||||||
stable = (set(before) | set(row)) - {'abstract', 'track', 'tags', 'metadata'}
|
|
||||||
if any(before.get(k) != row.get(k) for k in stable):
|
|
||||||
raise ValueError('Stable publication identity or decision changed')
|
|
||||||
return previous, current
|
|
||||||
|
|
||||||
|
|
||||||
def bounded(c, sql, params):
|
|
||||||
result = values(c.execute(sql + ' LIMIT 100001', params))
|
|
||||||
if len(result) > 100000:
|
|
||||||
raise ValueError('Source revision protection exceeds bounded scope')
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
def capture(c, inventory):
|
|
||||||
previous, current = validate(inventory)
|
|
||||||
source = c.execute('SELECT id FROM sources WHERE name=?', (inventory['source'],)).fetchone()
|
|
||||||
venue = c.execute('SELECT id FROM venues WHERE acronym=?', (inventory['venue'],)).fetchone()
|
|
||||||
if not source or not venue:
|
|
||||||
raise ValueError('Existing source and venue required')
|
|
||||||
sid, vid = source[0], venue[0]
|
|
||||||
editions = bounded(c, 'SELECT * FROM editions WHERE venue_id=? AND year=? AND event_key=? ORDER BY id',
|
|
||||||
(vid, inventory['year'], inventory['event_key']))
|
|
||||||
by_track = {r[3]: r[0] for r in editions}
|
|
||||||
# The reviewed scope includes all its existing target attempts, even when
|
|
||||||
# they originate from another source. No empty-target shortcut is used.
|
|
||||||
scope_subs = 'SELECT s.id FROM submissions s JOIN editions e ON e.id=s.edition_id WHERE e.venue_id=? AND e.year=? AND e.event_key=?'
|
|
||||||
scope_params = (vid, inventory['year'], inventory['event_key'])
|
|
||||||
scoped = bounded(c, 'SELECT * FROM submissions WHERE id IN (' + scope_subs + ') ORDER BY id', scope_params)
|
|
||||||
own_records = bounded(c, '''SELECT DISTINCT r.* FROM source_records r JOIN submission_sources ss ON ss.source_record_id=r.id
|
|
||||||
WHERE r.source_id=? AND ss.submission_id IN (''' + scope_subs + ') ORDER BY r.id', (sid, *scope_params))
|
|
||||||
if not own_records:
|
|
||||||
raise ValueError('Existing source observations required')
|
|
||||||
if any(r[2] not in current for r in own_records):
|
|
||||||
raise ValueError('Snapshot omits an existing source observation')
|
|
||||||
pids = sorted({r[1] for r in scoped})
|
|
||||||
if not pids or len(pids) > MAX_ROWS:
|
|
||||||
raise ValueError('Bounded nonempty protected paper scope required')
|
|
||||||
marks = ','.join('?' for _ in pids)
|
|
||||||
papers = 'SELECT id FROM papers WHERE id IN (' + marks + ')'
|
|
||||||
subs = 'SELECT id FROM submissions WHERE paper_id IN (' + papers + ')'
|
|
||||||
decisions = 'SELECT id FROM decisions WHERE submission_id IN (' + subs + ')'
|
|
||||||
queries = {
|
|
||||||
'papers': 'SELECT * FROM papers WHERE id IN (' + papers + ') ORDER BY id',
|
|
||||||
'identifiers': 'SELECT * FROM identifiers WHERE paper_id IN (' + papers + ') ORDER BY scheme,value',
|
|
||||||
'source_records': 'SELECT * FROM source_records WHERE paper_id IN (' + papers + ') ORDER BY id',
|
|
||||||
'submissions': 'SELECT * FROM submissions WHERE paper_id IN (' + papers + ') ORDER BY id',
|
|
||||||
'decisions': 'SELECT * FROM decisions WHERE submission_id IN (' + subs + ') ORDER BY id',
|
|
||||||
'submission_sources': 'SELECT * FROM submission_sources WHERE submission_id IN (' + subs + ') ORDER BY submission_id,source_record_id,source_submission_id',
|
|
||||||
'submission_identifiers': 'SELECT * FROM submission_identifiers WHERE submission_id IN (' + subs + ') ORDER BY edition_id,scheme,value',
|
|
||||||
'decision_sources': 'SELECT * FROM decision_sources WHERE decision_id IN (' + decisions + ') ORDER BY decision_id,source_record_id,source_decision_id',
|
|
||||||
'paper_tags': 'SELECT * FROM paper_tags WHERE paper_id IN (' + papers + ') ORDER BY paper_id,tag_id,evidence_key',
|
|
||||||
}
|
|
||||||
from paper_atlas.curation.annotations import available
|
|
||||||
if available(c):
|
|
||||||
queries['annotation_events'] = 'SELECT * FROM annotation_events WHERE paper_id IN (' + papers + ') ORDER BY id'
|
|
||||||
packet = {k: bounded(c, q, pids) for k, q in queries.items()}
|
|
||||||
coverage = bounded(c, '''SELECT es.* FROM edition_sources es JOIN editions e ON e.id=es.edition_id
|
|
||||||
WHERE es.source_id=? AND e.venue_id=? AND e.year=? AND e.event_key=? ORDER BY es.edition_id,es.list_url''', (sid, *scope_params))
|
|
||||||
if not coverage or any(r[2] != inventory['list_url'] for r in coverage):
|
|
||||||
raise ValueError('Unambiguous source listing required')
|
|
||||||
if len({r[0] for r in coverage}) != len(coverage):
|
|
||||||
raise ValueError('One source coverage entry per edition required')
|
|
||||||
heads = [[r[i] for i in range(len(r))] for r in versions.heads(c, sid)]
|
|
||||||
own_heads = [r for r in heads if r[1:] == [vid, inventory['year'], inventory['event_key'], inventory['list_url']]]
|
|
||||||
if len(own_heads) > 1:
|
|
||||||
raise ValueError('Forked source revision history')
|
|
||||||
parent = own_heads[0][0] if own_heads else None
|
|
||||||
return {'source_id': sid, 'venue_id': vid, 'editions': editions, 'coverage': coverage,
|
|
||||||
'packet': packet, 'own_records': own_records, 'revision_heads': heads,
|
|
||||||
'parent_sha256': parent, 'by_track': by_track,
|
|
||||||
'venue_names': sorted({inventory['venue']} | {r[0] for r in c.execute('SELECT alias FROM venue_aliases WHERE venue_id=?', (vid,))})}
|
|
||||||
|
|
||||||
|
|
||||||
def changes(c, inventory, before):
|
|
||||||
previous, current = validate(inventory)
|
|
||||||
packet = before['packet']; result = []; sid = before['source_id']
|
|
||||||
editions = {r[0]: r for r in before['editions']}
|
|
||||||
for raw in before['own_records']:
|
|
||||||
ext = raw[2]; old = previous[ext]; new = current[ext]
|
|
||||||
metadata = json.loads(raw[8])
|
|
||||||
candidates = [old, new]
|
|
||||||
if not any(raw[4] == r['url'] and raw[5] == r['title'] and raw[6] == r.get('abstract')
|
|
||||||
and json.loads(raw[7]) == r['authors'] and versions.subset(r['metadata'], metadata) for r in candidates):
|
|
||||||
raise ValueError('Current raw source differs from both reviewed snapshots')
|
|
||||||
if metadata.get('record_identifiers', {}).get('doi') != ext:
|
|
||||||
raise ValueError('Current stable DOI differs from source evidence')
|
|
||||||
attempts = [s for s in packet['submissions'] if s[3] == raw[0] and s[2] in editions]
|
|
||||||
if len(attempts) != 1:
|
|
||||||
raise ValueError('Exactly one publication attempt per source record required')
|
|
||||||
sub = attempts[0]; origin = editions[sub[2]]
|
|
||||||
if origin[3] not in {old['track'], new['track']} or sub[4] != ext or sub[1] != raw[3]:
|
|
||||||
raise ValueError('Current source attempt identity/scope differs')
|
|
||||||
if [x for x in packet['submission_sources'] if x[0] == sub[0]] != [[sub[0], raw[0], ext]]:
|
|
||||||
raise ValueError('Shared-source attempt requires separate reconciliation')
|
|
||||||
decisions = [d for d in packet['decisions'] if d[1] == sub[0]]
|
|
||||||
if not decisions or any(d[2] != 'accepted' for d in decisions):
|
|
||||||
raise ValueError('Only accepted publication observations can be revised')
|
|
||||||
target = before['by_track'].get(new['track'])
|
|
||||||
if target is None:
|
|
||||||
raise ValueError('Reviewed target edition must already exist')
|
|
||||||
identities = [x for x in packet['submission_identifiers'] if x[3] == sub[0]]
|
|
||||||
if not identities or any(x[0] != sub[2] for x in identities):
|
|
||||||
raise ValueError('Inconsistent scoped submission identifiers')
|
|
||||||
for _, scheme, value, _ in identities:
|
|
||||||
conflict = c.execute('SELECT submission_id FROM submission_identifiers WHERE edition_id=? AND scheme=? AND value=?', (target, scheme, value)).fetchone()
|
|
||||||
if conflict and conflict[0] != sub[0]:
|
|
||||||
raise ValueError('Target submission identifier collision')
|
|
||||||
replacement = copy.deepcopy(metadata); replacement.update(new['metadata'])
|
|
||||||
if sub[2] != target and KEY in metadata:
|
|
||||||
raise ValueError('An already curated attempt needs explicit scope-chain reconciliation')
|
|
||||||
if sub[2] != target or raw[6] != new.get('abstract') or replacement != metadata:
|
|
||||||
result.append({'source_record_id': raw[0], 'paper_id': raw[3], 'submission_id': sub[0],
|
|
||||||
'external_id': ext, 'from_edition_id': sub[2], 'target_edition_id': target,
|
|
||||||
'from_track': origin[3], 'track': new['track'],
|
|
||||||
'new_abstract': new.get('abstract'), 'metadata': replacement})
|
|
||||||
old_expected = Counter(r['track'] for r in previous.values()); new_expected = Counter(r['track'] for r in current.values())
|
|
||||||
observed = Counter()
|
|
||||||
own_ids = {r[0] for r in before['own_records']}
|
|
||||||
for s in packet['submissions']:
|
|
||||||
if s[3] in own_ids and s[2] in editions:
|
|
||||||
observed[s[2]] += 1
|
|
||||||
coverage = []
|
|
||||||
for raw in before['coverage']:
|
|
||||||
track = editions[raw[0]][3]
|
|
||||||
if raw[3] != 'partial' or raw[8] is not None or raw[5] != observed[raw[0]]:
|
|
||||||
raise ValueError('Coverage parent is not the verified partial source observation count')
|
|
||||||
if raw[4] not in {old_expected[track], new_expected[track]}:
|
|
||||||
raise ValueError('Coverage expected count differs from both source versions')
|
|
||||||
count = raw[5] + sum((r['target_edition_id'] == raw[0]) - (r['from_edition_id'] == raw[0]) for r in result)
|
|
||||||
if count < 0 or count > new_expected[track]:
|
|
||||||
raise ValueError('Corrected coverage exceeds current source membership')
|
|
||||||
coverage.append({'before': raw, 'expected_count': new_expected[track], 'imported_count': count})
|
|
||||||
covered = {r['before'][0] for r in coverage}
|
|
||||||
if any(r['target_edition_id'] not in covered for r in result):
|
|
||||||
raise ValueError('Target source coverage must already exist')
|
|
||||||
return result, coverage
|
|
||||||
|
|
||||||
|
|
||||||
def proposal(c, inventory, identity):
|
|
||||||
before = capture(c, inventory)
|
|
||||||
if before['parent_sha256']:
|
|
||||||
saved = json.loads(c.execute('SELECT records_json FROM source_scope_revisions WHERE version_sha256=?', (before['parent_sha256'],)).fetchone()[0])
|
|
||||||
if sorted(saved, key=lambda r: r['external_id']) != sorted(inventory['previous_rows'], key=lambda r: r['external_id']):
|
|
||||||
raise ValueError('Previous snapshot differs from active source version')
|
|
||||||
records, coverage = changes(c, inventory, before)
|
|
||||||
return {'format': 1, 'database': identity, 'inventory': inventory, 'before': before,
|
|
||||||
'changes': records, 'coverage': coverage}
|
|
||||||
|
|
||||||
|
|
||||||
def verify(c, plan, signature):
|
|
||||||
after = capture(c, plan['inventory']); expected = copy.deepcopy(plan['before']['packet'])
|
|
||||||
for change in plan['changes']:
|
|
||||||
for row in expected['submissions']:
|
|
||||||
if row[0] == change['submission_id']:
|
|
||||||
row[2] = change['target_edition_id']
|
|
||||||
for row in expected['submission_identifiers']:
|
|
||||||
if row[3] == change['submission_id']:
|
|
||||||
row[0] = change['target_edition_id']
|
|
||||||
for row in expected['source_records']:
|
|
||||||
if row[0] == change['source_record_id']:
|
|
||||||
row[6] = change['new_abstract']
|
|
||||||
row[8] = encoded_metadata(plan, change, signature)
|
|
||||||
expected['submission_identifiers'].sort(key=lambda r: (r[0], r[1], r[2]))
|
|
||||||
if after['packet'] != expected:
|
|
||||||
raise ValueError('Protected profiles/history/tags/identifiers differ; roll back')
|
|
||||||
if after['editions'] != plan['before']['editions']:
|
|
||||||
raise ValueError('Edition identities changed')
|
|
||||||
expected_heads = [r for r in plan['before']['revision_heads'] if r[0] != plan['before']['parent_sha256']]
|
|
||||||
inventory = plan['inventory']
|
|
||||||
expected_heads.append([signature, plan['before']['venue_id'], inventory['year'], inventory['event_key'], inventory['list_url']])
|
|
||||||
if after['revision_heads'] != sorted(expected_heads):
|
|
||||||
raise ValueError('Active source version differs')
|
|
||||||
expected_coverage = []
|
|
||||||
for item in plan['coverage']:
|
|
||||||
row = item['before'][:]; row[4] = item['expected_count']; row[5] = item['imported_count']
|
|
||||||
expected_coverage.append(row)
|
|
||||||
if after['coverage'] != expected_coverage:
|
|
||||||
raise ValueError('Reconciled coverage differs; roll back')
|
|
||||||
stored = c.execute('SELECT records_json,plan_json,parent_sha256 FROM source_scope_revisions WHERE version_sha256=?', (signature,)).fetchone()
|
|
||||||
if (not stored or json.loads(stored[0]) != plan['inventory']['rows']
|
|
||||||
or json.loads(stored[1]) != plan or stored[2] != plan['before']['parent_sha256']):
|
|
||||||
raise ValueError('Versioned source or complete before-images changed')
|
|
||||||
return {'records_changed': len(plan['changes']), 'events_moved': sum(r['from_edition_id'] != r['target_edition_id'] for r in plan['changes']),
|
|
||||||
'protected_papers': len(expected['papers']), 'protected_decisions': len(expected['decisions'])}
|
|
||||||
|
|
||||||
|
|
||||||
def encoded_metadata(plan, change, signature):
|
|
||||||
metadata = copy.deepcopy(change['metadata'])
|
|
||||||
if change['from_edition_id'] != change['target_edition_id']:
|
|
||||||
inventory = plan['inventory']
|
|
||||||
evidence = inventory['evidence'][0]
|
|
||||||
row = {**change, 'source': inventory['source'], 'doi': change['external_id'],
|
|
||||||
'venue_id': plan['before']['venue_id'], 'year': inventory['year'],
|
|
||||||
'event_key': inventory['event_key'], 'venue_names': plan['before']['venue_names'],
|
|
||||||
'section': change['track'], 'evidence_url': evidence['url'], 'evidence_sha256': evidence['sha256']}
|
|
||||||
metadata[KEY] = [curation(row, change['target_edition_id'], signature)]
|
|
||||||
return json.dumps(metadata, ensure_ascii=False)
|
|
||||||
|
|
||||||
|
|
||||||
def apply_plan(c, plan):
|
|
||||||
if plan.get('format') != 1:
|
|
||||||
raise ValueError('Unsupported source revision plan')
|
|
||||||
if not getattr(c, 'postgres', False) and not c.in_transaction:
|
|
||||||
c.execute('BEGIN IMMEDIATE')
|
|
||||||
signature = digest(plan)
|
|
||||||
if versions.available(c) and c.execute('SELECT 1 FROM source_scope_revisions WHERE version_sha256=?', (signature,)).fetchone():
|
|
||||||
return {'status': 'already_applied', 'version_sha256': signature, **verify(c, plan, signature)}
|
|
||||||
if getattr(c, 'postgres', False):
|
|
||||||
c.execute('LOCK TABLE sources,papers,identifiers,source_records,venues,venue_aliases,editions,edition_sources,submissions,decisions,submission_sources,submission_identifiers,decision_sources,tags,paper_tags IN SHARE ROW EXCLUSIVE MODE')
|
|
||||||
if proposal(c, plan['inventory'], plan['database']) != plan:
|
|
||||||
raise ValueError('Stale source revision plan; create and review a fresh plan')
|
|
||||||
versions.initialize(c)
|
|
||||||
for row in plan['changes']:
|
|
||||||
if row['from_edition_id'] != row['target_edition_id']:
|
|
||||||
c.execute('UPDATE submission_identifiers SET edition_id=? WHERE submission_id=?', (row['target_edition_id'], row['submission_id']))
|
|
||||||
c.execute('UPDATE submissions SET edition_id=? WHERE id=? AND edition_id=?', (row['target_edition_id'], row['submission_id'], row['from_edition_id']))
|
|
||||||
c.execute('UPDATE source_records SET abstract=?,metadata_json=? WHERE id=?',
|
|
||||||
(row['new_abstract'], encoded_metadata(plan, row, signature), row['source_record_id']))
|
|
||||||
for row in plan['coverage']:
|
|
||||||
before = row['before']
|
|
||||||
c.execute('UPDATE edition_sources SET expected_count=?,imported_count=? WHERE edition_id=? AND source_id=? AND list_url=?',
|
|
||||||
(row['expected_count'], row['imported_count'], before[0], before[1], before[2]))
|
|
||||||
inventory = plan['inventory']
|
|
||||||
c.execute('''INSERT INTO source_scope_revisions(version_sha256,source_id,venue_id,year,event_key,list_url,parent_sha256,records_json,plan_json)
|
|
||||||
VALUES(?,?,?,?,?,?,?,?,?)''', (signature, plan['before']['source_id'], plan['before']['venue_id'], inventory['year'], inventory['event_key'],
|
|
||||||
inventory['list_url'], plan['before']['parent_sha256'], json.dumps(inventory['rows'], ensure_ascii=False), json.dumps(plan, ensure_ascii=False)))
|
|
||||||
return {'status': 'applied', 'version_sha256': signature, **verify(c, plan, signature)}
|
|
||||||
|
|
||||||
|
|
||||||
def run(db, out, *, pg_service=None, pg_schema=None, inventory=None, plan=None, plan_sha256=None):
|
|
||||||
out = Path(out)
|
|
||||||
if out.exists() or not out.parent.is_dir():
|
|
||||||
raise ValueError('New output in existing directory required')
|
|
||||||
if bool(inventory) == bool(plan) or bool(plan) != bool(plan_sha256):
|
|
||||||
raise ValueError('Choose inventory preflight or plan plus exact file hash')
|
|
||||||
if inventory:
|
|
||||||
with closing(open_read_connection(db, pg_service=pg_service, pg_schema=pg_schema)) as c:
|
|
||||||
result = proposal(c, json.loads(Path(inventory).read_bytes()), database_identity(c, db))
|
|
||||||
else:
|
|
||||||
raw = Path(plan).read_bytes()
|
|
||||||
if hashlib.sha256(raw).hexdigest() != plan_sha256:
|
|
||||||
raise ValueError('Plan file hash differs')
|
|
||||||
proposed = json.loads(raw)
|
|
||||||
with closing(open_store(db, pg_service=pg_service, pg_schema=pg_schema).conn) as c:
|
|
||||||
try:
|
|
||||||
if proposed['database'] != database_identity(c, db):
|
|
||||||
raise ValueError('Plan belongs to another database')
|
|
||||||
if not getattr(c, 'postgres', False):
|
|
||||||
c.commit(); c.execute('BEGIN IMMEDIATE')
|
|
||||||
result = apply_plan(c, proposed); c.commit()
|
|
||||||
except BaseException:
|
|
||||||
c.rollback(); raise
|
|
||||||
out.write_text(json.dumps(result, ensure_ascii=False, indent=2) + '\n')
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None, *, prog=None):
|
|
||||||
parser = argparse.ArgumentParser(prog=prog); database_arguments(parser)
|
|
||||||
parser.add_argument('--inventory'); parser.add_argument('--plan'); parser.add_argument('--plan-sha256'); parser.add_argument('--out', required=True)
|
|
||||||
args = parser.parse_args(argv)
|
|
||||||
report = run(args.db, args.out, pg_service=args.pg_service, pg_schema=args.pg_schema,
|
|
||||||
inventory=args.inventory, plan=args.plan, plan_sha256=args.plan_sha256)
|
|
||||||
print(json.dumps({'status': report.get('status', 'planned'), 'out': args.out}), flush=True)
|
|
||||||
@@ -8,7 +8,7 @@ TABLES = ('sources', 'papers', 'identifiers', 'source_records', 'venues', 'editi
|
|||||||
'submission_sources', 'submission_identifiers', 'decision_sources',
|
'submission_sources', 'submission_identifiers', 'decision_sources',
|
||||||
'submission_merges', 'decision_merges', 'schema_migrations', 'tags',
|
'submission_merges', 'decision_merges', 'schema_migrations', 'tags',
|
||||||
'paper_tags', 'match_candidates', 'paper_merges', 'ingest_runs')
|
'paper_tags', 'match_candidates', 'paper_merges', 'ingest_runs')
|
||||||
OPTIONAL_TABLES = ('annotation_events', 'source_scope_revisions')
|
OPTIONAL_TABLES = ('annotation_events',)
|
||||||
CHECKS = {
|
CHECKS = {
|
||||||
'annotation_events': ["kind IN ('tag','research_type')", 'retracts_id IS NULL OR retracts_id<>id'],
|
'annotation_events': ["kind IN ('tag','research_type')", 'retracts_id IS NULL OR retracts_id<>id'],
|
||||||
'venues': ["venue_type IN ('conference','journal','review_platform')"],
|
'venues': ["venue_type IN ('conference','journal','review_platform')"],
|
||||||
|
|||||||
@@ -1,209 +0,0 @@
|
|||||||
"""Whole-source scope correction, stale refresh, full history and atomicity on both engines."""
|
|
||||||
from contextlib import closing, redirect_stdout
|
|
||||||
import copy
|
|
||||||
import hashlib
|
|
||||||
import io
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
from pathlib import Path
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from unittest.mock import patch
|
|
||||||
import uuid
|
|
||||||
|
|
||||||
from paper_atlas.db.backend import open_store
|
|
||||||
from paper_atlas.db.edition_scopes import KEY
|
|
||||||
from paper_atlas.db.source_revisions import available, require_row, require_manifest
|
|
||||||
from paper_atlas.db.store import Store
|
|
||||||
from paper_atlas.ingestion.source_revisions import proposal, apply_plan, run, validate
|
|
||||||
from paper_atlas.ingestion.event_identity import database_identity, digest, values
|
|
||||||
from paper_atlas.ingestion.records import coverage_options, ingest_file
|
|
||||||
from paper_atlas.postgres.client import connect
|
|
||||||
from paper_atlas.postgres.migration import migrate
|
|
||||||
from paper_atlas.postgres.schema import quote
|
|
||||||
|
|
||||||
URL = 'https://example.org/volume'
|
|
||||||
|
|
||||||
|
|
||||||
def seed(path):
|
|
||||||
old = []
|
|
||||||
for i in range(4):
|
|
||||||
doi = '10.1234/publication'+str(i)
|
|
||||||
old.append({'source': 'fixture', 'venue': 'MM', 'year': 2026, 'track': 'main' if i == 0 else 'unclassified',
|
|
||||||
'external_id': doi, 'url': 'https://doi.org/'+doi, 'title': 'Distinct publication '+str(i),
|
|
||||||
'authors': ['Alice Author', 'Bob Writer'], 'abstract': None if i == 1 else 'Source abstract '+str(i),
|
|
||||||
'identifiers': {'doi': doi}, 'submission_identifiers': {'publication_doi': doi},
|
|
||||||
'status': 'accepted', 'raw_decision': 'Publication observation', 'evidence_url': URL,
|
|
||||||
'observed_at': '2026-06-01T00:00:00Z', 'tags': [{'namespace': 'fixture:track', 'value': 'main' if i == 0 else 'unclassified'}],
|
|
||||||
'metadata': {'original': i, 'program': None, 'scientific_work_identity_approved': False}})
|
|
||||||
current = copy.deepcopy(old)
|
|
||||||
current[1].update(track='main', abstract='Reviewed program abstract with NUL\x00 and SOH\x01')
|
|
||||||
current[1]['metadata']['program'] = {'id': 'p1', 'evidence_url': URL}
|
|
||||||
current[1]['tags'][0]['value'] = 'main'
|
|
||||||
store = Store(path)
|
|
||||||
for row in old[:3]:
|
|
||||||
p, r = store.record(row['source'], row['external_id'], row['url'], row['title'], row['abstract'], row['authors'], row['identifiers'], row['metadata'])
|
|
||||||
store.decision(p, r, 'MM', 2026, row['track'], row['external_id'], URL, 'accepted', row['raw_decision'],
|
|
||||||
identities=[('publication_doi', row['external_id'])], observed_at=row['observed_at'])
|
|
||||||
store.record_tags(p, r, 'fixture', row['tags'])
|
|
||||||
if row == old[1]:
|
|
||||||
store.conn.execute('UPDATE papers SET abstract=? WHERE id=?', ('Canonical preprint abstract', p))
|
|
||||||
p, other = store.record('prior', 'old-forum', 'https://example.org/prior', row['title'], 'Canonical preprint abstract', row['authors'], row['identifiers'])
|
|
||||||
store.decision(p, other, 'PRIOR', 2025, 'main', 'old-forum', 'https://example.org/prior', 'rejected', 'Reject', observed_at='2025-01-01T00:00:00Z')
|
|
||||||
# An unrelated source already has an attempt in the populated target edition.
|
|
||||||
p, r = store.record('other', 'other', 'https://example.org/other', 'Unrelated target paper', 'Other abstract', ['Someone Else'])
|
|
||||||
store.decision(p, r, 'MM', 2026, 'main', 'other', 'https://example.org/other', 'accepted', 'Published')
|
|
||||||
sid = store.conn.execute("SELECT id FROM sources WHERE name='fixture'").fetchone()[0]
|
|
||||||
for track, expected, imported in [('main', 1, 1), ('unclassified', 3, 2)]:
|
|
||||||
eid = store.conn.execute('SELECT id FROM editions WHERE year=2026 AND track=?', (track,)).fetchone()[0]
|
|
||||||
store.conn.execute("INSERT INTO edition_sources VALUES(?,?,?,'partial',?,?,?,'2026-06-01',NULL)", (eid, sid, URL, expected, imported, 'checkpoint'))
|
|
||||||
store.conn.commit(); store.conn.close()
|
|
||||||
return {'format': 1, 'source': 'fixture', 'venue': 'MM', 'year': 2026, 'event_key': '', 'list_url': URL,
|
|
||||||
'previous_rows': old, 'rows': current, 'evidence': [{'url': URL, 'sha256': 'a'*64}]}
|
|
||||||
|
|
||||||
|
|
||||||
class RevisionCases:
|
|
||||||
def store(self):
|
|
||||||
return open_store(**self.target)
|
|
||||||
|
|
||||||
def plan(self, c):
|
|
||||||
return proposal(c, self.inventory, database_identity(c, self.target.get('db')))
|
|
||||||
|
|
||||||
def test_move_to_populated_target_preserves_full_history_and_before_images(self):
|
|
||||||
with closing(self.store().conn) as c:
|
|
||||||
plan = self.plan(c); self.assertEqual(len(plan['changes']), 1)
|
|
||||||
self.assertEqual(len(plan['before']['packet']['submissions']), 5)
|
|
||||||
before_papers = plan['before']['packet']['papers']
|
|
||||||
before_tags = plan['before']['packet']['paper_tags']
|
|
||||||
result = apply_plan(c, plan); c.commit()
|
|
||||||
self.assertEqual((result['events_moved'], result['records_changed']), (1, 1))
|
|
||||||
self.assertEqual(result['protected_decisions'], 5)
|
|
||||||
self.assertEqual(values(c.execute('SELECT * FROM papers ORDER BY id')), before_papers)
|
|
||||||
self.assertEqual(values(c.execute('SELECT * FROM paper_tags ORDER BY paper_id,tag_id,evidence_key')), before_tags)
|
|
||||||
self.assertEqual(c.execute("SELECT count(*) FROM decisions WHERE status='rejected'").fetchone()[0], 1)
|
|
||||||
self.assertEqual(values(c.execute('SELECT status,expected_count,imported_count,cursor,last_success_at,last_error FROM edition_sources ORDER BY edition_id')),
|
|
||||||
[['partial', 2, 2, 'checkpoint', '2026-06-01', None], ['partial', 2, 1, 'checkpoint', '2026-06-01', None]])
|
|
||||||
saved = json.loads(c.execute('SELECT plan_json FROM source_scope_revisions').fetchone()[0])
|
|
||||||
self.assertEqual(saved, plan)
|
|
||||||
self.assertEqual(apply_plan(c, plan)['status'], 'already_applied')
|
|
||||||
self.assertEqual(c.execute('SELECT count(*) FROM source_scope_revisions').fetchone()[0], 1)
|
|
||||||
|
|
||||||
def test_stale_direct_store_and_jsonl_refresh_fail_but_current_refresh_succeeds(self):
|
|
||||||
with closing(self.store().conn) as c:
|
|
||||||
plan = self.plan(c); apply_plan(c, plan); c.commit()
|
|
||||||
with closing(self.store().conn) as c:
|
|
||||||
store = object.__new__(Store); store.conn = c; store.event_identity = True
|
|
||||||
old = self.inventory['previous_rows'][1]
|
|
||||||
with self.assertRaisesRegex(ValueError, 'Stale source'):
|
|
||||||
store.record('fixture', old['external_id'], old['url'], old['title'], old['abstract'], old['authors'], old['identifiers'], old['metadata'])
|
|
||||||
c.rollback()
|
|
||||||
row = self.inventory['rows'][1]
|
|
||||||
p, r = store.record('fixture', row['external_id'], row['url'], row['title'], row['abstract'], row['authors'], row['identifiers'], row['metadata'])
|
|
||||||
self.assertEqual(c.execute('SELECT abstract FROM papers WHERE id=?', (p,)).fetchone()[0], 'Canonical preprint abstract')
|
|
||||||
self.assertEqual(json.loads(c.execute('SELECT metadata_json FROM source_records WHERE id=?', (r,)).fetchone()[0])[KEY][0]['target']['track'], 'main')
|
|
||||||
c.rollback()
|
|
||||||
path = self.root/'old.jsonl'; raw = (json.dumps(old)+'\n').encode(); path.write_bytes(raw)
|
|
||||||
path.with_suffix('.manifest.json').write_text(json.dumps({'source': 'fixture', 'venue': 'MM', 'year': 2026,
|
|
||||||
'track': 'unclassified', 'list_url': URL, 'status': 'partial', 'count': 1, 'file_sha256': hashlib.sha256(raw).hexdigest()}))
|
|
||||||
with self.assertRaisesRegex(ValueError, 'source snapshot'):
|
|
||||||
ingest_file(self.target.get('db'), path, **{k: v for k, v in self.target.items() if k != 'db'})
|
|
||||||
|
|
||||||
def test_unregistered_member_and_changed_program_metadata_are_rejected(self):
|
|
||||||
with closing(self.store().conn) as c:
|
|
||||||
apply_plan(c, self.plan(c)); c.commit()
|
|
||||||
sid = c.execute("SELECT id FROM sources WHERE name='fixture'").fetchone()[0]
|
|
||||||
cache = {}; unknown = copy.deepcopy(self.inventory['rows'][0]); unknown['external_id'] = '10.1234/unlisted'
|
|
||||||
with self.assertRaisesRegex(ValueError, 'absent from current'):
|
|
||||||
require_row(c, sid, unknown, {'list_url': URL}, cache)
|
|
||||||
bad = copy.deepcopy(self.inventory['rows'][1]); bad['metadata']['program'] = None
|
|
||||||
with self.assertRaisesRegex(ValueError, 'Stale source'):
|
|
||||||
require_row(c, sid, bad, {'list_url': URL}, cache)
|
|
||||||
good = copy.deepcopy(self.inventory['rows'][1]); good['metadata']['verified_work_anchor'] = {'reason': 'extra evidence'}
|
|
||||||
require_row(c, sid, good, {'list_url': URL}, cache)
|
|
||||||
with self.assertRaisesRegex(ValueError, 'coverage manifest'):
|
|
||||||
require_manifest(c, sid, {'venue': 'MM', 'year': 2026, 'track': 'main', 'list_url': URL, 'count': 1, 'status': 'partial'}, cache)
|
|
||||||
require_manifest(c, sid, {'venue': 'MM', 'year': 2026, 'track': 'main', 'list_url': URL, 'count': 2, 'status': 'partial'}, cache)
|
|
||||||
|
|
||||||
def test_stale_plan_and_target_identifier_collision_prevent_writes(self):
|
|
||||||
with closing(self.store().conn) as c:
|
|
||||||
plan = self.plan(c)
|
|
||||||
c.execute("UPDATE decisions SET raw_value='Additional evidence' WHERE status='rejected'")
|
|
||||||
with self.assertRaisesRegex(ValueError, 'Stale'):
|
|
||||||
apply_plan(c, plan)
|
|
||||||
c.rollback()
|
|
||||||
move = plan['changes'][0]
|
|
||||||
target_sub = c.execute('SELECT id FROM submissions WHERE edition_id=? LIMIT 1', (move['target_edition_id'],)).fetchone()[0]
|
|
||||||
c.execute('INSERT INTO submission_identifiers VALUES(?,?,?,?)', (move['target_edition_id'], 'publication_doi', move['external_id'], target_sub))
|
|
||||||
with self.assertRaisesRegex(ValueError, 'collision'):
|
|
||||||
self.plan(c)
|
|
||||||
c.rollback()
|
|
||||||
self.assertFalse(available(c))
|
|
||||||
|
|
||||||
def test_failed_postcheck_rolls_back_rows_extension_and_coverage(self):
|
|
||||||
with closing(self.store().conn) as c:
|
|
||||||
plan = self.plan(c)
|
|
||||||
with patch('paper_atlas.ingestion.source_revisions.verify', side_effect=ValueError('Injected verification failure')):
|
|
||||||
with self.assertRaisesRegex(ValueError, 'Injected'):
|
|
||||||
apply_plan(c, plan)
|
|
||||||
c.rollback()
|
|
||||||
self.assertFalse(available(c))
|
|
||||||
self.assertEqual(self.plan(c), plan)
|
|
||||||
|
|
||||||
def test_cli_binds_database_and_file_hash_without_writing_during_preflight(self):
|
|
||||||
inventory = self.root/'inventory.json'; inventory.write_text(json.dumps(self.inventory))
|
|
||||||
path = self.root/'plan.json'
|
|
||||||
args = {k: v for k, v in self.target.items() if k != 'db'}
|
|
||||||
run(self.target.get('db'), path, inventory=inventory, **args)
|
|
||||||
with closing(self.store().conn) as c:
|
|
||||||
self.assertFalse(available(c))
|
|
||||||
with self.assertRaisesRegex(ValueError, 'hash'):
|
|
||||||
run(self.target.get('db'), self.root/'bad.json', plan=path, plan_sha256='0'*64, **args)
|
|
||||||
report = run(self.target.get('db'), self.root/'applied.json', plan=path, plan_sha256=hashlib.sha256(path.read_bytes()).hexdigest(), **args)
|
|
||||||
self.assertEqual(report['events_moved'], 1)
|
|
||||||
|
|
||||||
def test_membership_identity_changes_and_incremental_shrink_remain_forbidden(self):
|
|
||||||
for mutate in [lambda x: x['rows'].pop(), lambda x: x['rows'][0].update(title='Different work'),
|
|
||||||
lambda x: x['rows'][0].update(status='rejected')]:
|
|
||||||
bad = copy.deepcopy(self.inventory); mutate(bad)
|
|
||||||
with self.assertRaises(ValueError):
|
|
||||||
validate(bad)
|
|
||||||
with self.assertRaisesRegex(ValueError, 'incremental'):
|
|
||||||
coverage_options({'coverage_mode': 'incremental', 'venue': 'MM', 'year': 2026, 'track': 'unclassified',
|
|
||||||
'status': 'partial', 'count': 1, 'coverage_previous_count': 2,
|
|
||||||
'coverage_previous_expected_count': 3, 'coverage_expected_count': 2})
|
|
||||||
|
|
||||||
|
|
||||||
class SQLiteSourceRevisionTests(RevisionCases, unittest.TestCase):
|
|
||||||
def setUp(self):
|
|
||||||
temp = tempfile.TemporaryDirectory(); self.addCleanup(temp.cleanup); self.root = Path(temp.name)
|
|
||||||
db = self.root/'source.sqlite'; self.inventory = seed(db); self.target = {'db': db}
|
|
||||||
|
|
||||||
@unittest.skipUnless(os.environ.get('PAPER_ATLAS_TEST_PG_SERVICE'), 'Requires approved remote PostgreSQL service')
|
|
||||||
def test_optional_revision_table_migrates_with_full_history(self):
|
|
||||||
with closing(self.store().conn) as c:
|
|
||||||
plan = self.plan(c); apply_plan(c, plan); c.commit()
|
|
||||||
service = os.environ['PAPER_ATLAS_TEST_PG_SERVICE']; schema = 'pa_rev_copy_'+uuid.uuid4().hex[:10]
|
|
||||||
self.addCleanup(drop_schema, service, schema)
|
|
||||||
with redirect_stdout(io.StringIO()):
|
|
||||||
migrate(self.target['db'], service, self.root/'copied.json', schema=schema, min_free_gib=0)
|
|
||||||
with closing(open_store(pg_service=service, pg_schema=schema).conn) as c:
|
|
||||||
self.assertEqual(json.loads(c.execute('SELECT plan_json FROM source_scope_revisions').fetchone()[0]), plan)
|
|
||||||
self.assertEqual(c.execute('SELECT count(*) FROM source_scope_revisions').fetchone()[0], 1)
|
|
||||||
|
|
||||||
|
|
||||||
def drop_schema(service, schema):
|
|
||||||
with connect(service) as c:
|
|
||||||
for name, in c.execute('SELECT nspname FROM pg_namespace WHERE starts_with(nspname,%s)', (schema,)):
|
|
||||||
c.execute('DROP SCHEMA '+quote(name)+' CASCADE')
|
|
||||||
|
|
||||||
|
|
||||||
@unittest.skipUnless(os.environ.get('PAPER_ATLAS_TEST_PG_SERVICE'), 'Requires approved remote PostgreSQL service')
|
|
||||||
class PostgreSQLSourceRevisionTests(RevisionCases, unittest.TestCase):
|
|
||||||
def setUp(self):
|
|
||||||
temp = tempfile.TemporaryDirectory(); self.addCleanup(temp.cleanup); self.root = Path(temp.name)
|
|
||||||
db = self.root/'source.sqlite'; self.inventory = seed(db)
|
|
||||||
service = os.environ['PAPER_ATLAS_TEST_PG_SERVICE']; schema = 'pa_rev_'+uuid.uuid4().hex[:12]
|
|
||||||
self.addCleanup(drop_schema, service, schema)
|
|
||||||
with redirect_stdout(io.StringIO()):
|
|
||||||
migrate(db, service, self.root/'migration.json', schema=schema, min_free_gib=0)
|
|
||||||
self.target = {'pg_service': service, 'pg_schema': schema}
|
|
||||||
Reference in New Issue
Block a user