Keep legacy source refreshes in their verified joint editions

.gitea/workflows/check.yml

@@ -49,6 +49,7 @@ jobs:
               subprocess.run([str(venv/'bin/paper-atlas'),'--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','collect','cvf','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','postgres','--help'],cwd=outside,check=True)
+              subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','joint-editions','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','import','records','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','import','arxiv','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','backfill','--help'],cwd=outside,check=True)

README.md

@@ -86,3 +86,5 @@ paper-atlas collect colm --cache /data/paper-index/raw/http-cache \
 CI 在已授权的远程 runner 上运行测试、构建 wheel，并从源码目录外验证安装后的命令及随包资源。数据库全量去重和覆盖率是独立的数据验收步骤，不因代码测试通过而自动完成。
 
 开发通过工作分支和 PR 交付，由用户手动合并。
+
+历史联合会议的既有届次归属可通过 [`maintain joint-editions`](docs/joint-editions.md) 先只读规划，再按计划哈希执行；保留论文与投稿决定来源，不重导入。

docs/joint-editions.md

@@ -0,0 +1,67 @@
+# Repairing joint conference membership
+
+EURAI's [official history](https://www.eurai.org/ecai/history) identifies ECAI 2018
+and 2022 as joint IJCAI/ECAI conferences. The pinned DBLP volumes
+`conf/ijcai/2018` and `conf/ijcai/2022` use `IJCAI` as their booktitle and title.
+Existing records can therefore already be present under IJCAI while an ECAI query
+shows a missing year. Adding ECAI as a global alias to the ordinary IJCAI venue
+would incorrectly include all other IJCAI years.
+
+The existing canonical `ECAI+IJCAI` venue has both member aliases. This maintenance
+command changes only the venue reference of the two existing `main` edition rows.
+Edition IDs, submission IDs, decisions, source text, authors, identifiers, tags,
+source links and coverage records stay intact. It does not merge or create papers
+or submissions, resolve per-paper tracks, or turn reviews into negative decisions.
+The DBLP collector applies the joint mapping only to these two exact main volume
+keys; original booktitle/title remain in source metadata with the membership evidence.
+The shared ingestion entry point also resolves these exact pinned source volumes before
+creating an edition, so refreshing an older normalized file cannot recreate a separate
+IJCAI edition. Other sources, years, workshop volumes and additional tracks retain their existing rules.
+
+## Review a read-only plan
+
+Use a runtime containing this repair command. The configured PostgreSQL writer lock
+is used only for the apply operation; do not replace a newer ingestion runtime with
+an older main-based feature branch.
+
+```sh
+PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain joint-editions \
+  --pg-service foundry-research --pg-schema research --out joint-plan.json
+```
+
+The preflight requires the existing joint venue and exactly its two member aliases,
+one nonempty unambiguous edition per year, and matching pinned DBLP volume evidence
+for every existing submission. A colliding target edition, missing provenance or
+unexpected source scope fails closed. The plan binds current database/schema, IDs,
+counts and full fingerprints of affected papers, identifiers, source records, tags,
+submissions, decisions and their provenance, and coverage rows. Observation time is
+recorded separately from conference year. These are scoped checks, not a final global
+identity or conference completeness audit.
+
+## Apply the exact reviewed plan
+
+Compute the file's SHA256 after review and pass it explicitly:
+
+```sh
+PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain joint-editions \
+  --pg-service foundry-research --pg-schema research --plan joint-plan.json \
+  --plan-sha256 "$PLAN_SHA256" --out joint-applied.json
+```
+
+The plan hash is checked before opening a writer. Current evidence is checked again
+under the dataset writer lock; both edition updates and an audit marker commit in one
+transaction. Protected row hashes are checked again before commit. Changed evidence
+requires a fresh reviewed plan. Existing output reports are preserved; choose a new
+path for another run. SQLite samples use `--db sample.sqlite`; the frozen production
+backup remains protected.
+
+If a response is lost after commit, rerun with the same plan and a new output path.
+When protected evidence is unchanged and both editions already point to the joint
+venue, the result is `already_joint` with zero updates. It does not replay imports or
+regenerate source records. Keep the plan, apply report and independent postcheck
+alongside collection evidence. The audit marker is `joint_ijcai_ecai_2018_2022_v1`.
+
+Publication counts can differ between the DBLP volume and the current publisher
+page. This membership repair preserves all existing rows rather than deleting entries
+to force an aggregate match; section-level main/invited/other classification and
+publication completeness require separate per-paper evidence.

docs/operations.md

@@ -120,36 +120,3 @@ paper-atlas maintain pmlr-tracks --db /data/paper-index/papers.sqlite \
 检查报告后，在同一命令添加 `--apply` 并使用新的 `--out` 路径执行迁移。计划使用共享维护锁，实际迁移使用独占锁；已有 arXiv 类别修复或其他写入运行时会拒绝启动，不会停止或重启它们。一次事务内核对整个来源集合、原始题目/摘要/作者、forum 身份、已有事件及目标冲突，更新原投稿的 edition 和身份归属，回填官方轨道出处及逐轨道覆盖数，最后复查轨道与外键后提交。论文、来源、投稿、决定的 ID 及决定观察均保留，不重新创建事件、不做实体合并、不改论文正文元数据或标签。
 
 报告记录每个迁移事件的旧 edition、原身份、完整决定观察及新轨道证据。`planned` 是未执行的计划；`complete` 才表示事务已提交；`error` 表示执行失败。若进程在提交前后中断而报告停在 `prepared`，先对照主库和报告核查，不根据日志缺失自行重启任务。新路径重跑会重新核对当前数据库，已正确的轨道不会再次迁移。共享多个来源、重复目标事件或冲突决定会停止迁移，需另行核对，不能直接删掉其中一条。
-
-## 追加同一目录的部分记录
-
-默认 `import records` 把 manifest 当成一份来源快照，覆盖统计使用该文件的数量。
-若目录已有2条，之后仅补入1条，不能把这一小批当成完整快照，否则会把目录计数改成1。
-为已核验的部分补充批次，在 manifest 显式声明：
-
-```json
-{
-  "coverage_mode": "incremental",
-  "coverage_previous_count": 2,
-  "coverage_previous_expected_count": 2,
-  "coverage_expected_count": 4,
-  "status": "partial",
-  "count": 1
-}
-```
-
-这只是新增字段示例，manifest仍须保留source、venue、year、track、list_url、文件哈希等来源证据。
-`count`是本文件行数，`coverage_previous_count`是导入前该来源/届次的实际记录数，
-`coverage_previous_expected_count`是旧覆盖行的expected_count（未知时用null），
-`coverage_expected_count`是独立目录证据支持的整份来源预期数，不能小于已知旧预期数。
-这个例子成功后覆盖统计为预期4、已导入3；文件checkpoint的records_written仍为1。
-刷新相同记录不重复累加。初次导入部分来源可声明previous_count为0、previous_expected_count为null。
-
-只支持当前最多一个规范目录URL且旧覆盖状态为partial的来源/届次；多目录、旧状态complete、
-旧计数与实际来源记录不符，或前置计数已变化，都会中止，先核对来源范围或重新取得完整快照。
-该模式在写入前检查父状态，并在同一文件事务中按实际来源记录重新计数、检查总量界限和乐观更新覆盖行。
-失败会回滚本文件的论文、来源、决定、标签和覆盖变化，保留失败运行记录；更早已提交文件不重放。
-增量模式始终保留partial，不能靠一个小批次推断完整论文集、摘要或公开拒稿覆盖。
-
-仍需先独立核对新增记录身份与决定，按原生单写入保护串行导入，再独立核验数据与原有历史。
-这些计数检查不证明外部目录完整，也不代替最终全库去重。

docs/usage.md

@@ -99,8 +99,6 @@ paper-atlas query --pg-service foundry-research \
 
 `--stats` 中 papers 是当前规则下的论文实体数；source_records 是来源记录数；decisions 是决定观察数。一个投稿可能有多个来源和多条观察，不能相加当作论文数。
 
-已有稳定标识或原来源记录匹配时，该身份优先；同标题、同作者的另一 arXiv 存档或扩展稿不会仅因相似标题被并入。没有身份锚点时，作者佐证的精确标题只有一个候选才可匹配；多个候选会报 `Ambiguous paper identity`，整个采集文件回滚并记录失败运行。应补有证据的标识或先核查版本关系，不删除重复候选或任取第一个。一个新来源明确同时携带两个既有稳定标识时，仍可按既有规则归并并保留投稿、标签和合并日志；未归并的同标题关系继续保留为待核查候选。这些规则不证明全库去重完成。
-
 ## 5. 投稿历史关系审计
 
 ```sh

paper_atlas/cli.py

@@ -25,6 +25,7 @@ COMMANDS = {
     ('catalogue','dblp'): ('paper_atlas.collectors.dblp', {}, '盘点 DBLP 历史会议目录'),
     ('import','arxiv'): ('paper_atlas.ingestion.arxiv', {}, '导入 arXiv 元数据 ZIP'),
     ('import','records'): ('paper_atlas.ingestion.records', {}, '导入核验过的会议文件'),
+    ('maintain','joint-editions'): ('paper_atlas.ingestion.joint_editions', {}, '按已核验计划修正既有联合会议届次归属'),
     ('maintain','merge'): ('paper_atlas.ingestion.maintenance', {}, '协调单写入合并与恢复'),
     ('maintain','backfill'): ('paper_atlas.ingestion.backfill', {}, '回填既有投稿的来源身份'),
     ('maintain','arxiv-tags'): ('paper_atlas.ingestion.arxiv_tags', {}, '恢复逐来源记录的 arXiv 类别证据'),

paper_atlas/collectors/dblp_records.py

@@ -88,6 +88,8 @@ def normalize_row(row,volumes,targets,include_workshops=False,through_year=2026)
               'scope_rule':edition['rule'],'author_names_raw':fields.get('author',[]),'publtype':row.get('publtype'),
               'bibliography':{key:value for key,value in fields.items() if key in ('booktitle','year','ee','url','pages','volume','number','publisher','series','isbn')},
               'ambiguous_identifiers':ambiguous,'publication_evidence':'inproceedings bibliography; not an original decision note'}
+    if edition.get('joint_edition_evidence'):
+        metadata['joint_edition_evidence'] = edition['joint_edition_evidence']
     if edition.get('track_scope_note'):
         metadata['track_scope_note']=edition['track_scope_note']
     flagged=bool(re.search(r'\b(?:withdrawn|retracted)\b',row.get('publtype') or '',re.I))

paper_atlas/collectors/dblp_scope.py

@@ -96,8 +96,15 @@ def scope(row,volume,through_year=2026):
     elif suffix.casefold() in ('poster','posters') and venue not in ('ICLR','ACL','EMNLP','NAACL','EACL','COLING','CoNLL'):track='poster'
     if venue=='AAAI+IAAI':track='proceedings_unspecified'
     if venue=='AAMAS' and year<2002:return None,'predecessor_not_aamas'
+    joint_evidence = None
+    if (family == 'ijcai' and base == 'IJCAI' and track == 'main'
+            and year in (2018, 2022) and volume['key'] == 'conf/ijcai/' + str(year)):
+        venue, aliases = 'ECAI+IJCAI', ['ECAI', 'IJCAI']
+        joint_evidence = 'https://www.eurai.org/ecai/history'
     result = {'venue':venue,'venue_aliases':aliases,'year':year,'track':track,'bibliographic_year':int(bibliographic[0]),
               'rule':family+':'+base,'proceedings_title':title,'edition_year_basis':year_basis}
+    if joint_evidence:
+        result['joint_edition_evidence'] = joint_evidence
     if venue=='MM' and track=='main':
         # A main MM volume also includes doctoral, tutorials, companion papers
         # and other sections. DBLP XML omits the per-paper TOC section.

paper_atlas/db/store.py

@@ -95,19 +95,13 @@ class Store:
             matches.add(old[0])
         key = normalized(title)
         title_matches = c.execute('SELECT id,authors_json,abstract FROM papers WHERE title_key=?',(key,)).fetchall()
-        # An existing identifier/source record anchors the entity. Title
-        # similarity must not pull in another deposit or an extended version.
-        if not matches:
-            title_only_matches = set()
-            authors_set = author_keys(authors)
-            for paper,old_authors,old_abstract in title_matches:
-                overlap = authors_set & author_keys(json.loads(old_authors))
-                same_source = c.execute('SELECT 1 FROM source_records WHERE paper_id=? AND source_id=? LIMIT 1',(paper,source_id)).fetchone()
-                if not same_source and len(key)>=25 and (len(overlap)>=2 or (overlap and abstract and abstract.strip()==(old_abstract or '').strip())):
-                    title_only_matches.add(paper)
-            if len(title_only_matches)>1:
-                raise ValueError('Ambiguous paper identity: multiple author-corroborated title matches; provide verified identifiers')
-            matches = title_only_matches
+        authors_set = author_keys(authors)
+        for paper,old_authors,old_abstract in title_matches:
+            overlap = authors_set & author_keys(json.loads(old_authors))
+            # Cross-platform title matching needs independent author corroboration.
+            same_source = c.execute('SELECT 1 FROM source_records WHERE paper_id=? AND source_id=? LIMIT 1',(paper,source_id)).fetchone()
+            if not same_source and len(key)>=25 and (len(overlap)>=2 or (overlap and abstract and abstract.strip()==(old_abstract or '').strip())):
+                matches.add(paper)
         if matches:
             paper = min(matches)
             for other in sorted(matches-{paper}):
@@ -150,6 +144,20 @@ class Store:
 
     def decision(self,paper,record,venue,year,track,external_id,url,status,raw_value,aliases=None,identities=None,observed_at=None,decision_date=None,source_decision_id='',submitted_at=None,guard_status=False):
         c = self.conn
+        # An already collected file may still call these exact pinned volumes
+        # IJCAI. Resolve membership before finding/creating an edition so a
+        # later refresh cannot split the repaired joint event again.
+        if venue == 'IJCAI' and year in (2018,2022) and track == 'main':
+            from paper_atlas.collectors.dblp import XML_SHA
+            evidence=c.execute('SELECT src.name,r.metadata_json FROM source_records r JOIN sources src ON src.id=r.source_id WHERE r.id=?',(record,)).fetchone()
+            if evidence and evidence[0] == 'dblp':
+                metadata=json.loads(evidence[1])
+                if (metadata.get('snapshot_sha256') == XML_SHA
+                        and metadata.get('proceedings_key') == 'conf/ijcai/'+str(year)
+                        and metadata.get('bibliographic_year') == year
+                        and (metadata.get('bibliography') or {}).get('booktitle') in (['IJCAI'],['IJCAI/ECAI'])):
+                    venue='ECAI+IJCAI'
+                    aliases=sorted(set(aliases or []) | {'ECAI','IJCAI'})
         existing=c.execute('SELECT id FROM venues WHERE lower(acronym)=lower(?)',(venue,)).fetchone()
         if existing:venue_id=existing[0]
         else:venue_id=inserted_id(c,"INSERT INTO venues(acronym,name,field,venue_type) VALUES(?,?,?,'conference')",(venue,venue,'unclassified'))

paper_atlas/ingestion/joint_editions.py

@@ -0,0 +1,204 @@
+"""Evidence-bound repair of two existing IJCAI/ECAI edition memberships."""
+import argparse
+from contextlib import closing
+import hashlib
+import json
+from pathlib import Path
+
+from paper_atlas.db.backend import database_arguments, open_read_connection, open_store
+from paper_atlas.db.submissions import timestamp
+from paper_atlas.collectors.dblp import XML_SHA
+
+YEARS = (2018, 2022)
+TARGET = 'ECAI+IJCAI'
+MEMBERS = {'ECAI', 'IJCAI'}
+EVIDENCE = 'https://www.eurai.org/ecai/history'
+MARKER = 'joint_ijcai_ecai_2018_2022_v1'
+
+
+def _digest(c, sql, params=()):
+    sha = hashlib.sha256(); count = 0
+    for row in c.execute(sql, params):
+        sha.update((json.dumps([row[i] for i in range(len(row))], ensure_ascii=False,
+                              separators=(',', ':')) + '\n').encode())
+        count += 1
+        if count > 100000:
+            raise ValueError('Joint repair evidence scope exceeded bounded row limit')
+    return {'rows': count, 'sha256': sha.hexdigest()}
+
+
+def fingerprints(c, ids):
+    marks = ','.join('?' for _ in ids)
+    submissions = 'SELECT id FROM submissions WHERE edition_id IN (' + marks + ')'
+    papers = 'SELECT paper_id FROM submissions WHERE edition_id IN (' + marks + ')'
+    decisions = 'SELECT id FROM decisions WHERE submission_id IN (' + submissions + ')'
+    checks = {
+        'papers': ('id IN (' + papers + ')', 'id'),
+        'identifiers': ('paper_id IN (' + papers + ')', 'scheme,value'),
+        'source_records': ('paper_id IN (' + papers + ')', 'id'),
+        'paper_tags': ('paper_id IN (' + papers + ')', 'paper_id,tag_id,evidence_key'),
+        'submissions': ('edition_id IN (' + marks + ')', 'id'),
+        'decisions': ('submission_id IN (' + submissions + ')', 'id'),
+        'submission_sources': ('submission_id IN (' + submissions + ')', 'submission_id,source_record_id,source_submission_id'),
+        'submission_identifiers': ('edition_id IN (' + marks + ')', 'edition_id,scheme,value'),
+        'decision_sources': ('decision_id IN (' + decisions + ')', 'decision_id,source_record_id,source_decision_id'),
+        'edition_sources': ('edition_id IN (' + marks + ')', 'edition_id,source_id,list_url'),
+    }
+    return {table: _digest(c, 'SELECT * FROM ' + table + ' WHERE ' + where + ' ORDER BY ' + order, ids)
+            for table, (where, order) in checks.items()}
+
+
+def inspect(c):
+    target = c.execute('SELECT id FROM venues WHERE acronym=?', (TARGET,)).fetchall()
+    if len(target) != 1:
+        raise ValueError('Exactly one existing canonical joint venue required')
+    target = target[0][0]
+    aliases = {r[0] for r in c.execute('SELECT alias FROM venue_aliases WHERE venue_id=?', (target,))}
+    if aliases != MEMBERS:
+        raise ValueError('Joint target must already have exactly ECAI and IJCAI aliases')
+    editions = []
+    for year in YEARS:
+        rows = c.execute('''SELECT e.id,e.venue_id,v.acronym FROM editions e JOIN venues v ON v.id=e.venue_id
+            WHERE v.acronym IN ('IJCAI','ECAI+IJCAI') AND e.year=? AND e.track='main' ''', (year,)).fetchall()
+        if len(rows) != 1:
+            raise ValueError('Expected one existing edition per year; missing or colliding editions require separate review')
+        edition, venue, acronym = (rows[0][i] for i in range(3))
+        count = c.execute('SELECT count(*) FROM submissions WHERE edition_id=?', (edition,)).fetchone()[0]
+        if not 0 < count <= 5000:
+            raise ValueError('Existing edition must have a nonempty bounded submission scope')
+        # Every existing submission must be supported by its own pinned DBLP
+        # volume record. No title/author matching, new attempts or text writes.
+        evidence = c.execute('''SELECT s.id,r.external_id,r.metadata_json FROM submissions s
+            JOIN submission_sources ss ON ss.submission_id=s.id
+            JOIN source_records r ON r.id=ss.source_record_id JOIN sources src ON src.id=r.source_id
+            WHERE s.edition_id=? AND src.name='dblp' ORDER BY s.id,r.id''', (edition,)).fetchall()
+        seen = set()
+        for row in evidence:
+            sid, external, metadata = (row[i] for i in range(3))
+            meta = json.loads(metadata)
+            bibliography = meta.get('bibliography') or {}
+            if (meta.get('snapshot_sha256') != XML_SHA
+                    or meta.get('proceedings_key') != 'conf/ijcai/' + str(year)
+                    or meta.get('bibliographic_year') != year
+                    or bibliography.get('booktitle') not in (['IJCAI'], ['IJCAI/ECAI'])
+                    or not external.startswith('conf/ijcai/')):
+                raise ValueError('Source evidence outside the exact verified IJCAI/ECAI volume')
+            seen.add(sid)
+        if len(seen) != count:
+            raise ValueError('Every submission must have matching original volume evidence')
+        editions.append({'id': edition, 'year': year, 'track': 'main', 'from_venue_id': venue,
+                         'from_venue': acronym, 'target_venue_id': target, 'submissions': count,
+                         'evidence_url': EVIDENCE, 'proceedings_key': 'conf/ijcai/' + str(year)})
+    ids = tuple(e['id'] for e in editions)
+    return {'format_version': 1, 'kind': MARKER, 'checked_at': timestamp(), 'editions': editions,
+            'content_fingerprints': fingerprints(c, ids), 'target_aliases': sorted(aliases),
+            'status': 'already_joint' if all(e['from_venue_id'] == target for e in editions) else 'ready',
+            'scope': 'Only these two existing edition venue references; no source text, paper, submission, decision, track, identifier or tag change.'}
+
+
+def validate_plan(expected):
+    if not isinstance(expected, dict):
+        raise ValueError('Repair plan must be a JSON object')
+    if set(expected) != {'format_version','kind','checked_at','editions','content_fingerprints','target_aliases','status','scope','database'}:
+        raise ValueError('Unexpected repair plan fields')
+    if (type(expected['format_version']) is not int or expected['format_version'] != 1
+            or expected['kind'] != MARKER or expected['status'] not in ('ready','already_joint')):
+        raise ValueError('Unsupported repair plan')
+    if (not isinstance(expected['database'],dict) or not isinstance(expected['content_fingerprints'],dict)
+            or not isinstance(expected['editions'],list) or len(expected['editions']) != 2
+            or expected['target_aliases'] != sorted(MEMBERS)):
+        raise ValueError('Invalid repair plan structure')
+    for edition, year in zip(expected['editions'], YEARS):
+        if not isinstance(edition,dict) or edition.get('year') != year:
+            raise ValueError('Repair plan must name the two exact ordered years')
+        for key in ('id','year','from_venue_id','target_venue_id','submissions'):
+            if type(edition.get(key)) is not int or edition[key] <= 0:
+                raise ValueError('Repair plan IDs/counts must be positive integers')
+
+
+def apply_plan(c, expected):
+    validate_plan(expected)
+    actual = inspect(c)
+    if actual['content_fingerprints'] != expected['content_fingerprints'] or actual['target_aliases'] != expected['target_aliases']:
+        raise ValueError('Stale repair plan: protected content or aliases changed')
+    if len(expected['editions']) != 2:
+        raise ValueError('Exactly two editions required')
+    for old, now in zip(expected['editions'], actual['editions']):
+        if set(old) != set(now):
+            raise ValueError('Unexpected edition fields')
+        for key in ('id','year','track','target_venue_id','submissions','evidence_url','proceedings_key'):
+            if old[key] != now[key]:
+                raise ValueError('Stale repair plan: edition differs')
+        if now['from_venue_id'] != old['from_venue_id'] and now['from_venue_id'] != now['target_venue_id']:
+            raise ValueError('Edition membership differs from plan')
+    if actual['status'] == 'already_joint':
+        return {**actual, 'status':'already_joint', 'updated_editions':0}
+    for edition in actual['editions']:
+        if edition['from_venue_id'] == edition['target_venue_id']:
+            continue
+        changed = c.execute('''UPDATE editions SET venue_id=? WHERE id=? AND venue_id=? AND year=? AND track=?''',
+                            (edition['target_venue_id'],edition['id'],edition['from_venue_id'],edition['year'],edition['track']))
+        if changed.rowcount != 1:
+            raise ValueError('Concurrent edition change; entire repair must roll back')
+    after = inspect(c)
+    if after['status'] != 'already_joint' or after['content_fingerprints'] != actual['content_fingerprints']:
+        raise ValueError('Postcheck changed protected content; entire repair must roll back')
+    c.execute('INSERT INTO schema_migrations(name) VALUES(?) ON CONFLICT DO NOTHING', (MARKER,))
+    return {**after, 'status':'applied', 'updated_editions':sum(e['from_venue_id'] != e['target_venue_id'] for e in actual['editions'])}
+
+
+def database_identity(c, db):
+    if getattr(c, 'postgres', False):
+        row = c.execute('SELECT current_database(),current_schema()').fetchone()
+        return {'engine':'postgresql','database':row[0],'schema':row[1]}
+    return {'engine':'sqlite','path':str(Path(db).resolve())}
+
+
+def run(db, out, *, pg_service=None, pg_schema=None, plan=None, plan_sha256=None):
+    out = Path(out).resolve()
+    if not out.parent.is_dir():
+        raise ValueError('Report parent directory must already exist')
+    if out.exists():
+        raise ValueError('Preserve existing evidence report; choose a new output path')
+    expected = None
+    if plan:
+        raw = Path(plan).read_bytes()
+        if hashlib.sha256(raw).hexdigest() != plan_sha256:
+            raise ValueError('Repair plan hash differs before opening database')
+        expected = json.loads(raw)
+        validate_plan(expected)
+    elif plan_sha256:
+        raise ValueError('Plan hash requires an existing plan')
+    if db and out == Path(db).resolve():
+        raise ValueError('Report must not overwrite database')
+    writer = open_store(db, pg_service=pg_service, pg_schema=pg_schema) if expected is not None else None
+    c = writer.conn if writer else open_read_connection(db, pg_service=pg_service, pg_schema=pg_schema)
+    with closing(c):
+        try:
+            if expected is not None and not getattr(c, 'postgres', False):
+                c.commit(); c.execute('BEGIN IMMEDIATE')
+            identity = database_identity(c, db)
+            if expected is not None and expected['database'] != identity:
+                raise ValueError('Plan belongs to another database/schema')
+            report = apply_plan(c, expected) if expected is not None else inspect(c)
+            report['database'] = identity
+            if expected is not None:
+                report['plan_sha256'] = plan_sha256
+            if expected is not None:
+                c.commit()
+        except BaseException:
+            if expected is not None: c.rollback()
+            raise
+    out.write_text(json.dumps(report,ensure_ascii=False,indent=2)+'\n')
+    print(json.dumps({'status':report['status'],'editions':len(report['editions']),
+                      'updated_editions':report.get('updated_editions',0),'report':str(out)}))
+    return report
+
+
+def main(argv=None, *, prog=None):
+    parser = argparse.ArgumentParser(prog=prog); database_arguments(parser)
+    parser.add_argument('--out', required=True)
+    parser.add_argument('--plan', help='Apply an already reviewed plan; omitted means read-only preflight')
+    parser.add_argument('--plan-sha256', help='Required exact plan SHA256 when applying')
+    args = parser.parse_args(argv)
+    run(args.db,args.out,pg_service=args.pg_service,pg_schema=args.pg_schema,plan=args.plan,plan_sha256=args.plan_sha256)

paper_atlas/ingestion/records.py

@@ -12,55 +12,6 @@ from paper_atlas.collectors.public_openreview import SOURCE as PUBLIC_SOURCE
 from paper_atlas.collectors.icml_public_archive import SOURCE as ARCHIVE_SOURCE
 
 
-def coverage_options(manifest):
-    mode = manifest.get('coverage_mode', 'snapshot')
-    if mode not in ('snapshot', 'incremental'):
-        raise ValueError('Unsupported coverage mode')
-    if mode == 'snapshot':
-        return None
-    required = ('venue', 'year', 'track', 'count', 'coverage_previous_count',
-                'coverage_previous_expected_count', 'coverage_expected_count')
-    if any(key not in manifest for key in required) or manifest.get('status') != 'partial':
-        raise ValueError('Incremental coverage requires a partial, explicitly scoped manifest')
-    previous = manifest['coverage_previous_count']
-    expected = manifest['coverage_expected_count']
-    prior_expected = manifest['coverage_previous_expected_count']
-    if (type(previous) is not int or previous < 0 or type(expected) is not int or expected < 1
-            or type(manifest['count']) is not int or manifest['count'] < 1
-            or expected < max(previous, manifest['count'])
-            or (prior_expected is not None and (type(prior_expected) is not int or prior_expected < 0
-                                               or expected < prior_expected))):
-        raise ValueError('Invalid incremental coverage counts')
-    return previous, prior_expected, expected
-
-
-def source_record_count(c, source, edition):
-    return c.execute('''SELECT count(DISTINCT ss.source_record_id) FROM submission_sources ss
-        JOIN submissions s ON s.id=ss.submission_id
-        JOIN source_records sr ON sr.id=ss.source_record_id
-        WHERE s.edition_id=? AND sr.source_id=?''', (edition, source)).fetchone()[0]
-
-
-def require_incremental_parent(c, source, manifest, options):
-    edition = c.execute('''SELECT e.id FROM editions e JOIN venues v ON v.id=e.venue_id
-        WHERE lower(v.acronym)=lower(?) AND e.year=? AND e.track=?''',
-        (manifest['venue'], manifest['year'], manifest['track'])).fetchone()
-    entries = c.execute('''SELECT list_url,status,expected_count,imported_count
-        FROM edition_sources WHERE edition_id=? AND source_id=?''',
-        (edition[0], source)).fetchall() if edition else []
-    previous, prior_expected, _ = options
-    if entries:
-        if len(entries) != 1 or entries[0][0] != manifest['list_url']:
-            raise ValueError('Incremental coverage requires one unambiguous source listing')
-        if tuple(entries[0][1:]) != ('partial', prior_expected, previous):
-            raise ValueError('Incremental coverage parent changed or is not partial')
-    elif previous != 0 or prior_expected is not None:
-        raise ValueError('Incremental coverage parent is absent')
-    actual = source_record_count(c, source, edition[0]) if edition else 0
-    if actual != previous:
-        raise ValueError('Incremental coverage parent differs from actual source records')
-
-
 def ingest_file(db,path, *, pg_service=None, pg_schema=None):
     path = Path(path)
     manifest_path = path.with_suffix('.manifest.json')
@@ -73,7 +24,6 @@ def ingest_file(db,path, *, pg_service=None, pg_schema=None):
     source_name = manifest.get('source')
     if not source_name:
         raise ValueError('A source manifest is required')
-    incremental = coverage_options(manifest)
     store = open_store(db, pg_service=pg_service, pg_schema=pg_schema)
     c = store.conn
     run = None
@@ -85,8 +35,6 @@ def ingest_file(db,path, *, pg_service=None, pg_schema=None):
         source_id = store.source(source_name,manifest['list_url'])
         run = inserted_id(c,'INSERT INTO ingest_runs(source_id,checkpoint_json) VALUES(?,?)',(source_id,json.dumps({'file':str(path),'expected_sha256':manifest.get('file_sha256')})))
         c.commit()
-        if incremental is not None:
-            require_incremental_parent(c, source_id, manifest, incremental)
         digest = hashlib.sha256()
         with path.open('rb') as stream:
             for line in stream:
@@ -134,25 +82,8 @@ def ingest_file(db,path, *, pg_service=None, pg_schema=None):
         if expected is not None and expected != count:
             raise ValueError(f'Manifest count {expected} differs from records {count}')
         complete=expected==count and manifest.get('status') not in ('partial','blocked') and not manifest.get('missing_abstract_count',0) and not missing_abstracts
-        imported = count
-        condition = ''
-        condition_params = ()
-        if incremental is not None:
-            previous, prior_expected, expected = incremental
-            imported = source_record_count(c, source_id, edition)
-            if imported < previous or imported > expected:
-                raise ValueError('Incremental coverage exceeds the declared source bounds')
-            complete = False
-            condition = " WHERE edition_sources.imported_count=? AND COALESCE(edition_sources.expected_count,-1)=? AND edition_sources.status='partial'"
-            condition_params = (previous, -1 if prior_expected is None else prior_expected)
-        result = c.execute('INSERT INTO edition_sources(edition_id,source_id,list_url,status,expected_count,imported_count,last_success_at) VALUES(?,?,?,?,?,?,?) ON CONFLICT(edition_id,source_id,list_url) DO UPDATE SET status=excluded.status,expected_count=excluded.expected_count,imported_count=excluded.imported_count,last_success_at=excluded.last_success_at,last_error=NULL'+condition,(edition,source_id,manifest['list_url'],'complete' if complete else 'partial',expected,imported,time.strftime('%Y-%m-%dT%H:%M:%SZ',time.gmtime()),*condition_params))
-        if incremental is not None and result.rowcount != 1:
-            raise ValueError('Incremental coverage parent changed during import')
-        checkpoint = {'file':str(path),'file_sha256':digest.hexdigest()}
-        if incremental is not None:
-            checkpoint['coverage'] = {'mode':'incremental','previous_count':previous,
-                                      'expected_count':expected,'imported_count':imported}
-        c.execute("UPDATE ingest_runs SET status='complete',finished_at=?,records_seen=?,records_written=?,checkpoint_json=? WHERE id=?",(timestamp(),count,count,json.dumps(checkpoint),run))
+        c.execute('INSERT INTO edition_sources(edition_id,source_id,list_url,status,expected_count,imported_count,last_success_at) VALUES(?,?,?,?,?,?,?) ON CONFLICT(edition_id,source_id,list_url) DO UPDATE SET status=excluded.status,expected_count=excluded.expected_count,imported_count=excluded.imported_count,last_success_at=excluded.last_success_at,last_error=NULL',(edition,source_id,manifest['list_url'],'complete' if complete else 'partial',expected,count,time.strftime('%Y-%m-%dT%H:%M:%SZ',time.gmtime())))
+        c.execute("UPDATE ingest_runs SET status='complete',finished_at=?,records_seen=?,records_written=?,checkpoint_json=? WHERE id=?",(timestamp(),count,count,json.dumps({'file':str(path),'file_sha256':digest.hexdigest()}),run))
         c.commit()
         print(json.dumps({'file':str(path),'status':'imported','records':count,'missing_abstract_count':missing_abstracts}),flush=True)
     except BaseException as exc:

tests/test_incremental_coverage.py

@@ -1,116 +0,0 @@
-from contextlib import redirect_stdout
-import hashlib
-import io
-import json
-from pathlib import Path
-import sqlite3
-import tempfile
-import unittest
-
-from paper_atlas.ingestion.records import ingest_file
-
-
-class IncrementalCoverageTests(unittest.TestCase):
-    def setUp(self):
-        self.tmp = tempfile.TemporaryDirectory()
-        self.addCleanup(self.tmp.cleanup)
-        self.root = Path(self.tmp.name)
-        self.db = self.root / 'papers.sqlite'
-
-    def file(self, name, external_ids, **options):
-        rows = [{'source':'example','external_id':key,'url':'https://example.org/'+key,
-                 'venue':'Example','year':2026,'track':'main','status':'accepted',
-                 'title':'Independent research contribution '+key,'abstract':'Known abstract',
-                 'authors':[], 'identifiers':{'doi':'10.1234/'+key}} for key in external_ids]
-        path = self.root / (name+'.jsonl')
-        path.write_bytes((''.join(json.dumps(row)+'\n' for row in rows)).encode())
-        manifest = {key:rows[0][key] for key in ('source','venue','year','track')}
-        manifest.update(list_url='https://example.org/2026', count=len(rows),status='partial',
-                        file_sha256=hashlib.sha256(path.read_bytes()).hexdigest())
-        manifest.update(options)
-        path.with_suffix('.manifest.json').write_text(json.dumps(manifest))
-        return path
-
-    def ingest(self, path):
-        with redirect_stdout(io.StringIO()):
-            ingest_file(self.db,path)
-
-    def incremental(self, previous=2, prior_expected=2, expected=4):
-        return {'coverage_mode':'incremental','coverage_previous_count':previous,
-                'coverage_previous_expected_count':prior_expected,'coverage_expected_count':expected}
-
-    def coverage(self):
-        with sqlite3.connect(self.db) as c:
-            return c.execute('SELECT status,expected_count,imported_count FROM edition_sources').fetchall()
-
-    def test_append_and_refresh_count_actual_source_records_without_erasing_earlier_batch(self):
-        self.ingest(self.file('first',['a','b']))
-        self.ingest(self.file('append',['c'],**self.incremental()))
-        self.assertEqual(self.coverage(),[('partial',4,3)])
-        self.ingest(self.file('refresh',['c'],**self.incremental(3,4,4)))
-        self.assertEqual(self.coverage(),[('partial',4,3)])
-        with sqlite3.connect(self.db) as c:
-            self.assertEqual(c.execute('SELECT count(*) FROM source_records').fetchone()[0],3)
-            self.assertEqual(c.execute('SELECT count(*) FROM submissions').fetchone()[0],3)
-            runs = c.execute('SELECT records_written,checkpoint_json FROM ingest_runs ORDER BY id').fetchall()
-            self.assertEqual([r[0] for r in runs],[2,1,1])
-            self.assertEqual(json.loads(runs[-1][1])['coverage'],
-                             {'mode':'incremental','previous_count':3,'expected_count':4,'imported_count':3})
-            self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(),[])
-
-    def test_stale_parent_stops_before_source_writes(self):
-        self.ingest(self.file('first',['a','b']))
-        with self.assertRaisesRegex(ValueError,'parent changed'):
-            self.ingest(self.file('stale',['c'],**self.incremental(1,2,4)))
-        self.assertEqual(self.coverage(),[('partial',2,2)])
-        with sqlite3.connect(self.db) as c:
-            self.assertEqual(c.execute("SELECT count(*) FROM source_records WHERE external_id='c'").fetchone()[0],0)
-            self.assertEqual(c.execute('SELECT status FROM ingest_runs ORDER BY id DESC LIMIT 1').fetchone()[0],'failed')
-
-    def test_late_source_bound_failure_rolls_back_new_records_and_coverage(self):
-        self.ingest(self.file('first',['a','b']))
-        with self.assertRaisesRegex(ValueError,'source bounds'):
-            self.ingest(self.file('overflow',['c'],**self.incremental(2,2,2)))
-        self.assertEqual(self.coverage(),[('partial',2,2)])
-        with sqlite3.connect(self.db) as c:
-            for table in ('papers','source_records','submissions','decisions'):
-                self.assertEqual(c.execute('SELECT count(*) FROM '+table).fetchone()[0],2)
-
-    def test_multiple_listings_cannot_be_silently_aggregated(self):
-        self.ingest(self.file('one',['a']))
-        self.ingest(self.file('two',['b'],list_url='https://example.org/another-list'))
-        before=self.coverage()
-        with self.assertRaisesRegex(ValueError,'unambiguous source listing'):
-            self.ingest(self.file('ambiguous',['c'],**self.incremental(2,1,4)))
-        self.assertEqual(self.coverage(),before)
-
-    def test_untracked_source_rows_require_reconciliation(self):
-        self.ingest(self.file('first',['a','b']))
-        with sqlite3.connect(self.db) as c:
-            c.execute('UPDATE edition_sources SET imported_count=1')
-        with self.assertRaisesRegex(ValueError,'actual source records'):
-            self.ingest(self.file('stale-summary',['c'],**self.incremental(1,2,4)))
-        self.assertEqual(self.coverage(),[('partial',2,1)])
-
-    def test_first_partial_chunk_can_declare_an_independently_known_source_total(self):
-        self.ingest(self.file('first',['a'],**self.incremental(0,None,4)))
-        self.assertEqual(self.coverage(),[('partial',4,1)])
-
-    def test_complete_listing_is_not_demoted_by_a_partial_chunk(self):
-        self.ingest(self.file('complete',['a','b'],status='complete'))
-        with self.assertRaisesRegex(ValueError,'not partial'):
-            self.ingest(self.file('append',['c'],**self.incremental()))
-        self.assertEqual(self.coverage(),[('complete',2,2)])
-
-    def test_invalid_mode_counts_and_missing_scope_fail_before_database_creation(self):
-        for options in ({'coverage_mode':'unknown'}, self.incremental(True,2,4),
-                        self.incremental(2,5,4),self.incremental(-1,2,4),
-                        {'coverage_mode':'incremental'},
-                        {**self.incremental(),'status':'complete'}):
-            with self.subTest(options=options):
-                with self.assertRaises(ValueError):self.ingest(self.file('invalid',['a'],**options))
-                self.assertFalse(self.db.exists())
-
-
-if __name__=='__main__':
-    unittest.main()

tests/test_index_core.py

@@ -53,50 +53,6 @@ class StoreTests(unittest.TestCase):
         self.assertNotEqual(other,paper)
         self.assertEqual(self.store.conn.execute('SELECT count(*) FROM match_candidates').fetchone()[0],1)
 
-    def test_identifier_anchor_does_not_absorb_a_same_title_arxiv_version(self):
-        conference,first=self.record('arxiv','1408.1485')
-        extended,second=self.record('arxiv','cs/0307069')
-        self.store.decision(extended,second,'ICLR',2025,'main','separate','https://example.org/reject','rejected','Reject')
-        resolved,_=self.record('dblp','conf/uai/example',{'arxiv':'1408.1485'})
-        self.assertEqual(resolved,conference);self.assertNotEqual(conference,extended)
-        c=self.store.conn
-        self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0],2)
-        self.assertEqual(c.execute('SELECT paper_id FROM submissions').fetchone()[0],extended)
-        self.assertEqual(c.execute('SELECT count(*) FROM paper_merges').fetchone()[0],0)
-        self.assertEqual(c.execute('SELECT count(*) FROM match_candidates').fetchone()[0],1)
-        # A refresh of the anchored original source is equally unambiguous.
-        self.assertEqual(self.record('arxiv','1408.1485')[0],conference)
-        self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0],2)
-
-    def test_ambiguous_title_aborts_whole_file_without_merging_existing_entities(self):
-        self.record('arxiv','1408.1485');self.record('arxiv','cs/0307069')
-        self.store.conn.commit()
-        rows=[{'source':'dblp','external_id':'conf/uai/new','url':'https://example.org/new',
-               'title':'An unrelated earlier record in this file','abstract':None,'authors':[],
-               'venue':'UAI','year':2001,'track':'main','status':'accepted'},
-              {'source':'dblp','external_id':'conf/uai/ambiguous','url':'https://example.org/ambiguous',
-               'title':'A sufficiently long example research paper title','abstract':None,
-               'authors':['Jane Doe','John Smith'],'identifiers':{'doi':'10.0000/not-yet-known'},
-               'venue':'UAI','year':2001,'track':'main','status':'accepted'}]
-        path=Path(self.tmp.name)/'ambiguous.jsonl'
-        path.write_text(''.join(json.dumps(row)+'\n' for row in rows))
-        path.with_suffix('.manifest.json').write_text(json.dumps({'source':'dblp','list_url':'https://example.org/list','count':2}))
-        with self.assertRaisesRegex(ValueError,'Ambiguous paper identity'):ingest_file(self.db,path)
-        c=self.store.conn
-        for table,count in [('papers',2),('source_records',2),('submissions',0),('decisions',0),('paper_merges',0)]:
-            self.assertEqual(c.execute('SELECT count(*) FROM '+table).fetchone()[0],count,table)
-        self.assertEqual(c.execute('SELECT status FROM ingest_runs').fetchone()[0],'failed')
-
-    def test_explicit_bridge_merges_only_its_identifier_matches(self):
-        first,_=self.record('arxiv','2601.00001',authors=['First Author'])
-        second,_=self.record('arxiv','2601.00002',{'doi':'10.0000/bridge'},authors=['Second Author'])
-        unrelated,_=self.record('arxiv','2601.00003')
-        resolved,_=self.record('official','bridge',{'arxiv':'2601.00001','doi':'10.0000/bridge'})
-        self.assertEqual(resolved,min(first,second))
-        self.assertEqual(self.store.conn.execute('SELECT count(*) FROM papers').fetchone()[0],2)
-        self.assertIsNotNone(self.store.conn.execute('SELECT 1 FROM papers WHERE id=?',(unrelated,)).fetchone())
-        self.assertEqual(self.store.conn.execute('SELECT count(*) FROM paper_merges').fetchone()[0],1)
-
     def test_manifest_count_mismatch_rolls_back_records(self):
         path=Path(self.tmp.name)/'source.jsonl'
         path.write_text(json.dumps({'source':'pmlr','external_id':'v1/paper','url':'https://example.org/paper','title':'Paper','abstract':'Abstract','authors':[],'venue':'ICML','year':2023,'track':'main','status':'accepted'})+'\n')

tests/test_joint_editions.py

@@ -0,0 +1,187 @@
+from contextlib import closing, redirect_stdout
+import hashlib
+import io
+import json
+import os
+from pathlib import Path
+import tempfile
+import unittest
+from unittest.mock import patch
+import uuid
+
+from paper_atlas.collectors.dblp import XML_SHA
+from paper_atlas.collectors.dblp_scope import scope
+from paper_atlas.collectors.dblp_records import normalize_row
+from paper_atlas.db.store import Store
+from paper_atlas.db.backend import open_store
+from paper_atlas.ingestion.joint_editions import (TARGET, MARKER, inspect, apply_plan, run)
+from paper_atlas.postgres.client import connect
+from paper_atlas.postgres.migration import migrate
+from paper_atlas.postgres.schema import quote
+from paper_atlas import querying
+
+
+def seed(store):
+    c = store.conn
+    c.execute("INSERT INTO venues(acronym,name,field,venue_type) VALUES(?,?,'reasoning','conference')", (TARGET,TARGET))
+    target = c.execute('SELECT id FROM venues WHERE acronym=?',(TARGET,)).fetchone()[0]
+    for alias in ['ECAI','IJCAI']:
+        c.execute('INSERT INTO venue_aliases VALUES(?,?)',(target,alias))
+    for year in [2018,2019,2022]:
+        metadata = {'snapshot_sha256':XML_SHA,'proceedings_key':'conf/ijcai/'+str(year),
+                    'bibliographic_year':year,'bibliography':{'booktitle':['IJCAI']}}
+        p,r=store.record('dblp','conf/ijcai/fixture'+str(year),'https://dblp.org/rec/fixture',
+                         'Original publication '+str(year),'Abstract\x00with\x01data',[],
+                         {'doi':'10.24963/ijcai.'+str(year)+'/1'}, {})
+        store.record_tags(p,r,'dblp',[{'namespace':'dblp:booktitle','value':'IJCAI'}])
+        store.decision(p,r,'IJCAI',year,'main','fixture'+str(year),'https://example.org/decision','accepted','listed')
+        stored=json.loads(c.execute('SELECT metadata_json FROM source_records WHERE id=?',(r,)).fetchone()[0]);stored.update(metadata)
+        c.execute('UPDATE source_records SET metadata_json=? WHERE id=?',(json.dumps(stored),r))
+    c.commit()
+
+
+def plan(c):
+    return {**inspect(c),'database':{'engine':'sqlite','path':'fixture'}}
+
+
+class JointEditionTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp=tempfile.TemporaryDirectory(); self.addCleanup(self.tmp.cleanup)
+        self.db=Path(self.tmp.name)/'sample.sqlite'
+        self.store=Store(self.db); self.addCleanup(self.store.conn.close)
+        seed(self.store); self.c=self.store.conn
+
+    def test_apply_preserves_every_protected_row_and_edition_id(self):
+        before=plan(self.c); ids=[r['id']for r in before['editions']]
+        result=apply_plan(self.c,before);self.c.commit()
+        self.assertEqual(result['updated_editions'],2)
+        self.assertEqual(result['content_fingerprints'],before['content_fingerprints'])
+        self.assertEqual([r['id']for r in result['editions']],ids)
+        untouched=self.c.execute('SELECT v.acronym FROM editions e JOIN venues v ON v.id=e.venue_id WHERE e.year=2019').fetchone()[0]
+        self.assertEqual(untouched,'IJCAI')
+        self.assertEqual(self.c.execute('PRAGMA foreign_key_check').fetchall(),[])
+        self.assertEqual(self.c.execute('SELECT count(*) FROM schema_migrations WHERE name=?',(MARKER,)).fetchone()[0],1)
+        with redirect_stdout(io.StringIO())as out:
+            querying.main(['--db',str(self.db),'--venue','ECAI'])
+        self.assertEqual(len(json.loads(out.getvalue())),2)
+        self.assertEqual(apply_plan(self.c,before)['updated_editions'],0)
+
+    def test_refresh_of_old_normalized_source_does_not_recreate_ordinary_edition(self):
+        before=plan(self.c);apply_plan(self.c,before)
+        for year in [2018,2022]:
+            row=self.c.execute('''SELECT r.paper_id,r.id,d.observed_at FROM source_records r
+                JOIN submissions s ON s.source_record_id=r.id JOIN decisions d ON d.submission_id=s.id
+                WHERE r.external_id=?''',('conf/ijcai/fixture'+str(year),)).fetchone()
+            self.store.decision(row[0],row[1],'IJCAI',year,'main','fixture'+str(year),
+                                'https://example.org/decision','accepted','listed',observed_at=row[2])
+        self.assertEqual(inspect(self.c)['content_fingerprints'],before['content_fingerprints'])
+        self.assertEqual(self.c.execute('SELECT count(*) FROM editions').fetchone()[0],3)
+
+    def test_any_changed_source_text_rejects_stale_plan_before_membership_updates(self):
+        before=plan(self.c)
+        self.c.execute("UPDATE source_records SET title='New evidence' WHERE external_id='conf/ijcai/fixture2018'")
+        with self.assertRaisesRegex(ValueError,'Stale repair plan'):
+            apply_plan(self.c,before)
+        self.assertEqual(inspect(self.c)['status'],'ready')
+
+    def test_conflicting_target_edition_is_never_merged_or_deleted(self):
+        target=self.c.execute('SELECT id FROM venues WHERE acronym=?',(TARGET,)).fetchone()[0]
+        self.c.execute("INSERT INTO editions(venue_id,year,track) VALUES(?,2018,'main')",(target,))
+        with self.assertRaisesRegex(ValueError,'colliding'):
+            inspect(self.c)
+        self.assertEqual(self.c.execute('SELECT count(*) FROM editions WHERE year=2018').fetchone()[0],2)
+
+    def test_wrong_source_volume_and_snapshot_fail_closed(self):
+        original=self.c.execute("SELECT metadata_json FROM source_records WHERE external_id='conf/ijcai/fixture2018'").fetchone()[0]
+        for key,value in [('proceedings_key','conf/ijcai/2018w'),('snapshot_sha256','wrong'),('bibliographic_year',2017),('bibliography',{'booktitle':['Other']})]:
+            m=json.loads(original);m[key]=value
+            self.c.execute("UPDATE source_records SET metadata_json=? WHERE external_id='conf/ijcai/fixture2018'",(json.dumps(m),))
+            with self.subTest(key=key),self.assertRaisesRegex(ValueError,'Source evidence'):
+                inspect(self.c)
+
+    def test_missing_source_support_and_wrong_aliases_are_not_guessed(self):
+        self.c.execute("DELETE FROM venue_aliases WHERE alias='ECAI'")
+        with self.assertRaisesRegex(ValueError,'aliases'):
+            inspect(self.c)
+        self.c.rollback()
+        self.c.execute('DELETE FROM submission_sources WHERE source_record_id=(SELECT id FROM source_records WHERE external_id=?)',('conf/ijcai/fixture2022',))
+        with self.assertRaisesRegex(ValueError,'Every submission'):
+            inspect(self.c)
+
+    def test_cli_plan_readonly_hash_binding_identity_and_repeat_apply(self):
+        out=Path(self.tmp.name)/'plan.json'
+        with redirect_stdout(io.StringIO()):run(self.db,out)
+        self.assertEqual(inspect(self.c)['status'],'ready')
+        raw=out.read_bytes();sha=hashlib.sha256(raw).hexdigest()
+        with patch('paper_atlas.ingestion.joint_editions.open_store')as writer:
+            with self.assertRaisesRegex(ValueError,'hash differs'):
+                run(self.db,Path(self.tmp.name)/'bad.json',plan=out,plan_sha256='0'*64)
+            writer.assert_not_called()
+        with redirect_stdout(io.StringIO()):
+            result=run(self.db,Path(self.tmp.name)/'applied.json',plan=out,plan_sha256=sha)
+            again=run(self.db,Path(self.tmp.name)/'retry.json',plan=out,plan_sha256=sha)
+        self.assertEqual(result['updated_editions'],2);self.assertEqual(again['updated_editions'],0)
+        self.assertEqual(result['plan_sha256'],sha);self.assertEqual(again['plan_sha256'],sha)
+        altered=json.loads(raw);altered['database']['path']='another.sqlite'
+        bad=Path(self.tmp.name)/'wrong-db.json';bad.write_text(json.dumps(altered))
+        with self.assertRaisesRegex(ValueError,'another database'):
+            run(self.db,Path(self.tmp.name)/'wrong-result.json',plan=bad,plan_sha256=hashlib.sha256(bad.read_bytes()).hexdigest())
+
+    def test_invalid_empty_plan_fails_before_opening_writer(self):
+        f=Path(self.tmp.name)/'empty.json';f.write_text('{}')
+        with patch('paper_atlas.ingestion.joint_editions.open_store')as writer:
+            with self.assertRaises(ValueError):
+                run(self.db,Path(self.tmp.name)/'empty-result.json',plan=f,plan_sha256=hashlib.sha256(f.read_bytes()).hexdigest())
+            writer.assert_not_called()
+
+    def test_collector_only_changes_the_two_exact_main_volume_keys(self):
+        for year in [2018,2019,2022]:
+            volume={'key':'conf/ijcai/'+str(year),'fields':{'title':['IJCAI '+str(year)],'year':[str(year)]}}
+            row={'record_type':'inproceedings','key':'conf/ijcai/One'+str(year),'fields':{'booktitle':['IJCAI'],
+                 'year':[str(year)],'crossref':[volume['key']],'title':['Original research title'],
+                 'author':['Jane Doe'],'ee':['https://doi.org/10.24963/ijcai.'+str(year)+'/1']}}
+            result,error=normalize_row(row,{volume['key']:volume},{'IJCAI','ECAI'})
+            self.assertIsNone(error)
+            self.assertEqual(result['venue'],TARGET if year in(2018,2022)else'IJCAI')
+            self.assertEqual(result['metadata']['bibliography']['booktitle'],['IJCAI'])
+            self.assertEqual(result['title'],'Original research title')
+            if year in(2018,2022):
+                self.assertIn('joint_edition_evidence',result['metadata'])
+                row['fields']['booktitle']=['IJCAI (Workshop)']
+                scoped,_=scope(row,volume)
+                self.assertEqual(scoped['venue'],'IJCAI')
+                self.assertEqual(scoped['track'],'workshop')
+
+
+@unittest.skipUnless(os.environ.get('PAPER_ATLAS_TEST_PG_SERVICE'),'Requires approved remote PostgreSQL service')
+class PostgreSQLJointTests(unittest.TestCase):
+    def setUp(self):
+        self.service=os.environ['PAPER_ATLAS_TEST_PG_SERVICE'];self.target='pa_joint_'+uuid.uuid4().hex[:12]
+        self.tmp=tempfile.TemporaryDirectory();self.addCleanup(self.tmp.cleanup)
+        self.db=Path(self.tmp.name)/'sample.sqlite';s=Store(self.db);seed(s);s.conn.close()
+        self.addCleanup(self.cleanup)
+        with redirect_stdout(io.StringIO()):migrate(self.db,self.service,Path(self.tmp.name)/'migration.json',schema=self.target,min_free_gib=0)
+
+    def cleanup(self):
+        with connect(self.service)as c:
+            for name, in c.execute('SELECT nspname FROM pg_namespace WHERE starts_with(nspname,%s)',(self.target,)).fetchall():
+                c.execute('DROP SCHEMA '+quote(name)+' CASCADE')
+
+    def test_native_repair_is_idempotent_keeps_codec_and_provenance(self):
+        p=Path(self.tmp.name)/'plan.json'
+        with redirect_stdout(io.StringIO()):
+            before=run(None,p,pg_service=self.service,pg_schema=self.target)
+            after=run(None,Path(self.tmp.name)/'applied.json',pg_service=self.service,pg_schema=self.target,plan=p,plan_sha256=hashlib.sha256(p.read_bytes()).hexdigest())
+        self.assertEqual(after['content_fingerprints'],before['content_fingerprints'])
+        with closing(open_store(pg_service=self.service,pg_schema=self.target).conn)as c:
+            self.assertEqual(apply_plan(c,before)['updated_editions'],0)
+            self.assertEqual(c.execute('SELECT abstract FROM papers ORDER BY id LIMIT 1').fetchone()[0],'Abstract\x00with\x01data')
+
+    def test_native_stale_source_plan_preserves_both_original_memberships(self):
+        with closing(open_store(pg_service=self.service,pg_schema=self.target).conn)as c:
+            before=plan(c)
+            c.execute("UPDATE source_records SET title='Changed' WHERE external_id=?",('conf/ijcai/fixture2022',));c.commit()
+            with self.assertRaisesRegex(ValueError,'Stale repair plan'):
+                apply_plan(c,before)
+            c.rollback()
+            self.assertEqual(inspect(c)['status'],'ready')

tests/test_postgres_ingestion.py

@@ -87,82 +87,6 @@ class PostgreSQLIngestionTests(unittest.TestCase):
         with redirect_stdout(io.StringIO()):
             ingest_file(None, path, pg_service=self.service, pg_schema=self.schema)
 
-    def test_incremental_source_coverage_counts_prior_chunks_and_refreshes_on_postgres(self):
-        rows = [self.row(source='dblp',external=key,status='accepted',year=2001,venue='UAI')
-                for key in ('incremental1','incremental2','incremental3')]
-        listing = 'https://example.org/uai/2001'
-        def manifest(path, **options):
-            target=path.with_suffix('.manifest.json')
-            value=json.loads(target.read_text())
-            value.update(list_url=listing,status='partial',**options)
-            target.write_text(json.dumps(value))
-        first=self.file('initial',rows[:2]);manifest(first);self.ingest(first)
-        second=self.file('append',[rows[2]])
-        manifest(second,coverage_mode='incremental',coverage_previous_count=2,
-                 coverage_previous_expected_count=2,coverage_expected_count=4)
-        self.ingest(second)
-        manifest(second,coverage_mode='incremental',coverage_previous_count=3,
-                 coverage_previous_expected_count=4,coverage_expected_count=4)
-        self.ingest(second)
-        with closing(ReadConnection(self.service,self.schema)) as c:
-            coverage=c.execute('SELECT status,expected_count,imported_count FROM edition_sources WHERE list_url=?',(listing,)).fetchone()
-            self.assertEqual(tuple(coverage.values),('partial',4,3))
-            stored=c.execute('SELECT abstract FROM source_records WHERE external_id=?',('incremental3',)).fetchone()
-            self.assertEqual(stored[0],rows[2]['abstract'])
-            self.assertEqual(c.execute("SELECT count(*) FROM source_records r JOIN sources s ON s.id=r.source_id WHERE s.name='dblp'").fetchone()[0],3)
-            run=c.execute('SELECT records_written,checkpoint_json FROM ingest_runs ORDER BY id DESC LIMIT 1').fetchone()
-            self.assertEqual(run[0],1)
-            self.assertEqual(json.loads(run[1])['coverage']['imported_count'],3)
-
-    def test_incremental_postgres_bound_failure_rolls_back_rows_and_summary(self):
-        listing='https://example.org/uai/2001'
-        rows=[self.row(source='dblp',external=key,status='accepted',year=2001,venue='UAI')
-              for key in ('bound1','bound2','bound3')]
-        first=self.file('initial',rows[:2]);target=first.with_suffix('.manifest.json')
-        value=json.loads(target.read_text());value.update(list_url=listing,status='partial');target.write_text(json.dumps(value))
-        self.ingest(first)
-        with closing(ReadConnection(self.service,self.schema)) as c:
-            before={table:c.execute('SELECT count(*) FROM '+table).fetchone()[0]
-                    for table in ('papers','source_records','submissions','decisions')}
-        path=self.file('overflow',[rows[2]]);target=path.with_suffix('.manifest.json')
-        value=json.loads(target.read_text());value.update(list_url=listing,status='partial',coverage_mode='incremental',
-            coverage_previous_count=2,coverage_previous_expected_count=2,coverage_expected_count=2)
-        target.write_text(json.dumps(value))
-        with self.assertRaisesRegex(ValueError,'source bounds'):self.ingest(path)
-        with closing(ReadConnection(self.service,self.schema)) as c:
-            self.assertEqual({table:c.execute('SELECT count(*) FROM '+table).fetchone()[0] for table in before},before)
-            coverage=c.execute('SELECT status,expected_count,imported_count FROM edition_sources WHERE list_url=?',(listing,)).fetchone()
-            self.assertEqual(tuple(coverage.values),('partial',2,2))
-            self.assertEqual(c.execute('SELECT status FROM ingest_runs ORDER BY id DESC LIMIT 1').fetchone()[0],'failed')
-
-    def test_incremental_postgres_does_not_overwrite_a_concurrently_changed_parent(self):
-        listing='https://example.org/uai/2001'
-        rows=[self.row(source='dblp',external=key,status='accepted',year=2001,venue='UAI')
-              for key in ('concurrent1','concurrent2','concurrent3')]
-        first=self.file('initial',rows[:2]);target=first.with_suffix('.manifest.json')
-        value=json.loads(target.read_text());value.update(list_url=listing,status='partial');target.write_text(json.dumps(value))
-        self.ingest(first)
-        path=self.file('racing',[rows[2]]);target=path.with_suffix('.manifest.json')
-        value=json.loads(target.read_text());value.update(list_url=listing,status='partial',coverage_mode='incremental',
-            coverage_previous_count=2,coverage_previous_expected_count=2,coverage_expected_count=4)
-        target.write_text(json.dumps(value))
-        original=Store.record
-        def change_parent(store,*args,**kwargs):
-            result=original(store,*args,**kwargs)
-            # A separate committed SQL writer need not honor the native
-            # advisory lock. Its update must survive our rejected file.
-            with connect(self.service) as other:
-                other.execute('SET search_path TO '+quote(self.schema)+',pg_catalog')
-                other.execute('UPDATE edition_sources SET expected_count=99 WHERE list_url=%s',(listing,))
-            return result
-        with patch.object(Store,'record',change_parent):
-            with self.assertRaisesRegex(ValueError,'parent changed during import'):self.ingest(path)
-        with closing(ReadConnection(self.service,self.schema)) as c:
-            coverage=c.execute('SELECT status,expected_count,imported_count FROM edition_sources WHERE list_url=?',(listing,)).fetchone()
-            self.assertEqual(tuple(coverage.values),('partial',99,2))
-            self.assertEqual(c.execute('SELECT count(*) FROM source_records WHERE external_id=?',('concurrent3',)).fetchone()[0],0)
-            self.assertEqual(c.execute('SELECT status FROM ingest_runs ORDER BY id DESC LIMIT 1').fetchone()[0],'failed')
-
     def test_native_writes_keep_nul_historical_ids_and_later_acceptance(self):
         store = self.store()
         try:
@@ -186,34 +110,6 @@ class PostgreSQLIngestionTests(unittest.TestCase):
         finally:
             reader.close()
 
-    def test_identity_anchor_keeps_other_arxiv_version_and_ambiguous_file_rolls_back(self):
-        title='An identical long research title shared by conference and extended versions'
-        authors=['Jane Doe','John Smith']
-        store=self.store()
-        try:
-            first,_=store.record('arxiv','1408.1485','https://arxiv.org/abs/1408.1485',title,'Conference abstract',authors)
-            second,_=store.record('arxiv','cs/0307069','https://arxiv.org/abs/cs/0307069',title,'Extended abstract',authors)
-            resolved,_=store.record('dblp','known','https://example.org/known',title,None,authors,{'arxiv':'1408.1485'})
-            self.assertEqual(resolved,first);self.assertNotEqual(first,second)
-            store.conn.commit()
-            self.assertEqual(store.conn.execute('SELECT count(*) FROM papers').fetchone()[0],3)
-            self.assertEqual(store.conn.execute('SELECT count(*) FROM paper_merges').fetchone()[0],1)  # migration fixture only
-            self.assertEqual(store.conn.execute('SELECT count(*) FROM match_candidates').fetchone()[0],1)
-        finally:store.conn.close()
-        rows=[self.row('official','earlier',status='accepted',venue='UAI',year=2001),
-              dict(self.row('official','ambiguous',status='accepted',venue='UAI',year=2001),
-                   title=title,abstract=None,authors=authors,identifiers={})]
-        with self.assertRaisesRegex(ValueError,'Ambiguous paper identity'):self.ingest(self.file('ambiguous-identity',rows))
-        store=self.store()
-        try:
-            self.assertEqual(store.conn.execute('SELECT count(*) FROM papers').fetchone()[0],3)
-            self.assertEqual(store.conn.execute('SELECT count(*) FROM source_records').fetchone()[0],4)
-            self.assertEqual(store.conn.execute('SELECT count(*) FROM submissions').fetchone()[0],1)
-            self.assertEqual(store.conn.execute('SELECT count(*) FROM paper_merges').fetchone()[0],1)
-            self.assertEqual(store.conn.execute('SELECT status FROM ingest_runs ORDER BY id DESC LIMIT 1').fetchone()[0],'failed')
-            self.assertEqual(store.conn.execute("SELECT count(*) FROM source_records WHERE external_id IN (?,?)",('earlier','ambiguous')).fetchone()[0],0)
-        finally:store.conn.close()
-
     def test_file_import_is_idempotent_and_refreshes_only_own_tags(self):
         row = self.row(); row['tags'] = [{'namespace': 'openreview_snapshot:area', 'value': 'theory'}]
         path = self.file('reject', [row]); self.ingest(path)