Prepare evidence-bound identifier additions without rewriting source records

.gitea/workflows/check.yml

@@ -49,12 +49,12 @@ jobs:
               subprocess.run([str(venv/'bin/paper-atlas'),'--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','collect','cvf','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','postgres','--help'],cwd=outside,check=True)
-              subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','joint-editions','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','import','records','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','import','arxiv','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','backfill','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','audit','public-openreview','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','audit','histories','--help'],cwd=outside,check=True)
+              subprocess.run([python,'-I','-B','-m','paper_atlas','curate','identifiers','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-c','from paper_atlas.postgres.writer import PostgresStore, WriteConnection'],cwd=outside,check=True)
               smoke = "from paper_atlas.db.store import Store; from paper_atlas.config import DEFAULT_REGISTRY; import json; assert {'COLM','MM','MLSys'} <= {v['acronym'] for v in json.loads(DEFAULT_REGISTRY.read_text())['venues']}; s=Store('installed.sqlite'); s.conn.close()"
               subprocess.run([python,'-I','-B','-c',smoke],cwd=outside,check=True)

README.md

@@ -86,5 +86,3 @@ paper-atlas collect colm --cache /data/paper-index/raw/http-cache \
 CI 在已授权的远程 runner 上运行测试、构建 wheel，并从源码目录外验证安装后的命令及随包资源。数据库全量去重和覆盖率是独立的数据验收步骤，不因代码测试通过而自动完成。
 
 开发通过工作分支和 PR 交付，由用户手动合并。
-
-历史联合会议的既有届次归属可通过 [`maintain joint-editions`](docs/joint-editions.md) 先只读规划，再按计划哈希执行；保留论文与投稿决定来源，不重导入。

docs/joint-editions.md

@@ -1,67 +0,0 @@
-# Repairing joint conference membership
-
-EURAI's [official history](https://www.eurai.org/ecai/history) identifies ECAI 2018
-and 2022 as joint IJCAI/ECAI conferences. The pinned DBLP volumes
-`conf/ijcai/2018` and `conf/ijcai/2022` use `IJCAI` as their booktitle and title.
-Existing records can therefore already be present under IJCAI while an ECAI query
-shows a missing year. Adding ECAI as a global alias to the ordinary IJCAI venue
-would incorrectly include all other IJCAI years.
-
-The existing canonical `ECAI+IJCAI` venue has both member aliases. This maintenance
-command changes only the venue reference of the two existing `main` edition rows.
-Edition IDs, submission IDs, decisions, source text, authors, identifiers, tags,
-source links and coverage records stay intact. It does not merge or create papers
-or submissions, resolve per-paper tracks, or turn reviews into negative decisions.
-The DBLP collector applies the joint mapping only to these two exact main volume
-keys; original booktitle/title remain in source metadata with the membership evidence.
-The shared ingestion entry point also resolves these exact pinned source volumes before
-creating an edition, so refreshing an older normalized file cannot recreate a separate
-IJCAI edition. Other sources, years, workshop volumes and additional tracks retain their existing rules.
-
-## Review a read-only plan
-
-Use a runtime containing this repair command. The configured PostgreSQL writer lock
-is used only for the apply operation; do not replace a newer ingestion runtime with
-an older main-based feature branch.
-
-```sh
-PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain joint-editions \
-  --pg-service foundry-research --pg-schema research --out joint-plan.json
-```
-
-The preflight requires the existing joint venue and exactly its two member aliases,
-one nonempty unambiguous edition per year, and matching pinned DBLP volume evidence
-for every existing submission. A colliding target edition, missing provenance or
-unexpected source scope fails closed. The plan binds current database/schema, IDs,
-counts and full fingerprints of affected papers, identifiers, source records, tags,
-submissions, decisions and their provenance, and coverage rows. Observation time is
-recorded separately from conference year. These are scoped checks, not a final global
-identity or conference completeness audit.
-
-## Apply the exact reviewed plan
-
-Compute the file's SHA256 after review and pass it explicitly:
-
-```sh
-PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain joint-editions \
-  --pg-service foundry-research --pg-schema research --plan joint-plan.json \
-  --plan-sha256 "$PLAN_SHA256" --out joint-applied.json
-```
-
-The plan hash is checked before opening a writer. Current evidence is checked again
-under the dataset writer lock; both edition updates and an audit marker commit in one
-transaction. Protected row hashes are checked again before commit. Changed evidence
-requires a fresh reviewed plan. Existing output reports are preserved; choose a new
-path for another run. SQLite samples use `--db sample.sqlite`; the frozen production
-backup remains protected.
-
-If a response is lost after commit, rerun with the same plan and a new output path.
-When protected evidence is unchanged and both editions already point to the joint
-venue, the result is `already_joint` with zero updates. It does not replay imports or
-regenerate source records. Keep the plan, apply report and independent postcheck
-alongside collection evidence. The audit marker is `joint_ijcai_ecai_2018_2022_v1`.
-
-Publication counts can differ between the DBLP volume and the current publisher
-page. This membership repair preserves all existing rows rather than deleting entries
-to force an aggregate match; section-level main/invited/other classification and
-publication completeness require separate per-paper evidence.

docs/usage.md

@@ -2,6 +2,55 @@
 
 先在已确认的远程环境安装项目，使用 `paper-atlas`。从源码运行时，将命令前缀替换为 `python3 -B -m paper_atlas`，其余参数相同。数据目录位于仓库外。
 
+## 按证据补充稳定标识
+
+`paper-atlas curate identifiers --plan /data/review/plan.json --output /data/review/prepared`
+只生成新的 JSONL、manifest、证据副本及 `curation-report.json`，不会连接数据库。
+输出目录必须不存在，父目录必须已存在；任何输入或断言失败，都不发布本批输出。
+原始文件不变，未选中的记录逐字节保留。题名、作者、年份、页码、投稿决定、标签和既有 metadata
+均不重写，仅补充 `identifiers` 并追加 `metadata.identifier_curation` 证据链。
+
+计划格式如下（哈希占位符须替换为真实 SHA256，所有路径按命令当前目录解析）：
+
+```json
+{
+  "format_version": 1,
+  "inputs": [{
+    "file": "/data/review/uai2001.jsonl",
+    "sha256": "<JSONL 文件 SHA256>",
+    "manifest_sha256": "<同名 .manifest.json 文件 SHA256>"
+  }],
+  "assertions": [{
+    "source": "dblp",
+    "external_id": "conf/uai/example",
+    "record_sha256": "<原始物理行字节 SHA256，包含该行原有换行符>",
+    "identifiers": {"arxiv": "1408.1485"},
+    "evidence": [{
+      "file": "/data/review/primary-metadata.json",
+      "sha256": "<证据 JSON 文件 SHA256>",
+      "url": "https://example.org/primary-record",
+      "checked_at": "2026-11-08T08:00:00Z",
+      "note": "说明人工核对的题名、作者、出版版本和稳定标识证据"
+    }]
+  }]
+}
+```
+
+输入 manifest 必须包含 `source/venue/year/track/count/list_url/file_sha256`，且与逐条记录一致。
+断言必须唯一、准确命中一条记录。仅支持无版本后缀的 arXiv 编号与小写规范 DOI；
+已有同类标识不允许覆盖，相同值可以补记证据。证据只接受不超过 16 MiB 的 JSON 元数据文件，
+来源 URL 要求 HTTPS 且不含凭据，核查时间必须带时区；不下载论文 PDF。
+
+`verification_scope=input_and_evidence_integrity_only` 和 `semantic_identity_verified=false`
+明确表示哈希校验不能证明两条记录属于同一作品。请先独立核对版本关系；仅年份、页码或相同题名
+不能作为合并依据。存在来源冲突的行应继续暂缓。新 manifest 保留原来的 partial 状态及摘要缺失数，
+不会把补充编号伪装成来源覆盖或摘要补全。证据副本按 SHA256 保存在输出目录 `evidence/`，
+原计划也保存在 `plan.json`，供之后复查。
+
+文件生成后仍需另做当前 PostgreSQL 身份、投稿和决定冲突预检；导入运行时必须包含
+“稳定标识优先、多个标题候选则整文件回滚”的身份保护，再以 `import records` 独立导入并验收。
+该命令不会自动导入或合并实体，也不代表最终全库去重通过。
+
 ## PostgreSQL 查询
 
 在远程环境安装 `.[postgres]`，通过权限为 `0600` 的 libpq service 文件连接，不把密码放入 URL、命令行或仓库。
@@ -99,6 +148,8 @@ paper-atlas query --pg-service foundry-research \
 
 `--stats` 中 papers 是当前规则下的论文实体数；source_records 是来源记录数；decisions 是决定观察数。一个投稿可能有多个来源和多条观察，不能相加当作论文数。
 
+已有稳定标识或原来源记录匹配时，该身份优先；同标题、同作者的另一 arXiv 存档或扩展稿不会仅因相似标题被并入。没有身份锚点时，作者佐证的精确标题只有一个候选才可匹配；多个候选会报 `Ambiguous paper identity`，整个采集文件回滚并记录失败运行。应补有证据的标识或先核查版本关系，不删除重复候选或任取第一个。一个新来源明确同时携带两个既有稳定标识时，仍可按既有规则归并并保留投稿、标签和合并日志；未归并的同标题关系继续保留为待核查候选。这些规则不证明全库去重完成。
+
 ## 5. 投稿历史关系审计
 
 ```sh

paper_atlas/cli.py

@@ -6,6 +6,7 @@ from paper_atlas import __version__
 
 
 COMMANDS = {
+    ('curate','identifiers'): ('paper_atlas.curation.identifiers', {}, '按证据补充稳定标识并生成独立待导入文件'),
     ('query',): ('paper_atlas.querying', {}, '查询论文、标签和投稿历史'),
     ('coverage',): ('paper_atlas.audits.coverage', {}, '查看全部候选会议的覆盖情况'),
     ('download',): ('paper_atlas.io.snapshots', {}, '下载并校验固定版本快照'),
@@ -25,7 +26,6 @@ COMMANDS = {
     ('catalogue','dblp'): ('paper_atlas.collectors.dblp', {}, '盘点 DBLP 历史会议目录'),
     ('import','arxiv'): ('paper_atlas.ingestion.arxiv', {}, '导入 arXiv 元数据 ZIP'),
     ('import','records'): ('paper_atlas.ingestion.records', {}, '导入核验过的会议文件'),
-    ('maintain','joint-editions'): ('paper_atlas.ingestion.joint_editions', {}, '按已核验计划修正既有联合会议届次归属'),
     ('maintain','merge'): ('paper_atlas.ingestion.maintenance', {}, '协调单写入合并与恢复'),
     ('maintain','backfill'): ('paper_atlas.ingestion.backfill', {}, '回填既有投稿的来源身份'),
     ('maintain','arxiv-tags'): ('paper_atlas.ingestion.arxiv_tags', {}, '恢复逐来源记录的 arXiv 类别证据'),
@@ -49,7 +49,7 @@ def main(argv=None):
             leaf=commands.add_parser(route[0],help=help_text,add_help=False)
         else:
             if route[0] not in groups:
-                group=commands.add_parser(route[0],help={'collect':'采集来源元数据','catalogue':'盘点历史目录','import':'导入数据库','maintain':'维护数据库','audit':'核验数据'}[route[0]])
+                group=commands.add_parser(route[0],help={'collect':'采集来源元数据','catalogue':'盘点历史目录','import':'导入数据库','maintain':'维护数据库','audit':'核验数据','curate':'按证据整理元数据'}[route[0]])
                 groups[route[0]]=group.add_subparsers(dest='operation',required=True)
             leaf=groups[route[0]].add_parser(route[1],help=help_text,add_help=False)
         leaf.set_defaults(handler=(route,module,kwargs))

paper_atlas/collectors/dblp_records.py

@@ -88,8 +88,6 @@ def normalize_row(row,volumes,targets,include_workshops=False,through_year=2026)
               'scope_rule':edition['rule'],'author_names_raw':fields.get('author',[]),'publtype':row.get('publtype'),
               'bibliography':{key:value for key,value in fields.items() if key in ('booktitle','year','ee','url','pages','volume','number','publisher','series','isbn')},
               'ambiguous_identifiers':ambiguous,'publication_evidence':'inproceedings bibliography; not an original decision note'}
-    if edition.get('joint_edition_evidence'):
-        metadata['joint_edition_evidence'] = edition['joint_edition_evidence']
     if edition.get('track_scope_note'):
         metadata['track_scope_note']=edition['track_scope_note']
     flagged=bool(re.search(r'\b(?:withdrawn|retracted)\b',row.get('publtype') or '',re.I))

paper_atlas/collectors/dblp_scope.py

@@ -96,15 +96,8 @@ def scope(row,volume,through_year=2026):
     elif suffix.casefold() in ('poster','posters') and venue not in ('ICLR','ACL','EMNLP','NAACL','EACL','COLING','CoNLL'):track='poster'
     if venue=='AAAI+IAAI':track='proceedings_unspecified'
     if venue=='AAMAS' and year<2002:return None,'predecessor_not_aamas'
-    joint_evidence = None
-    if (family == 'ijcai' and base == 'IJCAI' and track == 'main'
-            and year in (2018, 2022) and volume['key'] == 'conf/ijcai/' + str(year)):
-        venue, aliases = 'ECAI+IJCAI', ['ECAI', 'IJCAI']
-        joint_evidence = 'https://www.eurai.org/ecai/history'
     result = {'venue':venue,'venue_aliases':aliases,'year':year,'track':track,'bibliographic_year':int(bibliographic[0]),
               'rule':family+':'+base,'proceedings_title':title,'edition_year_basis':year_basis}
-    if joint_evidence:
-        result['joint_edition_evidence'] = joint_evidence
     if venue=='MM' and track=='main':
         # A main MM volume also includes doctoral, tutorials, companion papers
         # and other sections. DBLP XML omits the per-paper TOC section.

paper_atlas/curation/__init__.py

@@ -0,0 +1 @@
+"""Offline, evidence-bound preparation of metadata for separate ingestion."""

paper_atlas/curation/identifiers.py

@@ -0,0 +1,212 @@
+"""Add reviewed stable identifiers without rewriting bibliographic source fields.
+
+Hashes establish file integrity, not the truth of an identity assertion. Identity
+review and a fresh database preflight remain separate requirements.
+"""
+import argparse
+from datetime import datetime
+import hashlib
+import json
+from pathlib import Path
+import re
+import tempfile
+from urllib.parse import urlsplit
+
+
+SCOPE = 'input_and_evidence_integrity_only'
+MAX_METADATA_BYTES = 16 * 1024 * 1024
+
+
+def digest(data):
+    return hashlib.sha256(data).hexdigest()
+
+
+def object_keys(value, required, optional=()):
+    if not isinstance(value, dict) or not set(required) <= value.keys() or value.keys() - set(required) - set(optional):
+        raise ValueError('Missing or unsupported plan fields')
+
+
+def json_bytes(value):
+    return (json.dumps(value, ensure_ascii=False, separators=(',', ':'), allow_nan=False) + '\n').encode('utf-8')
+
+
+def decode(raw):
+    def unique(pairs):
+        result = {}
+        for key, value in pairs:
+            if key in result:
+                raise ValueError('Duplicate JSON key: ' + key)
+            result[key] = value
+        return result
+    return json.loads(raw, object_pairs_hook=unique,
+                      parse_constant=lambda value: (_ for _ in ()).throw(ValueError('Non-finite JSON value')))
+
+
+def metadata(path, expected=None):
+    with Path(path).open('rb') as stream:
+        raw = stream.read(MAX_METADATA_BYTES + 1)
+    if len(raw) > MAX_METADATA_BYTES:
+        raise ValueError('Metadata exceeds size limit')
+    if expected is not None:
+        check_hash(raw, expected)
+    return raw, decode(raw)
+
+
+def check_hash(raw, expected):
+    if not isinstance(expected, str) or not re.fullmatch('[0-9a-f]{64}', expected) or digest(raw) != expected:
+        raise ValueError('SHA256 mismatch')
+
+
+def identifier(scheme, value):
+    if not isinstance(value, str) or any(char.isspace() or ord(char) < 32 or ord(char) == 127 for char in value):
+        raise ValueError('Invalid identifier')
+    if scheme == 'arxiv':
+        pattern = r'(?:\d{2}(?:0[1-9]|1[0-2])\.\d{4,5}|[a-zA-Z][a-zA-Z.-]*/\d{2}(?:0[1-9]|1[0-2])\d{3})'
+        valid = re.fullmatch(pattern, value)
+    elif scheme == 'doi':
+        valid = re.fullmatch(r'10\.\d{4,9}/\S+', value) and value == value.lower()
+    else:
+        valid = False
+    if not valid:
+        raise ValueError('Unsupported or noncanonical identifier: ' + scheme)
+
+
+def prepare(plan_path, output):
+    plan_raw, plan = metadata(plan_path)
+    object_keys(plan, ('format_version', 'inputs', 'assertions'))
+    if type(plan['format_version']) is not int or plan['format_version'] != 1:
+        raise ValueError('Unsupported format version')
+    if not isinstance(plan['inputs'], list) or not plan['inputs'] or not isinstance(plan['assertions'], list) or not plan['assertions']:
+        raise ValueError('Inputs and assertions must be nonempty lists')
+    output = Path(output).absolute()
+    if output.exists():
+        raise ValueError('Output directory must not already exist')
+    if not output.parent.is_dir():
+        raise ValueError('Output parent directory must exist')
+    assertions = {}
+    evidence_files = {}
+    checked_evidence = {}
+    for assertion in plan['assertions']:
+        object_keys(assertion, ('source', 'external_id', 'record_sha256', 'identifiers', 'evidence'))
+        if any(not isinstance(assertion[key], str) or not assertion[key] for key in ('source', 'external_id')):
+            raise ValueError('Invalid source identity')
+        key = (assertion['source'], assertion['external_id'])
+        if key in assertions:
+            raise ValueError('Duplicate identity assertion')
+        ids = assertion['identifiers']
+        if not isinstance(ids, dict) or not ids:
+            raise ValueError('Identifiers must be a nonempty object')
+        for scheme, value in ids.items():
+            identifier(scheme, value)
+        if not isinstance(assertion['evidence'], list) or not assertion['evidence']:
+            raise ValueError('Evidence is required')
+        references = []
+        for evidence in assertion['evidence']:
+            object_keys(evidence, ('file', 'sha256', 'url', 'checked_at', 'note'))
+            if not all(isinstance(value, str) and value for value in evidence.values()):
+                raise ValueError('Invalid evidence fields')
+            url = urlsplit(evidence['url'])
+            if url.scheme != 'https' or not url.hostname or url.username or url.password:
+                raise ValueError('Evidence requires an HTTPS URL without credentials')
+            if datetime.fromisoformat(evidence['checked_at'].replace('Z', '+00:00')).utcoffset() is None:
+                raise ValueError('Evidence timestamp requires a timezone')
+            evidence_key = (str(Path(evidence['file']).absolute()), evidence['sha256'])
+            if evidence_key not in checked_evidence:
+                checked_evidence[evidence_key] = metadata(evidence['file'], evidence['sha256'])[0]
+            raw = checked_evidence[evidence_key]
+            evidence_files[evidence['sha256']] = raw
+            references.append({**evidence, 'file': 'evidence/' + evidence['sha256'] + '.json'})
+        assertions[key] = (assertion, references)
+    seen = set()
+    applied = set()
+    names = set()
+    report = {'format_version': 1, 'plan_sha256': digest(plan_raw),
+              'verification_scope': SCOPE, 'semantic_identity_verified': False,
+              'database_written': False, 'files': [], 'assertions_applied': 0, 'identifiers_added': 0}
+    with tempfile.TemporaryDirectory(prefix='.identifier-curation-', dir=output.parent) as temporary:
+        staging = Path(temporary) / 'prepared'
+        staging.mkdir()
+        (staging / 'evidence').mkdir()
+        (staging / 'plan.json').write_bytes(plan_raw)
+        for sha, raw in evidence_files.items():
+            (staging / 'evidence' / (sha + '.json')).write_bytes(raw)
+        for entry in plan['inputs']:
+            object_keys(entry, ('file', 'sha256', 'manifest_sha256'))
+            path = Path(entry['file'])
+            if path.suffix != '.jsonl' or path.name in names:
+                raise ValueError('Inputs require distinct JSONL basenames')
+            names.add(path.name)
+            _, manifest = metadata(path.with_suffix('.manifest.json'), entry['manifest_sha256'])
+            if not isinstance(manifest, dict) or not {'source','venue','year','track','count','list_url','file_sha256'} <= manifest.keys():
+                raise ValueError('Complete source manifest required')
+            if manifest['file_sha256'] != entry['sha256']:
+                raise ValueError('Manifest and plan hashes differ')
+            before = hashlib.sha256()
+            after = hashlib.sha256()
+            count = 0
+            with path.open('rb') as source, (staging / path.name).open('wb') as target:
+                for raw in source:
+                    before.update(raw)
+                    row = decode(raw)
+                    if any(row[field] != manifest[field] for field in ('source', 'venue', 'year', 'track')):
+                        raise ValueError('Record differs from source manifest')
+                    key = (row['source'], row['external_id'])
+                    if key in seen:
+                        raise ValueError('Duplicate source identity in input batch')
+                    seen.add(key)
+                    if key in assertions:
+                        assertion, references = assertions[key]
+                        check_hash(raw, assertion['record_sha256'])
+                        existing = row.get('identifiers', {})
+                        if existing is None:
+                            existing = {}
+                        if not isinstance(existing, dict):
+                            raise ValueError('Invalid existing identifiers')
+                        for scheme, value in assertion['identifiers'].items():
+                            if scheme in existing and existing[scheme] != value:
+                                raise ValueError('Existing identifier conflict')
+                        details = row.setdefault('metadata', {})
+                        if not isinstance(details, dict):
+                            raise ValueError('Invalid record metadata')
+                        history = details.setdefault('identifier_curation', [])
+                        if not isinstance(history, list):
+                            raise ValueError('Invalid curation history')
+                        history.append({'plan_sha256': digest(plan_raw), 'input_file_sha256': entry['sha256'],
+                                        'record_sha256': digest(raw), 'previous_identifiers': dict(existing),
+                                        'identifiers_asserted': assertion['identifiers'], 'evidence': references,
+                                        'verification_scope': SCOPE})
+                        report['identifiers_added'] += len(assertion['identifiers'].keys() - existing.keys())
+                        row['identifiers'] = {**existing, **assertion['identifiers']}
+                        raw = json_bytes(row)
+                        applied.add(key)
+                    target.write(raw)
+                    after.update(raw)
+                    count += 1
+            if before.hexdigest() != entry['sha256'] or not re.fullmatch('[0-9a-f]{64}', entry['sha256']):
+                raise ValueError('Input SHA256 mismatch')
+            if not count or type(manifest['count']) is not int or count != manifest['count']:
+                raise ValueError('Manifest count mismatch')
+            provenance = {'plan_sha256': digest(plan_raw), 'input_file_sha256': before.hexdigest(),
+                          'input_manifest_sha256': entry['manifest_sha256'], 'verification_scope': SCOPE}
+            prepared = {**manifest, 'file': str(output / path.name), 'file_sha256': after.hexdigest(),
+                        'identifier_curation': provenance}
+            (staging / path.with_suffix('.manifest.json').name).write_bytes(json_bytes(prepared))
+            report['files'].append({'file': str(output / path.name), 'count': count,
+                                    'input_sha256': before.hexdigest(), 'sha256': after.hexdigest()})
+        if applied != assertions.keys():
+            raise ValueError('Assertion target absent from input batch')
+        report['assertions_applied'] = len(applied)
+        (staging / 'curation-report.json').write_bytes(json_bytes(report))
+        # All files are on the same filesystem. No database or input mutation.
+        if output.exists():
+            raise ValueError('Output directory appeared during preparation')
+        staging.rename(output)
+    return report
+
+
+def main(argv=None, *, prog=None):
+    parser = argparse.ArgumentParser(prog=prog, description=__doc__)
+    parser.add_argument('--plan', required=True, type=Path)
+    parser.add_argument('--output', required=True, type=Path)
+    args = parser.parse_args(argv)
+    print(json.dumps(prepare(args.plan, args.output), ensure_ascii=False))

paper_atlas/db/store.py

@@ -95,13 +95,19 @@ class Store:
             matches.add(old[0])
         key = normalized(title)
         title_matches = c.execute('SELECT id,authors_json,abstract FROM papers WHERE title_key=?',(key,)).fetchall()
-        authors_set = author_keys(authors)
-        for paper,old_authors,old_abstract in title_matches:
-            overlap = authors_set & author_keys(json.loads(old_authors))
-            # Cross-platform title matching needs independent author corroboration.
-            same_source = c.execute('SELECT 1 FROM source_records WHERE paper_id=? AND source_id=? LIMIT 1',(paper,source_id)).fetchone()
-            if not same_source and len(key)>=25 and (len(overlap)>=2 or (overlap and abstract and abstract.strip()==(old_abstract or '').strip())):
-                matches.add(paper)
+        # An existing identifier/source record anchors the entity. Title
+        # similarity must not pull in another deposit or an extended version.
+        if not matches:
+            title_only_matches = set()
+            authors_set = author_keys(authors)
+            for paper,old_authors,old_abstract in title_matches:
+                overlap = authors_set & author_keys(json.loads(old_authors))
+                same_source = c.execute('SELECT 1 FROM source_records WHERE paper_id=? AND source_id=? LIMIT 1',(paper,source_id)).fetchone()
+                if not same_source and len(key)>=25 and (len(overlap)>=2 or (overlap and abstract and abstract.strip()==(old_abstract or '').strip())):
+                    title_only_matches.add(paper)
+            if len(title_only_matches)>1:
+                raise ValueError('Ambiguous paper identity: multiple author-corroborated title matches; provide verified identifiers')
+            matches = title_only_matches
         if matches:
             paper = min(matches)
             for other in sorted(matches-{paper}):
@@ -144,20 +150,6 @@ class Store:
 
     def decision(self,paper,record,venue,year,track,external_id,url,status,raw_value,aliases=None,identities=None,observed_at=None,decision_date=None,source_decision_id='',submitted_at=None,guard_status=False):
         c = self.conn
-        # An already collected file may still call these exact pinned volumes
-        # IJCAI. Resolve membership before finding/creating an edition so a
-        # later refresh cannot split the repaired joint event again.
-        if venue == 'IJCAI' and year in (2018,2022) and track == 'main':
-            from paper_atlas.collectors.dblp import XML_SHA
-            evidence=c.execute('SELECT src.name,r.metadata_json FROM source_records r JOIN sources src ON src.id=r.source_id WHERE r.id=?',(record,)).fetchone()
-            if evidence and evidence[0] == 'dblp':
-                metadata=json.loads(evidence[1])
-                if (metadata.get('snapshot_sha256') == XML_SHA
-                        and metadata.get('proceedings_key') == 'conf/ijcai/'+str(year)
-                        and metadata.get('bibliographic_year') == year
-                        and (metadata.get('bibliography') or {}).get('booktitle') in (['IJCAI'],['IJCAI/ECAI'])):
-                    venue='ECAI+IJCAI'
-                    aliases=sorted(set(aliases or []) | {'ECAI','IJCAI'})
         existing=c.execute('SELECT id FROM venues WHERE lower(acronym)=lower(?)',(venue,)).fetchone()
         if existing:venue_id=existing[0]
         else:venue_id=inserted_id(c,"INSERT INTO venues(acronym,name,field,venue_type) VALUES(?,?,?,'conference')",(venue,venue,'unclassified'))

paper_atlas/ingestion/joint_editions.py

@@ -1,204 +0,0 @@
-"""Evidence-bound repair of two existing IJCAI/ECAI edition memberships."""
-import argparse
-from contextlib import closing
-import hashlib
-import json
-from pathlib import Path
-
-from paper_atlas.db.backend import database_arguments, open_read_connection, open_store
-from paper_atlas.db.submissions import timestamp
-from paper_atlas.collectors.dblp import XML_SHA
-
-YEARS = (2018, 2022)
-TARGET = 'ECAI+IJCAI'
-MEMBERS = {'ECAI', 'IJCAI'}
-EVIDENCE = 'https://www.eurai.org/ecai/history'
-MARKER = 'joint_ijcai_ecai_2018_2022_v1'
-
-
-def _digest(c, sql, params=()):
-    sha = hashlib.sha256(); count = 0
-    for row in c.execute(sql, params):
-        sha.update((json.dumps([row[i] for i in range(len(row))], ensure_ascii=False,
-                              separators=(',', ':')) + '\n').encode())
-        count += 1
-        if count > 100000:
-            raise ValueError('Joint repair evidence scope exceeded bounded row limit')
-    return {'rows': count, 'sha256': sha.hexdigest()}
-
-
-def fingerprints(c, ids):
-    marks = ','.join('?' for _ in ids)
-    submissions = 'SELECT id FROM submissions WHERE edition_id IN (' + marks + ')'
-    papers = 'SELECT paper_id FROM submissions WHERE edition_id IN (' + marks + ')'
-    decisions = 'SELECT id FROM decisions WHERE submission_id IN (' + submissions + ')'
-    checks = {
-        'papers': ('id IN (' + papers + ')', 'id'),
-        'identifiers': ('paper_id IN (' + papers + ')', 'scheme,value'),
-        'source_records': ('paper_id IN (' + papers + ')', 'id'),
-        'paper_tags': ('paper_id IN (' + papers + ')', 'paper_id,tag_id,evidence_key'),
-        'submissions': ('edition_id IN (' + marks + ')', 'id'),
-        'decisions': ('submission_id IN (' + submissions + ')', 'id'),
-        'submission_sources': ('submission_id IN (' + submissions + ')', 'submission_id,source_record_id,source_submission_id'),
-        'submission_identifiers': ('edition_id IN (' + marks + ')', 'edition_id,scheme,value'),
-        'decision_sources': ('decision_id IN (' + decisions + ')', 'decision_id,source_record_id,source_decision_id'),
-        'edition_sources': ('edition_id IN (' + marks + ')', 'edition_id,source_id,list_url'),
-    }
-    return {table: _digest(c, 'SELECT * FROM ' + table + ' WHERE ' + where + ' ORDER BY ' + order, ids)
-            for table, (where, order) in checks.items()}
-
-
-def inspect(c):
-    target = c.execute('SELECT id FROM venues WHERE acronym=?', (TARGET,)).fetchall()
-    if len(target) != 1:
-        raise ValueError('Exactly one existing canonical joint venue required')
-    target = target[0][0]
-    aliases = {r[0] for r in c.execute('SELECT alias FROM venue_aliases WHERE venue_id=?', (target,))}
-    if aliases != MEMBERS:
-        raise ValueError('Joint target must already have exactly ECAI and IJCAI aliases')
-    editions = []
-    for year in YEARS:
-        rows = c.execute('''SELECT e.id,e.venue_id,v.acronym FROM editions e JOIN venues v ON v.id=e.venue_id
-            WHERE v.acronym IN ('IJCAI','ECAI+IJCAI') AND e.year=? AND e.track='main' ''', (year,)).fetchall()
-        if len(rows) != 1:
-            raise ValueError('Expected one existing edition per year; missing or colliding editions require separate review')
-        edition, venue, acronym = (rows[0][i] for i in range(3))
-        count = c.execute('SELECT count(*) FROM submissions WHERE edition_id=?', (edition,)).fetchone()[0]
-        if not 0 < count <= 5000:
-            raise ValueError('Existing edition must have a nonempty bounded submission scope')
-        # Every existing submission must be supported by its own pinned DBLP
-        # volume record. No title/author matching, new attempts or text writes.
-        evidence = c.execute('''SELECT s.id,r.external_id,r.metadata_json FROM submissions s
-            JOIN submission_sources ss ON ss.submission_id=s.id
-            JOIN source_records r ON r.id=ss.source_record_id JOIN sources src ON src.id=r.source_id
-            WHERE s.edition_id=? AND src.name='dblp' ORDER BY s.id,r.id''', (edition,)).fetchall()
-        seen = set()
-        for row in evidence:
-            sid, external, metadata = (row[i] for i in range(3))
-            meta = json.loads(metadata)
-            bibliography = meta.get('bibliography') or {}
-            if (meta.get('snapshot_sha256') != XML_SHA
-                    or meta.get('proceedings_key') != 'conf/ijcai/' + str(year)
-                    or meta.get('bibliographic_year') != year
-                    or bibliography.get('booktitle') not in (['IJCAI'], ['IJCAI/ECAI'])
-                    or not external.startswith('conf/ijcai/')):
-                raise ValueError('Source evidence outside the exact verified IJCAI/ECAI volume')
-            seen.add(sid)
-        if len(seen) != count:
-            raise ValueError('Every submission must have matching original volume evidence')
-        editions.append({'id': edition, 'year': year, 'track': 'main', 'from_venue_id': venue,
-                         'from_venue': acronym, 'target_venue_id': target, 'submissions': count,
-                         'evidence_url': EVIDENCE, 'proceedings_key': 'conf/ijcai/' + str(year)})
-    ids = tuple(e['id'] for e in editions)
-    return {'format_version': 1, 'kind': MARKER, 'checked_at': timestamp(), 'editions': editions,
-            'content_fingerprints': fingerprints(c, ids), 'target_aliases': sorted(aliases),
-            'status': 'already_joint' if all(e['from_venue_id'] == target for e in editions) else 'ready',
-            'scope': 'Only these two existing edition venue references; no source text, paper, submission, decision, track, identifier or tag change.'}
-
-
-def validate_plan(expected):
-    if not isinstance(expected, dict):
-        raise ValueError('Repair plan must be a JSON object')
-    if set(expected) != {'format_version','kind','checked_at','editions','content_fingerprints','target_aliases','status','scope','database'}:
-        raise ValueError('Unexpected repair plan fields')
-    if (type(expected['format_version']) is not int or expected['format_version'] != 1
-            or expected['kind'] != MARKER or expected['status'] not in ('ready','already_joint')):
-        raise ValueError('Unsupported repair plan')
-    if (not isinstance(expected['database'],dict) or not isinstance(expected['content_fingerprints'],dict)
-            or not isinstance(expected['editions'],list) or len(expected['editions']) != 2
-            or expected['target_aliases'] != sorted(MEMBERS)):
-        raise ValueError('Invalid repair plan structure')
-    for edition, year in zip(expected['editions'], YEARS):
-        if not isinstance(edition,dict) or edition.get('year') != year:
-            raise ValueError('Repair plan must name the two exact ordered years')
-        for key in ('id','year','from_venue_id','target_venue_id','submissions'):
-            if type(edition.get(key)) is not int or edition[key] <= 0:
-                raise ValueError('Repair plan IDs/counts must be positive integers')
-
-
-def apply_plan(c, expected):
-    validate_plan(expected)
-    actual = inspect(c)
-    if actual['content_fingerprints'] != expected['content_fingerprints'] or actual['target_aliases'] != expected['target_aliases']:
-        raise ValueError('Stale repair plan: protected content or aliases changed')
-    if len(expected['editions']) != 2:
-        raise ValueError('Exactly two editions required')
-    for old, now in zip(expected['editions'], actual['editions']):
-        if set(old) != set(now):
-            raise ValueError('Unexpected edition fields')
-        for key in ('id','year','track','target_venue_id','submissions','evidence_url','proceedings_key'):
-            if old[key] != now[key]:
-                raise ValueError('Stale repair plan: edition differs')
-        if now['from_venue_id'] != old['from_venue_id'] and now['from_venue_id'] != now['target_venue_id']:
-            raise ValueError('Edition membership differs from plan')
-    if actual['status'] == 'already_joint':
-        return {**actual, 'status':'already_joint', 'updated_editions':0}
-    for edition in actual['editions']:
-        if edition['from_venue_id'] == edition['target_venue_id']:
-            continue
-        changed = c.execute('''UPDATE editions SET venue_id=? WHERE id=? AND venue_id=? AND year=? AND track=?''',
-                            (edition['target_venue_id'],edition['id'],edition['from_venue_id'],edition['year'],edition['track']))
-        if changed.rowcount != 1:
-            raise ValueError('Concurrent edition change; entire repair must roll back')
-    after = inspect(c)
-    if after['status'] != 'already_joint' or after['content_fingerprints'] != actual['content_fingerprints']:
-        raise ValueError('Postcheck changed protected content; entire repair must roll back')
-    c.execute('INSERT INTO schema_migrations(name) VALUES(?) ON CONFLICT DO NOTHING', (MARKER,))
-    return {**after, 'status':'applied', 'updated_editions':sum(e['from_venue_id'] != e['target_venue_id'] for e in actual['editions'])}
-
-
-def database_identity(c, db):
-    if getattr(c, 'postgres', False):
-        row = c.execute('SELECT current_database(),current_schema()').fetchone()
-        return {'engine':'postgresql','database':row[0],'schema':row[1]}
-    return {'engine':'sqlite','path':str(Path(db).resolve())}
-
-
-def run(db, out, *, pg_service=None, pg_schema=None, plan=None, plan_sha256=None):
-    out = Path(out).resolve()
-    if not out.parent.is_dir():
-        raise ValueError('Report parent directory must already exist')
-    if out.exists():
-        raise ValueError('Preserve existing evidence report; choose a new output path')
-    expected = None
-    if plan:
-        raw = Path(plan).read_bytes()
-        if hashlib.sha256(raw).hexdigest() != plan_sha256:
-            raise ValueError('Repair plan hash differs before opening database')
-        expected = json.loads(raw)
-        validate_plan(expected)
-    elif plan_sha256:
-        raise ValueError('Plan hash requires an existing plan')
-    if db and out == Path(db).resolve():
-        raise ValueError('Report must not overwrite database')
-    writer = open_store(db, pg_service=pg_service, pg_schema=pg_schema) if expected is not None else None
-    c = writer.conn if writer else open_read_connection(db, pg_service=pg_service, pg_schema=pg_schema)
-    with closing(c):
-        try:
-            if expected is not None and not getattr(c, 'postgres', False):
-                c.commit(); c.execute('BEGIN IMMEDIATE')
-            identity = database_identity(c, db)
-            if expected is not None and expected['database'] != identity:
-                raise ValueError('Plan belongs to another database/schema')
-            report = apply_plan(c, expected) if expected is not None else inspect(c)
-            report['database'] = identity
-            if expected is not None:
-                report['plan_sha256'] = plan_sha256
-            if expected is not None:
-                c.commit()
-        except BaseException:
-            if expected is not None: c.rollback()
-            raise
-    out.write_text(json.dumps(report,ensure_ascii=False,indent=2)+'\n')
-    print(json.dumps({'status':report['status'],'editions':len(report['editions']),
-                      'updated_editions':report.get('updated_editions',0),'report':str(out)}))
-    return report
-
-
-def main(argv=None, *, prog=None):
-    parser = argparse.ArgumentParser(prog=prog); database_arguments(parser)
-    parser.add_argument('--out', required=True)
-    parser.add_argument('--plan', help='Apply an already reviewed plan; omitted means read-only preflight')
-    parser.add_argument('--plan-sha256', help='Required exact plan SHA256 when applying')
-    args = parser.parse_args(argv)
-    run(args.db,args.out,pg_service=args.pg_service,pg_schema=args.pg_schema,plan=args.plan,plan_sha256=args.plan_sha256)

tests/test_identifier_curation.py

@@ -0,0 +1,185 @@
+import copy
+import json
+from pathlib import Path
+import tempfile
+import unittest
+
+from paper_atlas.curation.identifiers import SCOPE, digest, json_bytes, prepare
+from paper_atlas.db.store import Store
+from paper_atlas.ingestion.records import ingest_file
+
+
+class IdentifierCurationTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        self.root = Path(self.tmp.name)
+        self.output = self.root / 'prepared'
+        self.row = {'source':'dblp','external_id':'conf/uai/example','url':'https://example.org/paper',
+                    'title':'A sufficiently long example research paper title','abstract':None,
+                    'authors':['Jane Doe','John Smith'],'venue':'UAI','year':2001,'track':'main',
+                    'status':'accepted','tags':[{'namespace':'dblp:topic','value':'version\u2029study\x00'}],
+                    'metadata':{'pages':'203-210','raw_title':'Original\u2029title'}}
+        self.evidence = self.root / 'evidence.json'
+        self.evidence.write_bytes(json_bytes({'url':'https://example.org/archive','comment':'Conference version'}))
+        self.path = self.root / 'records.jsonl'
+        self.raw = json_bytes(self.row)
+        self.path.write_bytes(self.raw)
+        self.manifest = {'source':'dblp','venue':'UAI','year':2001,'track':'main','count':1,
+                         'list_url':'https://example.org/list','file_sha256':digest(self.raw),
+                         'status':'partial','missing_abstract_count':1,'parser_version':2}
+        self.path.with_suffix('.manifest.json').write_bytes(json_bytes(self.manifest))
+        self.plan = {'format_version':1,'inputs':[self.input(self.path)],'assertions':[
+            {'source':'dblp','external_id':self.row['external_id'],'record_sha256':digest(self.raw),
+             'identifiers':{'arxiv':'1408.1485'},'evidence':[{
+                 'file':str(self.evidence),'sha256':digest(self.evidence.read_bytes()),
+                 'url':'https://example.org/archive','checked_at':'2026-11-08T08:00:00Z',
+                 'note':'Reviewed title, authors, publication and version evidence'}]}]}
+
+    def input(self, path):
+        return {'file':str(path),'sha256':digest(path.read_bytes()),
+                'manifest_sha256':digest(path.with_suffix('.manifest.json').read_bytes())}
+
+    def run_plan(self, plan=None):
+        path = self.root / 'plan.json'
+        path.write_bytes(json_bytes(self.plan if plan is None else plan))
+        return prepare(path, self.output)
+
+    def test_only_identifiers_and_provenance_change_and_raw_inputs_survive(self):
+        result = self.run_plan()
+        self.assertEqual(self.path.read_bytes(), self.raw)
+        prepared = json.loads((self.output / self.path.name).read_bytes())
+        provenance = prepared['metadata'].pop('identifier_curation')
+        self.assertEqual(prepared.pop('identifiers'), {'arxiv':'1408.1485'})
+        self.assertEqual(prepared, self.row)
+        self.assertEqual(provenance[0]['record_sha256'], digest(self.raw))
+        self.assertEqual((self.output / provenance[0]['evidence'][0]['file']).read_bytes(), self.evidence.read_bytes())
+        manifest = json.loads((self.output / 'records.manifest.json').read_bytes())
+        for key, value in self.manifest.items():
+            if key != 'file_sha256': self.assertEqual(manifest[key], value)
+        self.assertEqual(manifest['file_sha256'], digest((self.output / self.path.name).read_bytes()))
+        self.assertEqual(result['assertions_applied'], 1)
+        self.assertEqual(result['identifiers_added'], 1)
+        self.assertEqual(result['verification_scope'], SCOPE)
+        self.assertFalse(result['semantic_identity_verified'])
+        self.assertFalse(result['database_written'])
+        self.assertEqual(list(self.output.glob('*.jsonl')), [self.output / self.path.name])
+        with self.assertRaisesRegex(ValueError, 'already exist'): self.run_plan()
+
+    def test_late_invalid_file_publishes_nothing(self):
+        second = self.root / 'second.jsonl'
+        second.write_bytes(json_bytes({**self.row,'external_id':'second'}))
+        manifest = {**self.manifest,'file_sha256':digest(second.read_bytes()),'count':2}
+        second.with_suffix('.manifest.json').write_bytes(json_bytes(manifest))
+        self.plan['inputs'].append(self.input(second))
+        with self.assertRaisesRegex(ValueError, 'count mismatch'): self.run_plan()
+        self.assertFalse(self.output.exists())
+        self.assertFalse(list(self.root.glob('.identifier-curation-*')))
+        self.assertEqual(self.path.read_bytes(), self.raw)
+
+    def test_unasserted_rows_keep_exact_physical_bytes_and_doi_is_additive(self):
+        untouched = (json.dumps({**self.row,'external_id':'unselected'},ensure_ascii=False,indent=None) + '  ').encode()
+        self.path.write_bytes(self.raw + untouched)
+        self.path.with_suffix('.manifest.json').write_bytes(json_bytes({**self.manifest,
+            'file_sha256':digest(self.path.read_bytes()),'count':2,'missing_abstract_count':2}))
+        self.plan['inputs'] = [self.input(self.path)]
+        self.plan['assertions'][0]['identifiers']['doi'] = '10.1234/example'
+        result = self.run_plan()
+        with (self.output / self.path.name).open('rb') as stream:
+            first = json.loads(next(stream))
+            self.assertEqual(next(stream), untouched)
+        self.assertEqual(first['identifiers'], {'arxiv':'1408.1485','doi':'10.1234/example'})
+        self.assertEqual(result['identifiers_added'], 2)
+
+    def test_missing_duplicate_and_mismatched_assertions_fail_closed(self):
+        for mode in ('missing','duplicate','line_hash','duplicate_input'):
+            with self.subTest(mode=mode):
+                plan = copy.deepcopy(self.plan)
+                assertion = plan['assertions'][0]
+                if mode == 'missing': assertion['external_id'] = 'absent'
+                elif mode == 'duplicate': plan['assertions'].append(copy.deepcopy(assertion))
+                elif mode == 'line_hash': assertion['record_sha256'] = '0' * 64
+                else:
+                    other = self.root / 'duplicate.jsonl'
+                    other.write_bytes(self.raw)
+                    other.with_suffix('.manifest.json').write_bytes(json_bytes(self.manifest))
+                    plan['inputs'].append(self.input(other))
+                with self.assertRaises(ValueError): self.run_plan(plan)
+                self.assertFalse(self.output.exists())
+
+    def test_identifier_conflict_does_not_overwrite_and_same_value_is_auditable(self):
+        self.row['identifiers'] = {'arxiv':'cs/0307069'}
+        self.path.write_bytes(json_bytes(self.row))
+        self.manifest['file_sha256'] = digest(self.path.read_bytes())
+        self.path.with_suffix('.manifest.json').write_bytes(json_bytes(self.manifest))
+        self.plan['inputs'] = [self.input(self.path)]
+        self.plan['assertions'][0]['record_sha256'] = digest(self.path.read_bytes())
+        with self.assertRaisesRegex(ValueError, 'identifier conflict'): self.run_plan()
+        self.assertFalse(self.output.exists())
+        self.plan['assertions'][0]['identifiers']['arxiv'] = 'cs/0307069'
+        self.assertEqual(self.run_plan()['identifiers_added'], 0)
+
+    def test_unsupported_changes_identifiers_and_unbound_evidence_are_rejected(self):
+        for mode in ('title','bad_month','version','unknown_scheme','doi_case','evidence_hash','credentials','timestamp'):
+            with self.subTest(mode=mode):
+                plan = copy.deepcopy(self.plan)
+                assertion = plan['assertions'][0]
+                evidence = assertion['evidence'][0]
+                if mode == 'title': assertion['title'] = 'Rewrite forbidden'
+                elif mode == 'bad_month': assertion['identifiers'] = {'arxiv':'1413.12345'}
+                elif mode == 'version': assertion['identifiers'] = {'arxiv':'1408.1485v2'}
+                elif mode == 'unknown_scheme': assertion['identifiers'] = {'url':'https://example.org'}
+                elif mode == 'doi_case': assertion['identifiers'] = {'doi':'10.1234/UPPER'}
+                elif mode == 'evidence_hash': evidence['sha256'] = '0' * 64
+                elif mode == 'credentials': evidence['url'] = 'https://user:secret@example.org'
+                else: evidence['checked_at'] = '2026-11-08T08:00:00'
+                with self.assertRaises(ValueError): self.run_plan(plan)
+                self.assertFalse(self.output.exists())
+
+    def test_input_and_manifest_hashes_and_duplicate_json_keys_are_checked(self):
+        for mode in ('input','manifest','duplicate_json','invalid_evidence_json'):
+            with self.subTest(mode=mode):
+                plan = copy.deepcopy(self.plan)
+                if mode == 'input':
+                    plan['inputs'][0]['sha256'] = '0' * 64
+                elif mode == 'manifest':
+                    plan['inputs'][0]['manifest_sha256'] = '0' * 64
+                elif mode == 'duplicate_json':
+                    raw = self.raw[:-2] + b',"source":"dblp"}\n'
+                    self.path.write_bytes(raw)
+                    self.path.with_suffix('.manifest.json').write_bytes(json_bytes({**self.manifest,'file_sha256':digest(raw)}))
+                    plan['inputs'] = [self.input(self.path)]
+                else:
+                    self.evidence.write_bytes(b'not json')
+                    plan['assertions'][0]['evidence'][0]['sha256'] = digest(self.evidence.read_bytes())
+                with self.assertRaises(ValueError): self.run_plan(plan)
+                self.assertFalse(self.output.exists())
+                self.path.write_bytes(self.raw)
+                self.path.with_suffix('.manifest.json').write_bytes(json_bytes(self.manifest))
+
+    def test_curated_anchor_import_does_not_merge_extended_version_or_its_history(self):
+        db = self.root / 'papers.sqlite'
+        store = Store(db)
+        try:
+            original, _ = store.record('arxiv','1408.1485','https://example.org/one',self.row['title'],None,self.row['authors'],{'arxiv':'1408.1485'})
+            extended, record = store.record('arxiv','cs/0307069','https://example.org/two',self.row['title'],None,self.row['authors'],{'arxiv':'cs/0307069'})
+            store.decision(extended,record,'ICLR',2025,'main','another-attempt','https://example.org/reject','rejected','Reject')
+            store.conn.commit()
+        finally:
+            store.conn.close()
+        self.run_plan()
+        ingest_file(db,self.output / self.path.name)
+        store = Store(db)
+        try:
+            c = store.conn
+            self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0], 2)
+            self.assertEqual(c.execute('SELECT paper_id FROM source_records WHERE external_id=?',(self.row['external_id'],)).fetchone()[0], original)
+            self.assertEqual(c.execute('SELECT s.paper_id FROM submissions s JOIN editions e ON e.id=s.edition_id JOIN venues v ON v.id=e.venue_id WHERE v.acronym=?',('ICLR',)).fetchone()[0], extended)
+            self.assertEqual(c.execute('SELECT count(*) FROM paper_merges').fetchone()[0], 0)
+            self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(), [])
+        finally:
+            store.conn.close()
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_index_core.py

@@ -53,6 +53,50 @@ class StoreTests(unittest.TestCase):
         self.assertNotEqual(other,paper)
         self.assertEqual(self.store.conn.execute('SELECT count(*) FROM match_candidates').fetchone()[0],1)
 
+    def test_identifier_anchor_does_not_absorb_a_same_title_arxiv_version(self):
+        conference,first=self.record('arxiv','1408.1485')
+        extended,second=self.record('arxiv','cs/0307069')
+        self.store.decision(extended,second,'ICLR',2025,'main','separate','https://example.org/reject','rejected','Reject')
+        resolved,_=self.record('dblp','conf/uai/example',{'arxiv':'1408.1485'})
+        self.assertEqual(resolved,conference);self.assertNotEqual(conference,extended)
+        c=self.store.conn
+        self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0],2)
+        self.assertEqual(c.execute('SELECT paper_id FROM submissions').fetchone()[0],extended)
+        self.assertEqual(c.execute('SELECT count(*) FROM paper_merges').fetchone()[0],0)
+        self.assertEqual(c.execute('SELECT count(*) FROM match_candidates').fetchone()[0],1)
+        # A refresh of the anchored original source is equally unambiguous.
+        self.assertEqual(self.record('arxiv','1408.1485')[0],conference)
+        self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0],2)
+
+    def test_ambiguous_title_aborts_whole_file_without_merging_existing_entities(self):
+        self.record('arxiv','1408.1485');self.record('arxiv','cs/0307069')
+        self.store.conn.commit()
+        rows=[{'source':'dblp','external_id':'conf/uai/new','url':'https://example.org/new',
+               'title':'An unrelated earlier record in this file','abstract':None,'authors':[],
+               'venue':'UAI','year':2001,'track':'main','status':'accepted'},
+              {'source':'dblp','external_id':'conf/uai/ambiguous','url':'https://example.org/ambiguous',
+               'title':'A sufficiently long example research paper title','abstract':None,
+               'authors':['Jane Doe','John Smith'],'identifiers':{'doi':'10.0000/not-yet-known'},
+               'venue':'UAI','year':2001,'track':'main','status':'accepted'}]
+        path=Path(self.tmp.name)/'ambiguous.jsonl'
+        path.write_text(''.join(json.dumps(row)+'\n' for row in rows))
+        path.with_suffix('.manifest.json').write_text(json.dumps({'source':'dblp','list_url':'https://example.org/list','count':2}))
+        with self.assertRaisesRegex(ValueError,'Ambiguous paper identity'):ingest_file(self.db,path)
+        c=self.store.conn
+        for table,count in [('papers',2),('source_records',2),('submissions',0),('decisions',0),('paper_merges',0)]:
+            self.assertEqual(c.execute('SELECT count(*) FROM '+table).fetchone()[0],count,table)
+        self.assertEqual(c.execute('SELECT status FROM ingest_runs').fetchone()[0],'failed')
+
+    def test_explicit_bridge_merges_only_its_identifier_matches(self):
+        first,_=self.record('arxiv','2601.00001',authors=['First Author'])
+        second,_=self.record('arxiv','2601.00002',{'doi':'10.0000/bridge'},authors=['Second Author'])
+        unrelated,_=self.record('arxiv','2601.00003')
+        resolved,_=self.record('official','bridge',{'arxiv':'2601.00001','doi':'10.0000/bridge'})
+        self.assertEqual(resolved,min(first,second))
+        self.assertEqual(self.store.conn.execute('SELECT count(*) FROM papers').fetchone()[0],2)
+        self.assertIsNotNone(self.store.conn.execute('SELECT 1 FROM papers WHERE id=?',(unrelated,)).fetchone())
+        self.assertEqual(self.store.conn.execute('SELECT count(*) FROM paper_merges').fetchone()[0],1)
+
     def test_manifest_count_mismatch_rolls_back_records(self):
         path=Path(self.tmp.name)/'source.jsonl'
         path.write_text(json.dumps({'source':'pmlr','external_id':'v1/paper','url':'https://example.org/paper','title':'Paper','abstract':'Abstract','authors':[],'venue':'ICML','year':2023,'track':'main','status':'accepted'})+'\n')

tests/test_joint_editions.py

@@ -1,187 +0,0 @@
-from contextlib import closing, redirect_stdout
-import hashlib
-import io
-import json
-import os
-from pathlib import Path
-import tempfile
-import unittest
-from unittest.mock import patch
-import uuid
-
-from paper_atlas.collectors.dblp import XML_SHA
-from paper_atlas.collectors.dblp_scope import scope
-from paper_atlas.collectors.dblp_records import normalize_row
-from paper_atlas.db.store import Store
-from paper_atlas.db.backend import open_store
-from paper_atlas.ingestion.joint_editions import (TARGET, MARKER, inspect, apply_plan, run)
-from paper_atlas.postgres.client import connect
-from paper_atlas.postgres.migration import migrate
-from paper_atlas.postgres.schema import quote
-from paper_atlas import querying
-
-
-def seed(store):
-    c = store.conn
-    c.execute("INSERT INTO venues(acronym,name,field,venue_type) VALUES(?,?,'reasoning','conference')", (TARGET,TARGET))
-    target = c.execute('SELECT id FROM venues WHERE acronym=?',(TARGET,)).fetchone()[0]
-    for alias in ['ECAI','IJCAI']:
-        c.execute('INSERT INTO venue_aliases VALUES(?,?)',(target,alias))
-    for year in [2018,2019,2022]:
-        metadata = {'snapshot_sha256':XML_SHA,'proceedings_key':'conf/ijcai/'+str(year),
-                    'bibliographic_year':year,'bibliography':{'booktitle':['IJCAI']}}
-        p,r=store.record('dblp','conf/ijcai/fixture'+str(year),'https://dblp.org/rec/fixture',
-                         'Original publication '+str(year),'Abstract\x00with\x01data',[],
-                         {'doi':'10.24963/ijcai.'+str(year)+'/1'}, {})
-        store.record_tags(p,r,'dblp',[{'namespace':'dblp:booktitle','value':'IJCAI'}])
-        store.decision(p,r,'IJCAI',year,'main','fixture'+str(year),'https://example.org/decision','accepted','listed')
-        stored=json.loads(c.execute('SELECT metadata_json FROM source_records WHERE id=?',(r,)).fetchone()[0]);stored.update(metadata)
-        c.execute('UPDATE source_records SET metadata_json=? WHERE id=?',(json.dumps(stored),r))
-    c.commit()
-
-
-def plan(c):
-    return {**inspect(c),'database':{'engine':'sqlite','path':'fixture'}}
-
-
-class JointEditionTests(unittest.TestCase):
-    def setUp(self):
-        self.tmp=tempfile.TemporaryDirectory(); self.addCleanup(self.tmp.cleanup)
-        self.db=Path(self.tmp.name)/'sample.sqlite'
-        self.store=Store(self.db); self.addCleanup(self.store.conn.close)
-        seed(self.store); self.c=self.store.conn
-
-    def test_apply_preserves_every_protected_row_and_edition_id(self):
-        before=plan(self.c); ids=[r['id']for r in before['editions']]
-        result=apply_plan(self.c,before);self.c.commit()
-        self.assertEqual(result['updated_editions'],2)
-        self.assertEqual(result['content_fingerprints'],before['content_fingerprints'])
-        self.assertEqual([r['id']for r in result['editions']],ids)
-        untouched=self.c.execute('SELECT v.acronym FROM editions e JOIN venues v ON v.id=e.venue_id WHERE e.year=2019').fetchone()[0]
-        self.assertEqual(untouched,'IJCAI')
-        self.assertEqual(self.c.execute('PRAGMA foreign_key_check').fetchall(),[])
-        self.assertEqual(self.c.execute('SELECT count(*) FROM schema_migrations WHERE name=?',(MARKER,)).fetchone()[0],1)
-        with redirect_stdout(io.StringIO())as out:
-            querying.main(['--db',str(self.db),'--venue','ECAI'])
-        self.assertEqual(len(json.loads(out.getvalue())),2)
-        self.assertEqual(apply_plan(self.c,before)['updated_editions'],0)
-
-    def test_refresh_of_old_normalized_source_does_not_recreate_ordinary_edition(self):
-        before=plan(self.c);apply_plan(self.c,before)
-        for year in [2018,2022]:
-            row=self.c.execute('''SELECT r.paper_id,r.id,d.observed_at FROM source_records r
-                JOIN submissions s ON s.source_record_id=r.id JOIN decisions d ON d.submission_id=s.id
-                WHERE r.external_id=?''',('conf/ijcai/fixture'+str(year),)).fetchone()
-            self.store.decision(row[0],row[1],'IJCAI',year,'main','fixture'+str(year),
-                                'https://example.org/decision','accepted','listed',observed_at=row[2])
-        self.assertEqual(inspect(self.c)['content_fingerprints'],before['content_fingerprints'])
-        self.assertEqual(self.c.execute('SELECT count(*) FROM editions').fetchone()[0],3)
-
-    def test_any_changed_source_text_rejects_stale_plan_before_membership_updates(self):
-        before=plan(self.c)
-        self.c.execute("UPDATE source_records SET title='New evidence' WHERE external_id='conf/ijcai/fixture2018'")
-        with self.assertRaisesRegex(ValueError,'Stale repair plan'):
-            apply_plan(self.c,before)
-        self.assertEqual(inspect(self.c)['status'],'ready')
-
-    def test_conflicting_target_edition_is_never_merged_or_deleted(self):
-        target=self.c.execute('SELECT id FROM venues WHERE acronym=?',(TARGET,)).fetchone()[0]
-        self.c.execute("INSERT INTO editions(venue_id,year,track) VALUES(?,2018,'main')",(target,))
-        with self.assertRaisesRegex(ValueError,'colliding'):
-            inspect(self.c)
-        self.assertEqual(self.c.execute('SELECT count(*) FROM editions WHERE year=2018').fetchone()[0],2)
-
-    def test_wrong_source_volume_and_snapshot_fail_closed(self):
-        original=self.c.execute("SELECT metadata_json FROM source_records WHERE external_id='conf/ijcai/fixture2018'").fetchone()[0]
-        for key,value in [('proceedings_key','conf/ijcai/2018w'),('snapshot_sha256','wrong'),('bibliographic_year',2017),('bibliography',{'booktitle':['Other']})]:
-            m=json.loads(original);m[key]=value
-            self.c.execute("UPDATE source_records SET metadata_json=? WHERE external_id='conf/ijcai/fixture2018'",(json.dumps(m),))
-            with self.subTest(key=key),self.assertRaisesRegex(ValueError,'Source evidence'):
-                inspect(self.c)
-
-    def test_missing_source_support_and_wrong_aliases_are_not_guessed(self):
-        self.c.execute("DELETE FROM venue_aliases WHERE alias='ECAI'")
-        with self.assertRaisesRegex(ValueError,'aliases'):
-            inspect(self.c)
-        self.c.rollback()
-        self.c.execute('DELETE FROM submission_sources WHERE source_record_id=(SELECT id FROM source_records WHERE external_id=?)',('conf/ijcai/fixture2022',))
-        with self.assertRaisesRegex(ValueError,'Every submission'):
-            inspect(self.c)
-
-    def test_cli_plan_readonly_hash_binding_identity_and_repeat_apply(self):
-        out=Path(self.tmp.name)/'plan.json'
-        with redirect_stdout(io.StringIO()):run(self.db,out)
-        self.assertEqual(inspect(self.c)['status'],'ready')
-        raw=out.read_bytes();sha=hashlib.sha256(raw).hexdigest()
-        with patch('paper_atlas.ingestion.joint_editions.open_store')as writer:
-            with self.assertRaisesRegex(ValueError,'hash differs'):
-                run(self.db,Path(self.tmp.name)/'bad.json',plan=out,plan_sha256='0'*64)
-            writer.assert_not_called()
-        with redirect_stdout(io.StringIO()):
-            result=run(self.db,Path(self.tmp.name)/'applied.json',plan=out,plan_sha256=sha)
-            again=run(self.db,Path(self.tmp.name)/'retry.json',plan=out,plan_sha256=sha)
-        self.assertEqual(result['updated_editions'],2);self.assertEqual(again['updated_editions'],0)
-        self.assertEqual(result['plan_sha256'],sha);self.assertEqual(again['plan_sha256'],sha)
-        altered=json.loads(raw);altered['database']['path']='another.sqlite'
-        bad=Path(self.tmp.name)/'wrong-db.json';bad.write_text(json.dumps(altered))
-        with self.assertRaisesRegex(ValueError,'another database'):
-            run(self.db,Path(self.tmp.name)/'wrong-result.json',plan=bad,plan_sha256=hashlib.sha256(bad.read_bytes()).hexdigest())
-
-    def test_invalid_empty_plan_fails_before_opening_writer(self):
-        f=Path(self.tmp.name)/'empty.json';f.write_text('{}')
-        with patch('paper_atlas.ingestion.joint_editions.open_store')as writer:
-            with self.assertRaises(ValueError):
-                run(self.db,Path(self.tmp.name)/'empty-result.json',plan=f,plan_sha256=hashlib.sha256(f.read_bytes()).hexdigest())
-            writer.assert_not_called()
-
-    def test_collector_only_changes_the_two_exact_main_volume_keys(self):
-        for year in [2018,2019,2022]:
-            volume={'key':'conf/ijcai/'+str(year),'fields':{'title':['IJCAI '+str(year)],'year':[str(year)]}}
-            row={'record_type':'inproceedings','key':'conf/ijcai/One'+str(year),'fields':{'booktitle':['IJCAI'],
-                 'year':[str(year)],'crossref':[volume['key']],'title':['Original research title'],
-                 'author':['Jane Doe'],'ee':['https://doi.org/10.24963/ijcai.'+str(year)+'/1']}}
-            result,error=normalize_row(row,{volume['key']:volume},{'IJCAI','ECAI'})
-            self.assertIsNone(error)
-            self.assertEqual(result['venue'],TARGET if year in(2018,2022)else'IJCAI')
-            self.assertEqual(result['metadata']['bibliography']['booktitle'],['IJCAI'])
-            self.assertEqual(result['title'],'Original research title')
-            if year in(2018,2022):
-                self.assertIn('joint_edition_evidence',result['metadata'])
-                row['fields']['booktitle']=['IJCAI (Workshop)']
-                scoped,_=scope(row,volume)
-                self.assertEqual(scoped['venue'],'IJCAI')
-                self.assertEqual(scoped['track'],'workshop')
-
-
-@unittest.skipUnless(os.environ.get('PAPER_ATLAS_TEST_PG_SERVICE'),'Requires approved remote PostgreSQL service')
-class PostgreSQLJointTests(unittest.TestCase):
-    def setUp(self):
-        self.service=os.environ['PAPER_ATLAS_TEST_PG_SERVICE'];self.target='pa_joint_'+uuid.uuid4().hex[:12]
-        self.tmp=tempfile.TemporaryDirectory();self.addCleanup(self.tmp.cleanup)
-        self.db=Path(self.tmp.name)/'sample.sqlite';s=Store(self.db);seed(s);s.conn.close()
-        self.addCleanup(self.cleanup)
-        with redirect_stdout(io.StringIO()):migrate(self.db,self.service,Path(self.tmp.name)/'migration.json',schema=self.target,min_free_gib=0)
-
-    def cleanup(self):
-        with connect(self.service)as c:
-            for name, in c.execute('SELECT nspname FROM pg_namespace WHERE starts_with(nspname,%s)',(self.target,)).fetchall():
-                c.execute('DROP SCHEMA '+quote(name)+' CASCADE')
-
-    def test_native_repair_is_idempotent_keeps_codec_and_provenance(self):
-        p=Path(self.tmp.name)/'plan.json'
-        with redirect_stdout(io.StringIO()):
-            before=run(None,p,pg_service=self.service,pg_schema=self.target)
-            after=run(None,Path(self.tmp.name)/'applied.json',pg_service=self.service,pg_schema=self.target,plan=p,plan_sha256=hashlib.sha256(p.read_bytes()).hexdigest())
-        self.assertEqual(after['content_fingerprints'],before['content_fingerprints'])
-        with closing(open_store(pg_service=self.service,pg_schema=self.target).conn)as c:
-            self.assertEqual(apply_plan(c,before)['updated_editions'],0)
-            self.assertEqual(c.execute('SELECT abstract FROM papers ORDER BY id LIMIT 1').fetchone()[0],'Abstract\x00with\x01data')
-
-    def test_native_stale_source_plan_preserves_both_original_memberships(self):
-        with closing(open_store(pg_service=self.service,pg_schema=self.target).conn)as c:
-            before=plan(c)
-            c.execute("UPDATE source_records SET title='Changed' WHERE external_id=?",('conf/ijcai/fixture2022',));c.commit()
-            with self.assertRaisesRegex(ValueError,'Stale repair plan'):
-                apply_plan(c,before)
-            c.rollback()
-            self.assertEqual(inspect(c)['status'],'ready')

tests/test_postgres_ingestion.py

@@ -110,6 +110,34 @@ class PostgreSQLIngestionTests(unittest.TestCase):
         finally:
             reader.close()
 
+    def test_identity_anchor_keeps_other_arxiv_version_and_ambiguous_file_rolls_back(self):
+        title='An identical long research title shared by conference and extended versions'
+        authors=['Jane Doe','John Smith']
+        store=self.store()
+        try:
+            first,_=store.record('arxiv','1408.1485','https://arxiv.org/abs/1408.1485',title,'Conference abstract',authors)
+            second,_=store.record('arxiv','cs/0307069','https://arxiv.org/abs/cs/0307069',title,'Extended abstract',authors)
+            resolved,_=store.record('dblp','known','https://example.org/known',title,None,authors,{'arxiv':'1408.1485'})
+            self.assertEqual(resolved,first);self.assertNotEqual(first,second)
+            store.conn.commit()
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM papers').fetchone()[0],3)
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM paper_merges').fetchone()[0],1)  # migration fixture only
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM match_candidates').fetchone()[0],1)
+        finally:store.conn.close()
+        rows=[self.row('official','earlier',status='accepted',venue='UAI',year=2001),
+              dict(self.row('official','ambiguous',status='accepted',venue='UAI',year=2001),
+                   title=title,abstract=None,authors=authors,identifiers={})]
+        with self.assertRaisesRegex(ValueError,'Ambiguous paper identity'):self.ingest(self.file('ambiguous-identity',rows))
+        store=self.store()
+        try:
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM papers').fetchone()[0],3)
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM source_records').fetchone()[0],4)
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM submissions').fetchone()[0],1)
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM paper_merges').fetchone()[0],1)
+            self.assertEqual(store.conn.execute('SELECT status FROM ingest_runs ORDER BY id DESC LIMIT 1').fetchone()[0],'failed')
+            self.assertEqual(store.conn.execute("SELECT count(*) FROM source_records WHERE external_id IN (?,?)",('earlier','ambiguous')).fetchone()[0],0)
+        finally:store.conn.close()
+
     def test_file_import_is_idempotent_and_refreshes_only_own_tags(self):
         row = self.row(); row['tags'] = [{'namespace': 'openreview_snapshot:area', 'value': 'theory'}]
         path = self.file('reject', [row]); self.ingest(path)