Prepare evidence-bound identifier additions without rewriting source records

.gitea/workflows/check.yml

@@ -54,11 +54,10 @@ jobs:
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','backfill','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','audit','public-openreview','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','audit','histories','--help'],cwd=outside,check=True)
+              subprocess.run([python,'-I','-B','-m','paper_atlas','curate','identifiers','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-c','from paper_atlas.postgres.writer import PostgresStore, WriteConnection'],cwd=outside,check=True)
               smoke = "from paper_atlas.db.store import Store; from paper_atlas.config import DEFAULT_REGISTRY; import json; assert {'COLM','MM','MLSys'} <= {v['acronym'] for v in json.loads(DEFAULT_REGISTRY.read_text())['venues']}; s=Store('installed.sqlite'); s.conn.close()"
               subprocess.run([python,'-I','-B','-c',smoke],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','query','--db','installed.sqlite','--stats'],cwd=outside,check=True)
-              subprocess.run([python,'-I','-B','-m','paper_atlas','annotate','init','--db','installed.sqlite'],cwd=outside,check=True)
-              subprocess.run([python,'-I','-B','-m','paper_atlas','annotate','list','--db','installed.sqlite','--paper','1'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','audit','histories','--db','installed.sqlite','--examples','0'],cwd=outside,check=True)
           PY

README.md

@@ -86,5 +86,3 @@ paper-atlas collect colm --cache /data/paper-index/raw/http-cache \
 CI 在已授权的远程 runner 上运行测试、构建 wheel，并从源码目录外验证安装后的命令及随包资源。数据库全量去重和覆盖率是独立的数据验收步骤，不因代码测试通过而自动完成。
 
 开发通过工作分支和 PR 交付，由用户手动合并。
-
-研究类型与自定义标签可通过 [`annotate`](docs/annotations.md) 显式初始化、添加和撤回；两者独立查询，保留证据及完整更正历史，不从会议轨道推断研究类型。

docs/annotations.md

@@ -1,107 +0,0 @@
-# Research types and custom annotations
-
-Research type describes a claimed characteristic of the work (for example a survey,
-benchmark, empirical study or replication). It is independent of topic, paper length,
-conference track, presentation format and acceptance status. No type is inferred from
-those fields. Multiple source-specific or human classifications may coexist; no
-cross-platform vocabulary mapping or automatic classifier is provided.
-
-`annotation_events` stores both kinds with an explicit `kind` discriminator:
-`research_type` and `tag` are separate domains, even when namespace/value strings
-happen to coincide. Existing platform labels remain in `tags` / `paper_tags`.
-Each assertion carries namespace, value, actor, reason, evidence key, optional source
-record and evidence URL, and the time it was recorded. This is observation time, not
-paper publication time or a conference decision date.
-
-Values are not forced into a global taxonomy: `custom:method=empirical` and
-`platform:declared_type=empirical` retain separate provenance. An explicit platform
-assertion API requires a source record belonging to that paper and matching the
-namespace prefix. The manual CLI only accepts `custom:*`, and cannot retract platform
-assertions. No source type has been automatically assigned to existing papers.
-
-## Initialize explicitly
-
-On the published PostgreSQL dataset, using its protected service file:
-
-```sh
-PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas annotate init \
-  --pg-service foundry-research --pg-schema research
-```
-
-This creates one optional table, its indexes, and an upgrade marker in one transaction,
-using the existing dataset writer lock. It does not copy data, change existing papers
-or decisions, or create a PostgreSQL instance. Repeating it is safe; an unexplained
-pre-existing table/marker mismatch fails closed. Readers and ordinary imports do not
-install the extension. Legacy datasets remain readable. A SQLite **sample** uses
-`--db sample.sqlite`; the cutover SQLite backup remains protected against writes.
-
-## Add and retract evidence
-
-Use the canonical paper ID returned by `query`. For a reviewed classification:
-
-```sh
-paper-atlas annotate assert --pg-service foundry-research --pg-schema research \
-  --paper "$PAPER_ID" --kind research_type --namespace custom:method \
-  --value empirical --evidence-key manual:abstract-review \
-  --actor reviewer --reason 'Classified from the explicitly described study design' \
-  --request-id "$REQUEST_ID"
-```
-
-`$REQUEST_ID` must be a new, stable identifier for this action (for example a UUID).
-Retry with the same request ID and identical content after an interrupted response;
-it returns the original event ID. Reusing that ID with changed content is rejected.
-A topic annotation uses `--kind tag --namespace custom:topic --value robotics` instead.
-Optional `--source-record` and `--evidence-url` attach additional evidence; the source
-record must belong to this paper. URLs are retained as evidence and are not fetched.
-
-To withdraw one particular assertion, use its returned event ID:
-
-```sh
-paper-atlas annotate retract --pg-service foundry-research --pg-schema research \
-  --event "$EVENT_ID" --actor reviewer --reason 'Classification corrected after review' \
-  --request-id "$RETRACTION_REQUEST_ID"
-```
-
-Retraction adds an event referencing that assertion, preserving its original content.
-It does not remove another contributor's assertion of the same label, platform tags,
-source metadata or submission/decision history. A retracted assertion can be asserted
-again with a new request ID and evidence; it is a new observation. The API does not
-update or delete past annotation events. Database administrators still control direct
-SQL access. Work reviewed as the same paper may subsequently be merged by the existing
-identity workflow: annotation IDs, original paper IDs, evidence and retraction links
-are retained while the canonical paper reference follows the surviving entity.
-
-## Query and analyze
-
-```sh
-paper-atlas query --pg-service foundry-research --pg-schema research \
-  --research-type custom:method=empirical
-paper-atlas query --pg-service foundry-research --pg-schema research \
-  --tag custom:topic=robotics
-paper-atlas query --pg-service foundry-research --pg-schema research \
-  --paper "$PAPER_ID" --annotation-history
-paper-atlas annotate list --pg-service foundry-research --pg-schema research \
-  --paper "$PAPER_ID" --history
-```
-
-`--tag` includes original platform labels and active annotation tags. It never matches
-research types. `--research-type` matches active type assertions only. Paper details
-return separate `tags` (existing provenance), `annotation_tags`, and `research_types`;
-`--annotation-history` additionally includes retractions. A missing extension yields
-empty annotation results rather than changing the database.
-
-For direct SQL analysis, active research-type evidence is:
-
-```sql
-SELECT a.* FROM research.annotation_events a
-WHERE a.kind = 'research_type' AND a.retracts_id IS NULL
-  AND NOT EXISTS (SELECT 1 FROM research.annotation_events r WHERE r.retracts_id = a.id);
-```
-
-Several independent assertions can support one label; count distinct papers/labels
-when appropriate rather than counting evidence rows as papers. Namespaces stay
-separate. Standard paper queries decode the PostgreSQL text codec; direct SQL consumers
-must honor the dataset's documented codec, including NUL/SOH values. SQLite-to-PostgreSQL
-migration recognizes an installed annotation extension and verifies its full row hashes,
-constraints and identity sequence along with the original business tables. An old
-21-table snapshot stays a 21-table migration; no synthetic annotations are generated.

docs/usage.md

@@ -2,6 +2,55 @@
 
 先在已确认的远程环境安装项目，使用 `paper-atlas`。从源码运行时，将命令前缀替换为 `python3 -B -m paper_atlas`，其余参数相同。数据目录位于仓库外。
 
+## 按证据补充稳定标识
+
+`paper-atlas curate identifiers --plan /data/review/plan.json --output /data/review/prepared`
+只生成新的 JSONL、manifest、证据副本及 `curation-report.json`，不会连接数据库。
+输出目录必须不存在，父目录必须已存在；任何输入或断言失败，都不发布本批输出。
+原始文件不变，未选中的记录逐字节保留。题名、作者、年份、页码、投稿决定、标签和既有 metadata
+均不重写，仅补充 `identifiers` 并追加 `metadata.identifier_curation` 证据链。
+
+计划格式如下（哈希占位符须替换为真实 SHA256，所有路径按命令当前目录解析）：
+
+```json
+{
+  "format_version": 1,
+  "inputs": [{
+    "file": "/data/review/uai2001.jsonl",
+    "sha256": "<JSONL 文件 SHA256>",
+    "manifest_sha256": "<同名 .manifest.json 文件 SHA256>"
+  }],
+  "assertions": [{
+    "source": "dblp",
+    "external_id": "conf/uai/example",
+    "record_sha256": "<原始物理行字节 SHA256，包含该行原有换行符>",
+    "identifiers": {"arxiv": "1408.1485"},
+    "evidence": [{
+      "file": "/data/review/primary-metadata.json",
+      "sha256": "<证据 JSON 文件 SHA256>",
+      "url": "https://example.org/primary-record",
+      "checked_at": "2026-11-08T08:00:00Z",
+      "note": "说明人工核对的题名、作者、出版版本和稳定标识证据"
+    }]
+  }]
+}
+```
+
+输入 manifest 必须包含 `source/venue/year/track/count/list_url/file_sha256`，且与逐条记录一致。
+断言必须唯一、准确命中一条记录。仅支持无版本后缀的 arXiv 编号与小写规范 DOI；
+已有同类标识不允许覆盖，相同值可以补记证据。证据只接受不超过 16 MiB 的 JSON 元数据文件，
+来源 URL 要求 HTTPS 且不含凭据，核查时间必须带时区；不下载论文 PDF。
+
+`verification_scope=input_and_evidence_integrity_only` 和 `semantic_identity_verified=false`
+明确表示哈希校验不能证明两条记录属于同一作品。请先独立核对版本关系；仅年份、页码或相同题名
+不能作为合并依据。存在来源冲突的行应继续暂缓。新 manifest 保留原来的 partial 状态及摘要缺失数，
+不会把补充编号伪装成来源覆盖或摘要补全。证据副本按 SHA256 保存在输出目录 `evidence/`，
+原计划也保存在 `plan.json`，供之后复查。
+
+文件生成后仍需另做当前 PostgreSQL 身份、投稿和决定冲突预检；导入运行时必须包含
+“稳定标识优先、多个标题候选则整文件回滚”的身份保护，再以 `import records` 独立导入并验收。
+该命令不会自动导入或合并实体，也不代表最终全库去重通过。
+
 ## PostgreSQL 查询
 
 在远程环境安装 `.[postgres]`，通过权限为 `0600` 的 libpq service 文件连接，不把密码放入 URL、命令行或仓库。
@@ -99,6 +148,8 @@ paper-atlas query --pg-service foundry-research \
 
 `--stats` 中 papers 是当前规则下的论文实体数；source_records 是来源记录数；decisions 是决定观察数。一个投稿可能有多个来源和多条观察，不能相加当作论文数。
 
+已有稳定标识或原来源记录匹配时，该身份优先；同标题、同作者的另一 arXiv 存档或扩展稿不会仅因相似标题被并入。没有身份锚点时，作者佐证的精确标题只有一个候选才可匹配；多个候选会报 `Ambiguous paper identity`，整个采集文件回滚并记录失败运行。应补有证据的标识或先核查版本关系，不删除重复候选或任取第一个。一个新来源明确同时携带两个既有稳定标识时，仍可按既有规则归并并保留投稿、标签和合并日志；未归并的同标题关系继续保留为待核查候选。这些规则不证明全库去重完成。
+
 ## 5. 投稿历史关系审计
 
 ```sh

paper_atlas/cli.py

@@ -6,7 +6,7 @@ from paper_atlas import __version__
 
 
 COMMANDS = {
-    ('annotate',): ('paper_atlas.curation.annotations', {}, '管理独立的研究类型与自定义标签证据'),
+    ('curate','identifiers'): ('paper_atlas.curation.identifiers', {}, '按证据补充稳定标识并生成独立待导入文件'),
     ('query',): ('paper_atlas.querying', {}, '查询论文、标签和投稿历史'),
     ('coverage',): ('paper_atlas.audits.coverage', {}, '查看全部候选会议的覆盖情况'),
     ('download',): ('paper_atlas.io.snapshots', {}, '下载并校验固定版本快照'),
@@ -49,7 +49,7 @@ def main(argv=None):
             leaf=commands.add_parser(route[0],help=help_text,add_help=False)
         else:
             if route[0] not in groups:
-                group=commands.add_parser(route[0],help={'collect':'采集来源元数据','catalogue':'盘点历史目录','import':'导入数据库','maintain':'维护数据库','audit':'核验数据'}[route[0]])
+                group=commands.add_parser(route[0],help={'collect':'采集来源元数据','catalogue':'盘点历史目录','import':'导入数据库','maintain':'维护数据库','audit':'核验数据','curate':'按证据整理元数据'}[route[0]])
                 groups[route[0]]=group.add_subparsers(dest='operation',required=True)
             leaf=groups[route[0]].add_parser(route[1],help=help_text,add_help=False)
         leaf.set_defaults(handler=(route,module,kwargs))

paper_atlas/curation/__init__.py

@@ -1 +1 @@
-"""Evidence-bearing human and explicit source annotations, independent of outcomes."""
+"""Offline, evidence-bound preparation of metadata for separate ingestion."""

paper_atlas/curation/annotations.py

@@ -1,218 +0,0 @@
-"""Append-only tag/type assertions and explicit retractions; no inferred labels."""
-import argparse
-from contextlib import closing
-from importlib.resources import files
-import json
-import re
-from urllib.parse import urlparse
-
-from paper_atlas.db.backend import database_arguments, inserted_id, open_read_connection, open_store
-from paper_atlas.db.submissions import timestamp
-
-TABLE = 'annotation_events'
-MIGRATION = 'annotation_events_v1'
-COLUMNS = ('id', 'request_id', 'paper_id', 'original_paper_id', 'kind', 'namespace',
-           'value', 'source_record_id', 'evidence_key', 'evidence_url', 'reason',
-           'actor', 'observed_at', 'retracts_id')
-ACTIVE = '''a.retracts_id IS NULL AND NOT EXISTS
-    (SELECT 1 FROM annotation_events removed WHERE removed.retracts_id=a.id)'''
-
-
-def available(conn):
-    if getattr(conn, 'postgres', False):
-        if hasattr(conn, 'has_table'):
-            return conn.has_table(TABLE)
-        return bool(conn.execute("SELECT 1 FROM information_schema.tables WHERE table_schema=current_schema() AND table_name=?", (TABLE,)).fetchone())
-    return bool(conn.execute("SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (TABLE,)).fetchone())
-
-
-def _begin_write(conn):
-    if not getattr(conn, 'postgres', False) and not conn.in_transaction:
-        conn.execute('BEGIN IMMEDIATE')
-
-
-def initialize(conn):
-    """Explicit transactional schema extension; does not rescan or rewrite papers."""
-    _begin_write(conn)
-    exists = available(conn)
-    marker = conn.execute('SELECT 1 FROM schema_migrations WHERE name=?', (MIGRATION,)).fetchone()
-    if exists != bool(marker):
-        raise RuntimeError('Annotation table/upgrade marker mismatch; inspect before changing schema')
-    if exists:
-        result = conn.execute('SELECT * FROM annotation_events LIMIT 0')
-        description = result.cursor.description if getattr(conn, 'postgres', False) else result.description
-        if tuple(d[0] for d in description) != COLUMNS:
-            raise RuntimeError('Unexpected annotation schema')
-        return False
-    ddl = files('paper_atlas.db').joinpath('annotations.sql').read_text()
-    if getattr(conn, 'postgres', False):
-        ddl = re.sub(r'\bINTEGER\b', 'BIGINT', ddl)
-        ddl = re.sub(r'\bTEXT\b', 'TEXT COLLATE "C"', ddl)
-        ddl = ddl.replace('id BIGINT PRIMARY KEY', 'id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY', 1)
-    for statement in ddl.split(';'):
-        if statement.strip():
-            conn.execute(statement)
-    conn.execute('INSERT INTO schema_migrations(name) VALUES(?)', (MIGRATION,))
-    return True
-
-
-def _text(value, name, limit=512):
-    if not isinstance(value, str) or not value.strip() or len(value) > limit:
-        raise ValueError(name + ' must be nonempty text of at most ' + str(limit) + ' characters')
-    return value
-
-
-def _id(value, name):
-    if type(value) is not int or value <= 0:
-        raise ValueError(name + ' must be a positive integer')
-    return value
-
-
-def _url(value):
-    if value is not None:
-        _text(value, 'evidence_url', 4096)
-        parsed = urlparse(value)
-        if parsed.scheme not in ('https', 'http') or not parsed.hostname or parsed.username or parsed.password:
-            raise ValueError('evidence_url must be an HTTP(S) URL without credentials')
-    return value
-
-
-def _require(conn):
-    if not available(conn) or not conn.execute('SELECT 1 FROM schema_migrations WHERE name=?', (MIGRATION,)).fetchone():
-        raise RuntimeError('Run paper-atlas annotate init explicitly before adding annotations')
-
-
-def _row(row):
-    return dict(zip(COLUMNS, (row[i] for i in range(len(COLUMNS)))))
-
-
-def _replay(conn, request_id, payload):
-    old = conn.execute('SELECT * FROM annotation_events WHERE request_id=?', (request_id,)).fetchone()
-    if old is None:
-        return None
-    old = _row(old)
-    if any(old[k] != value for k, value in payload.items()):
-        raise ValueError('request_id already identifies different annotation content')
-    return old['id']
-
-
-def assert_annotation(conn, paper, kind, namespace, value, *, request_id, evidence_key,
-                      reason, actor, evidence_url=None, source_record_id=None):
-    _begin_write(conn)
-    _require(conn)
-    paper = _id(paper, 'paper')
-    if kind not in ('tag', 'research_type'):
-        raise ValueError('kind must be tag or research_type')
-    _text(namespace, 'namespace', 128)
-    if not re.fullmatch(r'[a-z][a-z0-9_]*:[a-z][a-z0-9_.-]*', namespace):
-        raise ValueError('namespace must have an explicit source:scope')
-    payload = dict(original_paper_id=paper, kind=kind, namespace=namespace,
-                   value=_text(value, 'value'), request_id=_text(request_id, 'request_id', 128),
-                   source_record_id=source_record_id, evidence_key=_text(evidence_key, 'evidence_key', 256),
-                   reason=_text(reason, 'reason', 8192), actor=_text(actor, 'actor', 256),
-                   evidence_url=_url(evidence_url), retracts_id=None)
-    if source_record_id is not None:
-        _id(source_record_id, 'source_record_id')
-    replay = _replay(conn, request_id, payload)
-    if replay is not None:
-        return replay
-    if not conn.execute('SELECT 1 FROM papers WHERE id=?', (paper,)).fetchone():
-        raise ValueError('Existing canonical paper required; no automatic identity merge')
-    prefix = namespace.split(':', 1)[0]
-    if prefix != 'custom' and source_record_id is None:
-        raise ValueError('Platform annotations require their own source record')
-    if source_record_id is not None:
-        source = conn.execute('''SELECT r.paper_id,s.name FROM source_records r
-            JOIN sources s ON s.id=r.source_id WHERE r.id=?''', (source_record_id,)).fetchone()
-        if not source or source[0] != paper or (prefix != 'custom' and prefix != source[1]):
-            raise ValueError('Annotation source record/namespace must belong to this paper and platform')
-    payload.update(paper_id=paper, observed_at=timestamp())
-    fields = tuple(payload)
-    return inserted_id(conn, 'INSERT INTO annotation_events(' + ','.join(fields) + ') VALUES(' +
-                       ','.join('?' for _ in fields) + ')', tuple(payload[k] for k in fields))
-
-
-def retract_annotation(conn, event, *, request_id, reason, actor, evidence_url=None, custom_only=False):
-    _begin_write(conn)
-    _require(conn)
-    event = _id(event, 'event')
-    payload = dict(request_id=_text(request_id, 'request_id', 128), retracts_id=event,
-                   reason=_text(reason, 'reason', 8192), actor=_text(actor, 'actor', 256), evidence_url=_url(evidence_url))
-    target = conn.execute('SELECT * FROM annotation_events WHERE id=?', (event,)).fetchone()
-    if target is None or target[-1] is not None:
-        raise ValueError('Retraction must reference an existing assertion')
-    target = _row(target)
-    if custom_only and not target['namespace'].startswith('custom:'):
-        raise ValueError('Manual CLI cannot retract platform evidence')
-    replay = _replay(conn, request_id, payload)
-    if replay is not None:
-        return replay
-    if conn.execute('SELECT 1 FROM annotation_events WHERE retracts_id=?', (event,)).fetchone():
-        raise ValueError('Assertion already retracted; its history is retained')
-    payload.update({k: target[k] for k in ('paper_id', 'original_paper_id', 'kind', 'namespace', 'value', 'source_record_id', 'evidence_key')})
-    payload['observed_at'] = timestamp()
-    fields = tuple(payload)
-    return inserted_id(conn, 'INSERT INTO annotation_events(' + ','.join(fields) + ') VALUES(' +
-                       ','.join('?' for _ in fields) + ')', tuple(payload[k] for k in fields))
-
-
-def observations(conn, paper, *, kind=None, history=False):
-    if not available(conn):
-        return []
-    where = ['a.paper_id=?']; params = [_id(paper, 'paper')]
-    if not history:
-        where.append(ACTIVE)
-    if kind is not None:
-        if kind not in ('tag', 'research_type'):
-            raise ValueError('Invalid annotation kind')
-        where.append('a.kind=?'); params.append(kind)
-    return [_row(r) for r in conn.execute('SELECT a.* FROM annotation_events a WHERE ' +
-                                         ' AND '.join(where) + ' ORDER BY a.id', params)]
-
-
-def main(argv=None, *, prog=None):
-    parser = argparse.ArgumentParser(prog=prog, description='显式初始化、追加或撤回带证据的自定义标签和研究类型')
-    actions = parser.add_subparsers(dest='action', required=True)
-    for action in ('init', 'assert', 'retract', 'list'):
-        p = actions.add_parser(action)
-        database_arguments(p)
-        if action in ('assert', 'retract'):
-            for field in ('request-id', 'reason', 'actor'):
-                p.add_argument('--' + field, required=True)
-            p.add_argument('--evidence-url')
-        if action in ('assert', 'list'):
-            p.add_argument('--paper', type=int, required=True)
-            p.add_argument('--kind', choices=('tag', 'research_type'), required=action == 'assert')
-        if action == 'assert':
-            for field in ('namespace', 'value', 'evidence-key'):
-                p.add_argument('--' + field, required=True)
-            p.add_argument('--source-record', type=int)
-        if action == 'retract':
-            p.add_argument('--event', type=int, required=True)
-        if action == 'list':
-            p.add_argument('--history', action='store_true')
-    args = parser.parse_args(argv)
-    if args.action == 'list':
-        with closing(open_read_connection(args.db, pg_service=args.pg_service, pg_schema=args.pg_schema)) as conn:
-            result = observations(conn, args.paper, kind=args.kind, history=args.history)
-    else:
-        if args.action == 'assert' and not args.namespace.startswith('custom:'):
-            parser.error('Manual CLI requires custom:* namespace; platform assertions use validated source APIs')
-        with closing(open_store(args.db, pg_service=args.pg_service, pg_schema=args.pg_schema).conn) as conn:
-            try:
-                if not getattr(conn, 'postgres', False):
-                    conn.commit()
-                    conn.execute('BEGIN IMMEDIATE')
-                if args.action == 'init':
-                    result = {'initialized': initialize(conn)}
-                else:
-                    kwargs = dict(request_id=args.request_id, reason=args.reason, actor=args.actor, evidence_url=args.evidence_url)
-                    event = (assert_annotation(conn, args.paper, args.kind, args.namespace, args.value,
-                              evidence_key=args.evidence_key, source_record_id=args.source_record, **kwargs)
-                             if args.action == 'assert' else retract_annotation(conn, args.event, custom_only=True, **kwargs))
-                    result = {'event_id': event}
-                conn.commit()
-            except BaseException:
-                conn.rollback()
-                raise
-    print(json.dumps(result, ensure_ascii=False, indent=2))

paper_atlas/curation/identifiers.py

@@ -0,0 +1,212 @@
+"""Add reviewed stable identifiers without rewriting bibliographic source fields.
+
+Hashes establish file integrity, not the truth of an identity assertion. Identity
+review and a fresh database preflight remain separate requirements.
+"""
+import argparse
+from datetime import datetime
+import hashlib
+import json
+from pathlib import Path
+import re
+import tempfile
+from urllib.parse import urlsplit
+
+
+SCOPE = 'input_and_evidence_integrity_only'
+MAX_METADATA_BYTES = 16 * 1024 * 1024
+
+
+def digest(data):
+    return hashlib.sha256(data).hexdigest()
+
+
+def object_keys(value, required, optional=()):
+    if not isinstance(value, dict) or not set(required) <= value.keys() or value.keys() - set(required) - set(optional):
+        raise ValueError('Missing or unsupported plan fields')
+
+
+def json_bytes(value):
+    return (json.dumps(value, ensure_ascii=False, separators=(',', ':'), allow_nan=False) + '\n').encode('utf-8')
+
+
+def decode(raw):
+    def unique(pairs):
+        result = {}
+        for key, value in pairs:
+            if key in result:
+                raise ValueError('Duplicate JSON key: ' + key)
+            result[key] = value
+        return result
+    return json.loads(raw, object_pairs_hook=unique,
+                      parse_constant=lambda value: (_ for _ in ()).throw(ValueError('Non-finite JSON value')))
+
+
+def metadata(path, expected=None):
+    with Path(path).open('rb') as stream:
+        raw = stream.read(MAX_METADATA_BYTES + 1)
+    if len(raw) > MAX_METADATA_BYTES:
+        raise ValueError('Metadata exceeds size limit')
+    if expected is not None:
+        check_hash(raw, expected)
+    return raw, decode(raw)
+
+
+def check_hash(raw, expected):
+    if not isinstance(expected, str) or not re.fullmatch('[0-9a-f]{64}', expected) or digest(raw) != expected:
+        raise ValueError('SHA256 mismatch')
+
+
+def identifier(scheme, value):
+    if not isinstance(value, str) or any(char.isspace() or ord(char) < 32 or ord(char) == 127 for char in value):
+        raise ValueError('Invalid identifier')
+    if scheme == 'arxiv':
+        pattern = r'(?:\d{2}(?:0[1-9]|1[0-2])\.\d{4,5}|[a-zA-Z][a-zA-Z.-]*/\d{2}(?:0[1-9]|1[0-2])\d{3})'
+        valid = re.fullmatch(pattern, value)
+    elif scheme == 'doi':
+        valid = re.fullmatch(r'10\.\d{4,9}/\S+', value) and value == value.lower()
+    else:
+        valid = False
+    if not valid:
+        raise ValueError('Unsupported or noncanonical identifier: ' + scheme)
+
+
+def prepare(plan_path, output):
+    plan_raw, plan = metadata(plan_path)
+    object_keys(plan, ('format_version', 'inputs', 'assertions'))
+    if type(plan['format_version']) is not int or plan['format_version'] != 1:
+        raise ValueError('Unsupported format version')
+    if not isinstance(plan['inputs'], list) or not plan['inputs'] or not isinstance(plan['assertions'], list) or not plan['assertions']:
+        raise ValueError('Inputs and assertions must be nonempty lists')
+    output = Path(output).absolute()
+    if output.exists():
+        raise ValueError('Output directory must not already exist')
+    if not output.parent.is_dir():
+        raise ValueError('Output parent directory must exist')
+    assertions = {}
+    evidence_files = {}
+    checked_evidence = {}
+    for assertion in plan['assertions']:
+        object_keys(assertion, ('source', 'external_id', 'record_sha256', 'identifiers', 'evidence'))
+        if any(not isinstance(assertion[key], str) or not assertion[key] for key in ('source', 'external_id')):
+            raise ValueError('Invalid source identity')
+        key = (assertion['source'], assertion['external_id'])
+        if key in assertions:
+            raise ValueError('Duplicate identity assertion')
+        ids = assertion['identifiers']
+        if not isinstance(ids, dict) or not ids:
+            raise ValueError('Identifiers must be a nonempty object')
+        for scheme, value in ids.items():
+            identifier(scheme, value)
+        if not isinstance(assertion['evidence'], list) or not assertion['evidence']:
+            raise ValueError('Evidence is required')
+        references = []
+        for evidence in assertion['evidence']:
+            object_keys(evidence, ('file', 'sha256', 'url', 'checked_at', 'note'))
+            if not all(isinstance(value, str) and value for value in evidence.values()):
+                raise ValueError('Invalid evidence fields')
+            url = urlsplit(evidence['url'])
+            if url.scheme != 'https' or not url.hostname or url.username or url.password:
+                raise ValueError('Evidence requires an HTTPS URL without credentials')
+            if datetime.fromisoformat(evidence['checked_at'].replace('Z', '+00:00')).utcoffset() is None:
+                raise ValueError('Evidence timestamp requires a timezone')
+            evidence_key = (str(Path(evidence['file']).absolute()), evidence['sha256'])
+            if evidence_key not in checked_evidence:
+                checked_evidence[evidence_key] = metadata(evidence['file'], evidence['sha256'])[0]
+            raw = checked_evidence[evidence_key]
+            evidence_files[evidence['sha256']] = raw
+            references.append({**evidence, 'file': 'evidence/' + evidence['sha256'] + '.json'})
+        assertions[key] = (assertion, references)
+    seen = set()
+    applied = set()
+    names = set()
+    report = {'format_version': 1, 'plan_sha256': digest(plan_raw),
+              'verification_scope': SCOPE, 'semantic_identity_verified': False,
+              'database_written': False, 'files': [], 'assertions_applied': 0, 'identifiers_added': 0}
+    with tempfile.TemporaryDirectory(prefix='.identifier-curation-', dir=output.parent) as temporary:
+        staging = Path(temporary) / 'prepared'
+        staging.mkdir()
+        (staging / 'evidence').mkdir()
+        (staging / 'plan.json').write_bytes(plan_raw)
+        for sha, raw in evidence_files.items():
+            (staging / 'evidence' / (sha + '.json')).write_bytes(raw)
+        for entry in plan['inputs']:
+            object_keys(entry, ('file', 'sha256', 'manifest_sha256'))
+            path = Path(entry['file'])
+            if path.suffix != '.jsonl' or path.name in names:
+                raise ValueError('Inputs require distinct JSONL basenames')
+            names.add(path.name)
+            _, manifest = metadata(path.with_suffix('.manifest.json'), entry['manifest_sha256'])
+            if not isinstance(manifest, dict) or not {'source','venue','year','track','count','list_url','file_sha256'} <= manifest.keys():
+                raise ValueError('Complete source manifest required')
+            if manifest['file_sha256'] != entry['sha256']:
+                raise ValueError('Manifest and plan hashes differ')
+            before = hashlib.sha256()
+            after = hashlib.sha256()
+            count = 0
+            with path.open('rb') as source, (staging / path.name).open('wb') as target:
+                for raw in source:
+                    before.update(raw)
+                    row = decode(raw)
+                    if any(row[field] != manifest[field] for field in ('source', 'venue', 'year', 'track')):
+                        raise ValueError('Record differs from source manifest')
+                    key = (row['source'], row['external_id'])
+                    if key in seen:
+                        raise ValueError('Duplicate source identity in input batch')
+                    seen.add(key)
+                    if key in assertions:
+                        assertion, references = assertions[key]
+                        check_hash(raw, assertion['record_sha256'])
+                        existing = row.get('identifiers', {})
+                        if existing is None:
+                            existing = {}
+                        if not isinstance(existing, dict):
+                            raise ValueError('Invalid existing identifiers')
+                        for scheme, value in assertion['identifiers'].items():
+                            if scheme in existing and existing[scheme] != value:
+                                raise ValueError('Existing identifier conflict')
+                        details = row.setdefault('metadata', {})
+                        if not isinstance(details, dict):
+                            raise ValueError('Invalid record metadata')
+                        history = details.setdefault('identifier_curation', [])
+                        if not isinstance(history, list):
+                            raise ValueError('Invalid curation history')
+                        history.append({'plan_sha256': digest(plan_raw), 'input_file_sha256': entry['sha256'],
+                                        'record_sha256': digest(raw), 'previous_identifiers': dict(existing),
+                                        'identifiers_asserted': assertion['identifiers'], 'evidence': references,
+                                        'verification_scope': SCOPE})
+                        report['identifiers_added'] += len(assertion['identifiers'].keys() - existing.keys())
+                        row['identifiers'] = {**existing, **assertion['identifiers']}
+                        raw = json_bytes(row)
+                        applied.add(key)
+                    target.write(raw)
+                    after.update(raw)
+                    count += 1
+            if before.hexdigest() != entry['sha256'] or not re.fullmatch('[0-9a-f]{64}', entry['sha256']):
+                raise ValueError('Input SHA256 mismatch')
+            if not count or type(manifest['count']) is not int or count != manifest['count']:
+                raise ValueError('Manifest count mismatch')
+            provenance = {'plan_sha256': digest(plan_raw), 'input_file_sha256': before.hexdigest(),
+                          'input_manifest_sha256': entry['manifest_sha256'], 'verification_scope': SCOPE}
+            prepared = {**manifest, 'file': str(output / path.name), 'file_sha256': after.hexdigest(),
+                        'identifier_curation': provenance}
+            (staging / path.with_suffix('.manifest.json').name).write_bytes(json_bytes(prepared))
+            report['files'].append({'file': str(output / path.name), 'count': count,
+                                    'input_sha256': before.hexdigest(), 'sha256': after.hexdigest()})
+        if applied != assertions.keys():
+            raise ValueError('Assertion target absent from input batch')
+        report['assertions_applied'] = len(applied)
+        (staging / 'curation-report.json').write_bytes(json_bytes(report))
+        # All files are on the same filesystem. No database or input mutation.
+        if output.exists():
+            raise ValueError('Output directory appeared during preparation')
+        staging.rename(output)
+    return report
+
+
+def main(argv=None, *, prog=None):
+    parser = argparse.ArgumentParser(prog=prog, description=__doc__)
+    parser.add_argument('--plan', required=True, type=Path)
+    parser.add_argument('--output', required=True, type=Path)
+    args = parser.parse_args(argv)
+    print(json.dumps(prepare(args.plan, args.output), ensure_ascii=False))

paper_atlas/db/annotations.sql

@@ -1,15 +0,0 @@
-CREATE TABLE annotation_events (
- id INTEGER PRIMARY KEY,
- request_id TEXT NOT NULL UNIQUE,
- paper_id INTEGER NOT NULL REFERENCES papers(id),
- original_paper_id INTEGER NOT NULL,
- kind TEXT NOT NULL CHECK(kind IN ('tag','research_type')),
- namespace TEXT NOT NULL, value TEXT NOT NULL,
- source_record_id INTEGER REFERENCES source_records(id),
- evidence_key TEXT NOT NULL, evidence_url TEXT,
- reason TEXT NOT NULL, actor TEXT NOT NULL, observed_at TEXT NOT NULL,
- retracts_id INTEGER UNIQUE REFERENCES annotation_events(id),
- CHECK(retracts_id IS NULL OR retracts_id<>id)
-);
-CREATE INDEX annotation_events_paper ON annotation_events(paper_id,kind,id);
-CREATE INDEX annotation_events_label ON annotation_events(kind,namespace,value,paper_id);

paper_atlas/db/store.py

@@ -55,9 +55,6 @@ class Store:
         c.execute('UPDATE identifiers SET paper_id=? WHERE paper_id=?',(target,other))
         c.execute('UPDATE source_records SET paper_id=? WHERE paper_id=?',(target,other))
         c.execute('UPDATE submissions SET paper_id=? WHERE paper_id=?',(target,other))
-        from paper_atlas.curation.annotations import available
-        if available(c):
-            c.execute('UPDATE annotation_events SET paper_id=? WHERE paper_id=?', (target,other))
         c.execute('INSERT INTO paper_tags SELECT ?,tag_id,source_record_id,evidence_key FROM paper_tags WHERE paper_id=? ON CONFLICT DO NOTHING',(target,other))
         c.execute('DELETE FROM paper_tags WHERE paper_id=?',(other,))
         c.execute('UPDATE paper_merges SET surviving_paper_id=? WHERE surviving_paper_id=?',(target,other))
@@ -98,13 +95,19 @@ class Store:
             matches.add(old[0])
         key = normalized(title)
         title_matches = c.execute('SELECT id,authors_json,abstract FROM papers WHERE title_key=?',(key,)).fetchall()
-        authors_set = author_keys(authors)
-        for paper,old_authors,old_abstract in title_matches:
-            overlap = authors_set & author_keys(json.loads(old_authors))
-            # Cross-platform title matching needs independent author corroboration.
-            same_source = c.execute('SELECT 1 FROM source_records WHERE paper_id=? AND source_id=? LIMIT 1',(paper,source_id)).fetchone()
-            if not same_source and len(key)>=25 and (len(overlap)>=2 or (overlap and abstract and abstract.strip()==(old_abstract or '').strip())):
-                matches.add(paper)
+        # An existing identifier/source record anchors the entity. Title
+        # similarity must not pull in another deposit or an extended version.
+        if not matches:
+            title_only_matches = set()
+            authors_set = author_keys(authors)
+            for paper,old_authors,old_abstract in title_matches:
+                overlap = authors_set & author_keys(json.loads(old_authors))
+                same_source = c.execute('SELECT 1 FROM source_records WHERE paper_id=? AND source_id=? LIMIT 1',(paper,source_id)).fetchone()
+                if not same_source and len(key)>=25 and (len(overlap)>=2 or (overlap and abstract and abstract.strip()==(old_abstract or '').strip())):
+                    title_only_matches.add(paper)
+            if len(title_only_matches)>1:
+                raise ValueError('Ambiguous paper identity: multiple author-corroborated title matches; provide verified identifiers')
+            matches = title_only_matches
         if matches:
             paper = min(matches)
             for other in sorted(matches-{paper}):

paper_atlas/postgres/migration.py

@@ -143,7 +143,7 @@ def migrate(db, service, out, *, schema='research', storage_path=None, min_free_
     free_space(storage_path, min_free_gib)
     control = schema + '_migration'
     report = {'status': 'preflight', 'source': str(db), 'service': service, 'schema': schema,
-              'started_at': timestamp(), 'tables': {}, 'scope': 'All baseline business tables and recognized installed extensions; FTS5 rebuilt as native PostgreSQL GIN.'}
+              'started_at': timestamp(), 'tables': {}, 'scope': 'All 21 business tables; FTS5 internals rebuilt as native PostgreSQL GIN.'}
     with Path(str(db) + '.maintenance.lock').open('a') as lock:
         fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
         if Path(str(db) + '-wal').exists() and Path(str(db) + '-wal').stat().st_size:
@@ -240,7 +240,7 @@ def migrate(db, service, out, *, schema='research', storage_path=None, min_free_
                     JOIN pg_am am ON am.oid=ix.relam WHERE ns.nspname=%s AND ix.relname='papers_search_gin' ''', (stage,)).fetchone()
                 if search != ('gin', True, True, 'papers'):
                     raise ValueError('Native full-text index is not valid and ready')
-                for table in plan:
+                for table in TABLES:
                     pg.execute('ANALYZE ' + qualify(stage, table))
                 invalid = pg.execute('''SELECT count(*) FROM pg_constraint co JOIN pg_class cl ON cl.oid=co.conrelid
                     JOIN pg_namespace ns ON ns.oid=cl.relnamespace WHERE ns.nspname=%s AND NOT co.convalidated''', (stage,)).fetchone()[0]

paper_atlas/postgres/schema.py

@@ -8,9 +8,7 @@ TABLES = ('sources', 'papers', 'identifiers', 'source_records', 'venues', 'editi
           'submission_sources', 'submission_identifiers', 'decision_sources',
           'submission_merges', 'decision_merges', 'schema_migrations', 'tags',
           'paper_tags', 'match_candidates', 'paper_merges', 'ingest_runs')
-OPTIONAL_TABLES = ('annotation_events',)
 CHECKS = {
-    'annotation_events': ["kind IN ('tag','research_type')", 'retracts_id IS NULL OR retracts_id<>id'],
     'venues': ["venue_type IN ('conference','journal','review_platform')"],
     'edition_sources': ["status IN ('pending','running','complete','partial','blocked','unavailable','not_yet_published')"],
     'decisions': ["status IN ('accepted','rejected','desk_rejected','withdrawn','under_review','unknown')"],
@@ -52,10 +50,10 @@ def quote(name):
 def inspect(connection):
     business = {name for schema, name, kind, *_ in connection.execute('PRAGMA table_list')
                 if schema == 'main' and kind == 'table' and not name.startswith('sqlite_')}
-    if not set(TABLES) <= business or business - set(TABLES) - set(OPTIONAL_TABLES):
+    if business != set(TABLES):
         raise ValueError('Unexpected business tables: ' + repr(sorted(business ^ set(TABLES))))
     result = {}
-    for table in (*TABLES, *(name for name in OPTIONAL_TABLES if name in business)):
+    for table in TABLES:
         columns = [{'name': r[1], 'type': r[2].upper(), 'notnull': bool(r[3]),
                     'default': r[4], 'pk': r[5]}
                    for r in connection.execute('PRAGMA table_info(' + quote(table) + ')')]

paper_atlas/querying.py

@@ -29,8 +29,6 @@ def main(argv=None, *, prog=None):
     parser.add_argument('--pg-schema', help='Optional PostgreSQL schema override; otherwise use the configured search_path')
     parser.add_argument('--query', help='Title/abstract search: FTS5 for SQLite, web-search syntax for PostgreSQL')
     parser.add_argument('--paper',type=int,help='Paper ID; include sources, tags and submission history')
-    parser.add_argument('--research-type', help='Independent type filter as namespace=value; no type inferred from track')
-    parser.add_argument('--annotation-history', action='store_true', help='Include retraction evidence for --paper')
     parser.add_argument('--tag', help='Filter as namespace=value, e.g. arxiv:category=cs.AI')
     parser.add_argument('--venue', help='Conference acronym')
     parser.add_argument('--year', type=int)
@@ -49,8 +47,6 @@ def main(argv=None, *, prog=None):
         conn = sqlite3.connect(Path(args.db).resolve().as_uri()+'?mode=ro', uri=True)
         conn.row_factory = sqlite3.Row
         conn.execute('BEGIN')  # One consistent read snapshot while ingestion continues.
-    from paper_atlas.curation.annotations import available, observations, ACTIVE
-    has_annotations = available(conn)
     if args.stats:
         result = {name:conn.execute('SELECT count(*) FROM '+name).fetchone()[0] for name in ('papers','source_records','submissions','decisions','tags','match_candidates')}
         result['records_by_source'] = [dict(r) for r in conn.execute('SELECT s.name,count(r.id) AS imported_records FROM sources s LEFT JOIN source_records r ON r.source_id=s.id GROUP BY s.id ORDER BY s.name')]
@@ -73,24 +69,8 @@ def main(argv=None, *, prog=None):
                 where.append('p.id IN (SELECT rowid FROM papers_fts WHERE papers_fts MATCH ?)')
             params.append(args.query)
         if args.tag:
-            if '=' not in args.tag:
-                parser.error('--tag requires namespace=value')
             namespace,value = args.tag.split('=',1)
-            platform = 'EXISTS(SELECT 1 FROM paper_tags pt JOIN tags t ON t.id=pt.tag_id WHERE pt.paper_id=p.id AND t.namespace=? AND t.value=?)'
-            params.extend([namespace,value])
-            if has_annotations:
-                platform = '(' + platform + ' OR EXISTS(SELECT 1 FROM annotation_events a WHERE a.paper_id=p.id AND a.kind=\'tag\' AND a.namespace=? AND a.value=? AND ' + ACTIVE + '))'
-                params.extend([namespace,value])
-            where.append(platform)
-        if args.research_type:
-            if '=' not in args.research_type:
-                parser.error('--research-type requires namespace=value')
-            namespace,value = args.research_type.split('=',1)
-            if has_annotations:
-                where.append('EXISTS(SELECT 1 FROM annotation_events a WHERE a.paper_id=p.id AND a.kind=\'research_type\' AND a.namespace=? AND a.value=? AND ' + ACTIVE + ')')
-                params.extend([namespace,value])
-            else:
-                where.append('1=0')
+            where.append('EXISTS(SELECT 1 FROM paper_tags pt JOIN tags t ON t.id=pt.tag_id WHERE pt.paper_id=p.id AND t.namespace=? AND t.value=?)'); params.extend([namespace,value])
         if args.outcome:
             where.append('p.id '+('IN' if args.outcome=='accepted' else 'NOT IN')+' ('+ACCEPTED_PAPERS+')')
         if args.venue or args.year or args.status or args.track:
@@ -118,10 +98,6 @@ def main(argv=None, *, prog=None):
             paper['known_acceptances']=known_acceptances(conn,row['id'])
             paper['publication_outcome']='accepted' if paper['known_acceptances'] else 'no_known_acceptance'
             if args.paper is not None:
-                paper['research_types'] = observations(conn, row['id'], kind='research_type')
-                paper['annotation_tags'] = observations(conn, row['id'], kind='tag')
-                if args.annotation_history:
-                    paper['annotation_history'] = observations(conn, row['id'], history=True)
                 paper['tags'] = [dict(r) for r in conn.execute('SELECT t.namespace,t.value,pt.evidence_key FROM paper_tags pt JOIN tags t ON t.id=pt.tag_id WHERE pt.paper_id=?',(row['id'],))]
                 paper['sources'] = [dict(r) for r in conn.execute('SELECT s.name,r.external_id,r.url,r.title,r.abstract,r.retrieved_at FROM source_records r JOIN sources s ON s.id=r.source_id WHERE r.paper_id=?',(row['id'],))]
                 submissions = []

pyproject.toml

@@ -21,5 +21,5 @@ include = ["paper_atlas*"]
 exclude = ["tests*"]
 
 [tool.setuptools.package-data]
-"paper_atlas.db" = ["schema.sql", "annotations.sql"]
+"paper_atlas.db" = ["schema.sql"]
 "paper_atlas.resources" = ["venues.json"]

tests/test_annotations.py

@@ -1,227 +0,0 @@
-from contextlib import closing, redirect_stdout
-import io
-import json
-import os
-from pathlib import Path
-import tempfile
-import unittest
-import uuid
-
-from paper_atlas import querying
-from paper_atlas.curation.annotations import (assert_annotation, available, initialize, main,
-                                             observations, retract_annotation)
-from paper_atlas.db.backend import open_store
-from paper_atlas.db.store import Store
-from paper_atlas.postgres import schema
-from paper_atlas.postgres.client import connect
-from paper_atlas.postgres.migration import migrate
-
-
-class AnnotationTests(unittest.TestCase):
-    def setUp(self):
-        self.tmp = tempfile.TemporaryDirectory(); self.addCleanup(self.tmp.cleanup)
-        self.db = Path(self.tmp.name) / 'sample.sqlite'
-        self.store = Store(self.db); self.addCleanup(self.store.conn.close)
-        self.c = self.store.conn
-        self.paper, self.record = self.store.record('fixture', 'one', 'https://example.org/one', 'First research', None, [])
-        self.other, self.other_record = self.store.record('fixture', 'two', 'https://example.org/two', 'Other research', None, [])
-        self.store.record_tags(self.paper, self.record, 'fixture', [{'namespace':'fixture:topic','value':'original'}])
-        self.c.commit()
-
-    def add(self, **changes):
-        args = dict(paper=self.paper, kind='tag', namespace='custom:topic', value='reading',
-                    request_id='request-one', evidence_key='manual:reading', reason='Reviewed metadata', actor='reviewer')
-        args.update(changes)
-        return assert_annotation(self.c, **args)
-
-    def query(self, *args):
-        self.c.commit()
-        with redirect_stdout(io.StringIO()) as output:
-            querying.main(['--db', str(self.db), *map(str, args)])
-        return json.loads(output.getvalue())
-
-    def test_read_does_not_install_extension_and_write_requires_explicit_init(self):
-        self.assertFalse(available(self.c))
-        self.assertEqual(observations(self.c, self.paper), [])
-        with self.assertRaisesRegex(RuntimeError, 'init explicitly'):
-            self.add()
-        self.assertEqual(self.query('--research-type', 'custom:method=empirical'), [])
-        self.assertFalse(available(self.c))
-        self.assertEqual(len(schema.inspect(self.c)), 21)
-
-    def test_schema_upgrade_is_transactional_and_marker_mismatch_fails_closed(self):
-        self.assertTrue(initialize(self.c)); self.c.rollback()
-        self.assertFalse(available(self.c))
-        self.c.execute("INSERT INTO schema_migrations(name) VALUES('annotation_events_v1')")
-        with self.assertRaisesRegex(RuntimeError, 'mismatch'):
-            initialize(self.c)
-        self.c.rollback()
-        self.assertTrue(initialize(self.c)); self.assertFalse(initialize(self.c))
-        self.assertEqual(len(schema.inspect(self.c)), 22)
-
-    def test_unmarked_table_cannot_receive_assertions(self):
-        initialize(self.c)
-        self.c.execute("DELETE FROM schema_migrations WHERE name='annotation_events_v1'")
-        with self.assertRaisesRegex(RuntimeError, 'init explicitly'):
-            self.add()
-        self.assertEqual(self.c.execute('SELECT count(*) FROM annotation_events').fetchone()[0], 0)
-
-    def test_types_tags_and_platform_labels_are_independent_and_queryable(self):
-        initialize(self.c)
-        self.add()
-        self.add(kind='research_type', namespace='custom:topic', request_id='type', value='reading')
-        self.assertEqual(len(self.query('--tag', 'custom:topic=reading')), 1)
-        self.assertEqual(len(self.query('--research-type', 'custom:topic=reading')), 1)
-        self.assertEqual(len(self.query('--tag', 'fixture:topic=original')), 1)
-        detail = self.query('--paper', self.paper)[0]
-        self.assertEqual(len(detail['annotation_tags']), 1)
-        self.assertEqual(len(detail['research_types']), 1)
-        self.assertNotIn('annotation_history', detail)
-        self.assertEqual(detail['tags'][0]['namespace'], 'fixture:topic')
-
-    def test_repeated_request_is_idempotent_and_changed_content_fails(self):
-        initialize(self.c); event = self.add()
-        self.assertEqual(self.add(), event)
-        for change in ({'value':'different'}, {'kind':'research_type'}, {'reason':'changed'}, {'paper':self.other}):
-            with self.assertRaisesRegex(ValueError, 'different annotation'):
-                self.add(**change)
-        self.assertEqual(self.c.execute('SELECT count(*) FROM annotation_events').fetchone()[0], 1)
-
-    def test_retraction_keeps_original_and_other_assertions_of_same_label(self):
-        initialize(self.c); first = self.add(); second = self.add(request_id='second', evidence_key='independent')
-        kwargs = dict(request_id='retract', reason='No longer relevant', actor='reviewer')
-        removed = retract_annotation(self.c, first, **kwargs)
-        self.assertEqual(retract_annotation(self.c, first, **kwargs), removed)
-        self.assertEqual([r['id'] for r in observations(self.c, self.paper)], [second])
-        self.assertEqual(len(self.query('--paper', self.paper, '--annotation-history')[0]['annotation_history']), 3)
-        with self.assertRaisesRegex(ValueError, 'already retracted'):
-            retract_annotation(self.c, first, **{**kwargs,'request_id':'duplicate'})
-        with self.assertRaisesRegex(ValueError, 'existing assertion'):
-            retract_annotation(self.c, removed, **{**kwargs,'request_id':'nested'})
-
-    def test_source_assertions_require_matching_paper_and_platform(self):
-        initialize(self.c)
-        for change in ({'namespace':'fixture:type'}, {'namespace':'other:type','source_record_id':self.record},
-                       {'namespace':'fixture:type','source_record_id':self.other_record}):
-            with self.assertRaises(ValueError):
-                self.add(**change)
-        event = self.add(kind='research_type', namespace='fixture:declared_type', source_record_id=self.record)
-        with self.assertRaisesRegex(ValueError, 'Manual CLI'):
-            retract_annotation(self.c, event, request_id='cannot', reason='x', actor='y', custom_only=True)
-        self.assertEqual(observations(self.c, self.paper)[0]['source_record_id'], self.record)
-
-    def test_platform_tag_refresh_preserves_both_custom_kinds(self):
-        initialize(self.c); self.add(); self.add(kind='research_type', request_id='type')
-        self.store.record_tags(self.paper, self.record, 'fixture', [{'namespace':'fixture:topic','value':'refreshed'}])
-        self.assertEqual(len(observations(self.c, self.paper)), 2)
-        self.assertEqual(len(self.query('--tag','fixture:topic=refreshed')), 1)
-        self.assertEqual(self.query('--tag','fixture:topic=original'), [])
-
-    def test_merge_preserves_retractions_evidence_and_original_paper_ids(self):
-        initialize(self.c); event = self.add(source_record_id=self.record)
-        removed = retract_annotation(self.c, event, request_id='withdraw', reason='corrected', actor='reviewer')
-        active = self.add(request_id='active', kind='research_type', value='survey')
-        self.store.merge(self.other, self.paper, 'Independently reviewed identity')
-        rows = observations(self.c, self.other, history=True)
-        self.assertEqual([r['id'] for r in rows], [event, removed, active])
-        self.assertEqual({r['original_paper_id'] for r in rows}, {self.paper})
-        self.assertEqual({r['paper_id'] for r in rows}, {self.other})
-        self.assertEqual([r['id'] for r in observations(self.c, self.other)], [active])
-        self.assertEqual(self.add(source_record_id=self.record), event)  # exact retry survives canonical merge
-        self.assertEqual(self.c.execute('PRAGMA foreign_key_check').fetchall(), [])
-
-    def test_invalid_input_does_not_create_events(self):
-        initialize(self.c)
-        for change in ({'paper':True}, {'paper':999999}, {'kind':'main'}, {'namespace':'arxiv'},
-                       {'namespace':'custom:topic;DROP'}, {'value':' '}, {'reason':''},
-                       {'actor':''}, {'request_id':''}, {'evidence_key':''}, {'source_record_id':True},
-                       {'evidence_url':'file:///etc/passwd'}, {'evidence_url':'https://user:secret@example.org/'}):
-            with self.subTest(change=change), self.assertRaises(ValueError):
-                self.add(**change)
-        self.assertEqual(self.c.execute('SELECT count(*) FROM annotation_events').fetchone()[0], 0)
-
-    def test_cli_manual_scope_and_complete_assert_retract_lifecycle(self):
-        with redirect_stdout(io.StringIO()):
-            main(['init','--db',str(self.db)])
-            main(['assert','--db',str(self.db),'--paper',str(self.paper),'--kind','research_type',
-                  '--namespace','custom:method','--value','empirical','--evidence-key','manual:review',
-                  '--request-id','cli-one','--reason','Read abstract','--actor','reviewer'])
-        self.assertEqual(len(self.query('--research-type','custom:method=empirical')), 1)
-        event = observations(self.c, self.paper)[0]['id']
-        with redirect_stdout(io.StringIO()):
-            main(['retract','--db',str(self.db),'--event',str(event),'--request-id','cli-two',
-                  '--reason','Corrected classification','--actor','reviewer'])
-        self.assertEqual(self.query('--research-type','custom:method=empirical'), [])
-        with redirect_stdout(io.StringIO()) as out:
-            main(['list','--db',str(self.db),'--paper',str(self.paper),'--history'])
-        self.assertEqual(len(json.loads(out.getvalue())), 2)
-
-
-@unittest.skipUnless(os.environ.get('PAPER_ATLAS_TEST_PG_SERVICE'), 'Requires approved remote PostgreSQL service')
-class PostgreSQLAnnotationTests(unittest.TestCase):
-    def setUp(self):
-        self.service = os.environ['PAPER_ATLAS_TEST_PG_SERVICE']
-        self.target = 'pa_anno_' + uuid.uuid4().hex[:12]
-        self.tmp = tempfile.TemporaryDirectory(); self.addCleanup(self.tmp.cleanup)
-        self.db = Path(self.tmp.name) / 'source.sqlite'
-        self.s = Store(self.db)
-        self.paper, self.record = self.s.record('fixture','one','https://example.org','Paper',None,[])
-        self.s.conn.commit()
-        self.addCleanup(self.s.conn.close); self.addCleanup(self.cleanup)
-
-    def cleanup(self):
-        with connect(self.service) as c:
-            for name, in c.execute('SELECT nspname FROM pg_namespace WHERE starts_with(nspname,%s)', (self.target,)):
-                c.execute('DROP SCHEMA ' + schema.quote(name) + ' CASCADE')
-
-    def migrate(self):
-        self.s.conn.commit()
-        self.s.conn.close()  # Freeze/checkpoint the fixture, just like a real source migration.
-        with redirect_stdout(io.StringIO()):
-            return migrate(self.db, self.service, Path(self.tmp.name)/'report.json', schema=self.target, min_free_gib=0)
-
-    def args(self, **changes):
-        a = dict(paper=self.paper,kind='research_type',namespace='custom:method',value='survey\x00and\x01',
-                 request_id='native',evidence_key='manual',reason='Reviewed\x00evidence',actor='reviewer')
-        a.update(changes); return a
-
-    def test_native_schema_upgrade_roundtrip_retraction_and_queries(self):
-        self.migrate()
-        with closing(open_store(pg_service=self.service, pg_schema=self.target).conn) as c:
-            self.assertTrue(initialize(c)); self.assertFalse(initialize(c))
-            event = assert_annotation(c, **self.args()); c.commit()
-            self.assertEqual(assert_annotation(c, **self.args()), event)
-            self.assertEqual(observations(c,self.paper)[0]['value'], 'survey\x00and\x01')
-            c.commit()
-        with redirect_stdout(io.StringIO()) as out:
-            querying.main(['--pg-service',self.service,'--pg-schema',self.target,'--paper',str(self.paper)])
-        self.assertEqual(json.loads(out.getvalue())[0]['research_types'][0]['id'], event)
-        with closing(open_store(pg_service=self.service,pg_schema=self.target).conn) as c:
-            retract_annotation(c,event,request_id='removed',reason='Changed',actor='reviewer');c.commit()
-            self.assertEqual(observations(c,self.paper), [])
-            self.assertEqual(len(observations(c,self.paper,history=True)), 2)
-
-    def test_initialized_sqlite_extension_migrates_all_rows_hashes_and_identity(self):
-        initialize(self.s.conn)
-        event = assert_annotation(self.s.conn, **self.args())
-        retract_annotation(self.s.conn,event,request_id='retracted',reason='Corrected',actor='reviewer')
-        report = self.migrate()
-        self.assertEqual(len(report['tables']),22)
-        self.assertEqual(report['tables']['annotation_events']['source_rows'],2)
-        self.assertEqual(report['tables']['annotation_events']['source_sha256'],report['tables']['annotation_events']['target_sha256'])
-        with closing(open_store(pg_service=self.service,pg_schema=self.target).conn) as c:
-            new = assert_annotation(c, **self.args(request_id='after-migration'))
-            self.assertGreater(new,2); c.commit()
-            self.assertEqual(len(observations(c,self.paper,history=True)),3)
-
-    def test_native_upgrade_rollback_keeps_legacy_schema_usable(self):
-        self.migrate()
-        with closing(open_store(pg_service=self.service,pg_schema=self.target).conn) as c:
-            initialize(c); c.rollback()
-            self.assertFalse(available(c))
-            self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0],1)
-            self.assertTrue(initialize(c))
-            with self.assertRaises(ValueError):
-                assert_annotation(c, **self.args(namespace='wrong:type',source_record_id=self.record))
-            self.assertEqual(c.execute('SELECT count(*) FROM annotation_events').fetchone()[0],0)
-            c.rollback()

tests/test_identifier_curation.py

@@ -0,0 +1,185 @@
+import copy
+import json
+from pathlib import Path
+import tempfile
+import unittest
+
+from paper_atlas.curation.identifiers import SCOPE, digest, json_bytes, prepare
+from paper_atlas.db.store import Store
+from paper_atlas.ingestion.records import ingest_file
+
+
+class IdentifierCurationTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        self.root = Path(self.tmp.name)
+        self.output = self.root / 'prepared'
+        self.row = {'source':'dblp','external_id':'conf/uai/example','url':'https://example.org/paper',
+                    'title':'A sufficiently long example research paper title','abstract':None,
+                    'authors':['Jane Doe','John Smith'],'venue':'UAI','year':2001,'track':'main',
+                    'status':'accepted','tags':[{'namespace':'dblp:topic','value':'version\u2029study\x00'}],
+                    'metadata':{'pages':'203-210','raw_title':'Original\u2029title'}}
+        self.evidence = self.root / 'evidence.json'
+        self.evidence.write_bytes(json_bytes({'url':'https://example.org/archive','comment':'Conference version'}))
+        self.path = self.root / 'records.jsonl'
+        self.raw = json_bytes(self.row)
+        self.path.write_bytes(self.raw)
+        self.manifest = {'source':'dblp','venue':'UAI','year':2001,'track':'main','count':1,
+                         'list_url':'https://example.org/list','file_sha256':digest(self.raw),
+                         'status':'partial','missing_abstract_count':1,'parser_version':2}
+        self.path.with_suffix('.manifest.json').write_bytes(json_bytes(self.manifest))
+        self.plan = {'format_version':1,'inputs':[self.input(self.path)],'assertions':[
+            {'source':'dblp','external_id':self.row['external_id'],'record_sha256':digest(self.raw),
+             'identifiers':{'arxiv':'1408.1485'},'evidence':[{
+                 'file':str(self.evidence),'sha256':digest(self.evidence.read_bytes()),
+                 'url':'https://example.org/archive','checked_at':'2026-11-08T08:00:00Z',
+                 'note':'Reviewed title, authors, publication and version evidence'}]}]}
+
+    def input(self, path):
+        return {'file':str(path),'sha256':digest(path.read_bytes()),
+                'manifest_sha256':digest(path.with_suffix('.manifest.json').read_bytes())}
+
+    def run_plan(self, plan=None):
+        path = self.root / 'plan.json'
+        path.write_bytes(json_bytes(self.plan if plan is None else plan))
+        return prepare(path, self.output)
+
+    def test_only_identifiers_and_provenance_change_and_raw_inputs_survive(self):
+        result = self.run_plan()
+        self.assertEqual(self.path.read_bytes(), self.raw)
+        prepared = json.loads((self.output / self.path.name).read_bytes())
+        provenance = prepared['metadata'].pop('identifier_curation')
+        self.assertEqual(prepared.pop('identifiers'), {'arxiv':'1408.1485'})
+        self.assertEqual(prepared, self.row)
+        self.assertEqual(provenance[0]['record_sha256'], digest(self.raw))
+        self.assertEqual((self.output / provenance[0]['evidence'][0]['file']).read_bytes(), self.evidence.read_bytes())
+        manifest = json.loads((self.output / 'records.manifest.json').read_bytes())
+        for key, value in self.manifest.items():
+            if key != 'file_sha256': self.assertEqual(manifest[key], value)
+        self.assertEqual(manifest['file_sha256'], digest((self.output / self.path.name).read_bytes()))
+        self.assertEqual(result['assertions_applied'], 1)
+        self.assertEqual(result['identifiers_added'], 1)
+        self.assertEqual(result['verification_scope'], SCOPE)
+        self.assertFalse(result['semantic_identity_verified'])
+        self.assertFalse(result['database_written'])
+        self.assertEqual(list(self.output.glob('*.jsonl')), [self.output / self.path.name])
+        with self.assertRaisesRegex(ValueError, 'already exist'): self.run_plan()
+
+    def test_late_invalid_file_publishes_nothing(self):
+        second = self.root / 'second.jsonl'
+        second.write_bytes(json_bytes({**self.row,'external_id':'second'}))
+        manifest = {**self.manifest,'file_sha256':digest(second.read_bytes()),'count':2}
+        second.with_suffix('.manifest.json').write_bytes(json_bytes(manifest))
+        self.plan['inputs'].append(self.input(second))
+        with self.assertRaisesRegex(ValueError, 'count mismatch'): self.run_plan()
+        self.assertFalse(self.output.exists())
+        self.assertFalse(list(self.root.glob('.identifier-curation-*')))
+        self.assertEqual(self.path.read_bytes(), self.raw)
+
+    def test_unasserted_rows_keep_exact_physical_bytes_and_doi_is_additive(self):
+        untouched = (json.dumps({**self.row,'external_id':'unselected'},ensure_ascii=False,indent=None) + '  ').encode()
+        self.path.write_bytes(self.raw + untouched)
+        self.path.with_suffix('.manifest.json').write_bytes(json_bytes({**self.manifest,
+            'file_sha256':digest(self.path.read_bytes()),'count':2,'missing_abstract_count':2}))
+        self.plan['inputs'] = [self.input(self.path)]
+        self.plan['assertions'][0]['identifiers']['doi'] = '10.1234/example'
+        result = self.run_plan()
+        with (self.output / self.path.name).open('rb') as stream:
+            first = json.loads(next(stream))
+            self.assertEqual(next(stream), untouched)
+        self.assertEqual(first['identifiers'], {'arxiv':'1408.1485','doi':'10.1234/example'})
+        self.assertEqual(result['identifiers_added'], 2)
+
+    def test_missing_duplicate_and_mismatched_assertions_fail_closed(self):
+        for mode in ('missing','duplicate','line_hash','duplicate_input'):
+            with self.subTest(mode=mode):
+                plan = copy.deepcopy(self.plan)
+                assertion = plan['assertions'][0]
+                if mode == 'missing': assertion['external_id'] = 'absent'
+                elif mode == 'duplicate': plan['assertions'].append(copy.deepcopy(assertion))
+                elif mode == 'line_hash': assertion['record_sha256'] = '0' * 64
+                else:
+                    other = self.root / 'duplicate.jsonl'
+                    other.write_bytes(self.raw)
+                    other.with_suffix('.manifest.json').write_bytes(json_bytes(self.manifest))
+                    plan['inputs'].append(self.input(other))
+                with self.assertRaises(ValueError): self.run_plan(plan)
+                self.assertFalse(self.output.exists())
+
+    def test_identifier_conflict_does_not_overwrite_and_same_value_is_auditable(self):
+        self.row['identifiers'] = {'arxiv':'cs/0307069'}
+        self.path.write_bytes(json_bytes(self.row))
+        self.manifest['file_sha256'] = digest(self.path.read_bytes())
+        self.path.with_suffix('.manifest.json').write_bytes(json_bytes(self.manifest))
+        self.plan['inputs'] = [self.input(self.path)]
+        self.plan['assertions'][0]['record_sha256'] = digest(self.path.read_bytes())
+        with self.assertRaisesRegex(ValueError, 'identifier conflict'): self.run_plan()
+        self.assertFalse(self.output.exists())
+        self.plan['assertions'][0]['identifiers']['arxiv'] = 'cs/0307069'
+        self.assertEqual(self.run_plan()['identifiers_added'], 0)
+
+    def test_unsupported_changes_identifiers_and_unbound_evidence_are_rejected(self):
+        for mode in ('title','bad_month','version','unknown_scheme','doi_case','evidence_hash','credentials','timestamp'):
+            with self.subTest(mode=mode):
+                plan = copy.deepcopy(self.plan)
+                assertion = plan['assertions'][0]
+                evidence = assertion['evidence'][0]
+                if mode == 'title': assertion['title'] = 'Rewrite forbidden'
+                elif mode == 'bad_month': assertion['identifiers'] = {'arxiv':'1413.12345'}
+                elif mode == 'version': assertion['identifiers'] = {'arxiv':'1408.1485v2'}
+                elif mode == 'unknown_scheme': assertion['identifiers'] = {'url':'https://example.org'}
+                elif mode == 'doi_case': assertion['identifiers'] = {'doi':'10.1234/UPPER'}
+                elif mode == 'evidence_hash': evidence['sha256'] = '0' * 64
+                elif mode == 'credentials': evidence['url'] = 'https://user:secret@example.org'
+                else: evidence['checked_at'] = '2026-11-08T08:00:00'
+                with self.assertRaises(ValueError): self.run_plan(plan)
+                self.assertFalse(self.output.exists())
+
+    def test_input_and_manifest_hashes_and_duplicate_json_keys_are_checked(self):
+        for mode in ('input','manifest','duplicate_json','invalid_evidence_json'):
+            with self.subTest(mode=mode):
+                plan = copy.deepcopy(self.plan)
+                if mode == 'input':
+                    plan['inputs'][0]['sha256'] = '0' * 64
+                elif mode == 'manifest':
+                    plan['inputs'][0]['manifest_sha256'] = '0' * 64
+                elif mode == 'duplicate_json':
+                    raw = self.raw[:-2] + b',"source":"dblp"}\n'
+                    self.path.write_bytes(raw)
+                    self.path.with_suffix('.manifest.json').write_bytes(json_bytes({**self.manifest,'file_sha256':digest(raw)}))
+                    plan['inputs'] = [self.input(self.path)]
+                else:
+                    self.evidence.write_bytes(b'not json')
+                    plan['assertions'][0]['evidence'][0]['sha256'] = digest(self.evidence.read_bytes())
+                with self.assertRaises(ValueError): self.run_plan(plan)
+                self.assertFalse(self.output.exists())
+                self.path.write_bytes(self.raw)
+                self.path.with_suffix('.manifest.json').write_bytes(json_bytes(self.manifest))
+
+    def test_curated_anchor_import_does_not_merge_extended_version_or_its_history(self):
+        db = self.root / 'papers.sqlite'
+        store = Store(db)
+        try:
+            original, _ = store.record('arxiv','1408.1485','https://example.org/one',self.row['title'],None,self.row['authors'],{'arxiv':'1408.1485'})
+            extended, record = store.record('arxiv','cs/0307069','https://example.org/two',self.row['title'],None,self.row['authors'],{'arxiv':'cs/0307069'})
+            store.decision(extended,record,'ICLR',2025,'main','another-attempt','https://example.org/reject','rejected','Reject')
+            store.conn.commit()
+        finally:
+            store.conn.close()
+        self.run_plan()
+        ingest_file(db,self.output / self.path.name)
+        store = Store(db)
+        try:
+            c = store.conn
+            self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0], 2)
+            self.assertEqual(c.execute('SELECT paper_id FROM source_records WHERE external_id=?',(self.row['external_id'],)).fetchone()[0], original)
+            self.assertEqual(c.execute('SELECT s.paper_id FROM submissions s JOIN editions e ON e.id=s.edition_id JOIN venues v ON v.id=e.venue_id WHERE v.acronym=?',('ICLR',)).fetchone()[0], extended)
+            self.assertEqual(c.execute('SELECT count(*) FROM paper_merges').fetchone()[0], 0)
+            self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(), [])
+        finally:
+            store.conn.close()
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_index_core.py

@@ -53,6 +53,50 @@ class StoreTests(unittest.TestCase):
         self.assertNotEqual(other,paper)
         self.assertEqual(self.store.conn.execute('SELECT count(*) FROM match_candidates').fetchone()[0],1)
 
+    def test_identifier_anchor_does_not_absorb_a_same_title_arxiv_version(self):
+        conference,first=self.record('arxiv','1408.1485')
+        extended,second=self.record('arxiv','cs/0307069')
+        self.store.decision(extended,second,'ICLR',2025,'main','separate','https://example.org/reject','rejected','Reject')
+        resolved,_=self.record('dblp','conf/uai/example',{'arxiv':'1408.1485'})
+        self.assertEqual(resolved,conference);self.assertNotEqual(conference,extended)
+        c=self.store.conn
+        self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0],2)
+        self.assertEqual(c.execute('SELECT paper_id FROM submissions').fetchone()[0],extended)
+        self.assertEqual(c.execute('SELECT count(*) FROM paper_merges').fetchone()[0],0)
+        self.assertEqual(c.execute('SELECT count(*) FROM match_candidates').fetchone()[0],1)
+        # A refresh of the anchored original source is equally unambiguous.
+        self.assertEqual(self.record('arxiv','1408.1485')[0],conference)
+        self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0],2)
+
+    def test_ambiguous_title_aborts_whole_file_without_merging_existing_entities(self):
+        self.record('arxiv','1408.1485');self.record('arxiv','cs/0307069')
+        self.store.conn.commit()
+        rows=[{'source':'dblp','external_id':'conf/uai/new','url':'https://example.org/new',
+               'title':'An unrelated earlier record in this file','abstract':None,'authors':[],
+               'venue':'UAI','year':2001,'track':'main','status':'accepted'},
+              {'source':'dblp','external_id':'conf/uai/ambiguous','url':'https://example.org/ambiguous',
+               'title':'A sufficiently long example research paper title','abstract':None,
+               'authors':['Jane Doe','John Smith'],'identifiers':{'doi':'10.0000/not-yet-known'},
+               'venue':'UAI','year':2001,'track':'main','status':'accepted'}]
+        path=Path(self.tmp.name)/'ambiguous.jsonl'
+        path.write_text(''.join(json.dumps(row)+'\n' for row in rows))
+        path.with_suffix('.manifest.json').write_text(json.dumps({'source':'dblp','list_url':'https://example.org/list','count':2}))
+        with self.assertRaisesRegex(ValueError,'Ambiguous paper identity'):ingest_file(self.db,path)
+        c=self.store.conn
+        for table,count in [('papers',2),('source_records',2),('submissions',0),('decisions',0),('paper_merges',0)]:
+            self.assertEqual(c.execute('SELECT count(*) FROM '+table).fetchone()[0],count,table)
+        self.assertEqual(c.execute('SELECT status FROM ingest_runs').fetchone()[0],'failed')
+
+    def test_explicit_bridge_merges_only_its_identifier_matches(self):
+        first,_=self.record('arxiv','2601.00001',authors=['First Author'])
+        second,_=self.record('arxiv','2601.00002',{'doi':'10.0000/bridge'},authors=['Second Author'])
+        unrelated,_=self.record('arxiv','2601.00003')
+        resolved,_=self.record('official','bridge',{'arxiv':'2601.00001','doi':'10.0000/bridge'})
+        self.assertEqual(resolved,min(first,second))
+        self.assertEqual(self.store.conn.execute('SELECT count(*) FROM papers').fetchone()[0],2)
+        self.assertIsNotNone(self.store.conn.execute('SELECT 1 FROM papers WHERE id=?',(unrelated,)).fetchone())
+        self.assertEqual(self.store.conn.execute('SELECT count(*) FROM paper_merges').fetchone()[0],1)
+
     def test_manifest_count_mismatch_rolls_back_records(self):
         path=Path(self.tmp.name)/'source.jsonl'
         path.write_text(json.dumps({'source':'pmlr','external_id':'v1/paper','url':'https://example.org/paper','title':'Paper','abstract':'Abstract','authors':[],'venue':'ICML','year':2023,'track':'main','status':'accepted'})+'\n')

tests/test_postgres_ingestion.py

@@ -110,6 +110,34 @@ class PostgreSQLIngestionTests(unittest.TestCase):
         finally:
             reader.close()
 
+    def test_identity_anchor_keeps_other_arxiv_version_and_ambiguous_file_rolls_back(self):
+        title='An identical long research title shared by conference and extended versions'
+        authors=['Jane Doe','John Smith']
+        store=self.store()
+        try:
+            first,_=store.record('arxiv','1408.1485','https://arxiv.org/abs/1408.1485',title,'Conference abstract',authors)
+            second,_=store.record('arxiv','cs/0307069','https://arxiv.org/abs/cs/0307069',title,'Extended abstract',authors)
+            resolved,_=store.record('dblp','known','https://example.org/known',title,None,authors,{'arxiv':'1408.1485'})
+            self.assertEqual(resolved,first);self.assertNotEqual(first,second)
+            store.conn.commit()
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM papers').fetchone()[0],3)
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM paper_merges').fetchone()[0],1)  # migration fixture only
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM match_candidates').fetchone()[0],1)
+        finally:store.conn.close()
+        rows=[self.row('official','earlier',status='accepted',venue='UAI',year=2001),
+              dict(self.row('official','ambiguous',status='accepted',venue='UAI',year=2001),
+                   title=title,abstract=None,authors=authors,identifiers={})]
+        with self.assertRaisesRegex(ValueError,'Ambiguous paper identity'):self.ingest(self.file('ambiguous-identity',rows))
+        store=self.store()
+        try:
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM papers').fetchone()[0],3)
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM source_records').fetchone()[0],4)
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM submissions').fetchone()[0],1)
+            self.assertEqual(store.conn.execute('SELECT count(*) FROM paper_merges').fetchone()[0],1)
+            self.assertEqual(store.conn.execute('SELECT status FROM ingest_runs ORDER BY id DESC LIMIT 1').fetchone()[0],'failed')
+            self.assertEqual(store.conn.execute("SELECT count(*) FROM source_records WHERE external_id IN (?,?)",('earlier','ambiguous')).fetchone()[0],0)
+        finally:store.conn.close()
+
     def test_file_import_is_idempotent_and_refreshes_only_own_tags(self):
         row = self.row(); row['tags'] = [{'namespace': 'openreview_snapshot:area', 'value': 'theory'}]
         path = self.file('reject', [row]); self.ingest(path)