Merge main and preserve coverage and history audit checks

.gitea/workflows/check.yml

@@ -49,11 +49,11 @@ jobs:
               subprocess.run([str(venv/'bin/paper-atlas'),'--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','collect','cvf','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','postgres','--help'],cwd=outside,check=True)
-              subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','joint-editions','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','import','records','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','import','arxiv','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','backfill','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','audit','public-openreview','--help'],cwd=outside,check=True)
+              subprocess.run([python,'-I','-B','-m','paper_atlas','coverage','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-m','paper_atlas','audit','histories','--help'],cwd=outside,check=True)
               subprocess.run([python,'-I','-B','-c','from paper_atlas.postgres.writer import PostgresStore, WriteConnection'],cwd=outside,check=True)
               smoke = "from paper_atlas.db.store import Store; from paper_atlas.config import DEFAULT_REGISTRY; import json; assert {'COLM','MM','MLSys'} <= {v['acronym'] for v in json.loads(DEFAULT_REGISTRY.read_text())['venues']}; s=Store('installed.sqlite'); s.conn.close()"

README.md

@@ -51,7 +51,8 @@ PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas query --pg-service fo
 PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas query --pg-service foundry-research --paper 123
 
 # 列出全部目标会议，包括尚未入库的项目
-paper-atlas coverage --db /data/paper-index/papers.sqlite \
+PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas coverage \
+  --pg-service foundry-research --pg-schema research \
   --out /data/paper-index/collected/venue-coverage-current.json
 
 # 采集与导入分开；这里只采集到独立文件
@@ -86,5 +87,3 @@ paper-atlas collect colm --cache /data/paper-index/raw/http-cache \
 CI 在已授权的远程 runner 上运行测试、构建 wheel，并从源码目录外验证安装后的命令及随包资源。数据库全量去重和覆盖率是独立的数据验收步骤，不因代码测试通过而自动完成。
 
 开发通过工作分支和 PR 交付，由用户手动合并。
-
-历史联合会议的既有届次归属可通过 [`maintain joint-editions`](docs/joint-editions.md) 先只读规划，再按计划哈希执行；保留论文与投稿决定来源，不重导入。

docs/joint-editions.md

@@ -1,67 +0,0 @@
-# Repairing joint conference membership
-
-EURAI's [official history](https://www.eurai.org/ecai/history) identifies ECAI 2018
-and 2022 as joint IJCAI/ECAI conferences. The pinned DBLP volumes
-`conf/ijcai/2018` and `conf/ijcai/2022` use `IJCAI` as their booktitle and title.
-Existing records can therefore already be present under IJCAI while an ECAI query
-shows a missing year. Adding ECAI as a global alias to the ordinary IJCAI venue
-would incorrectly include all other IJCAI years.
-
-The existing canonical `ECAI+IJCAI` venue has both member aliases. This maintenance
-command changes only the venue reference of the two existing `main` edition rows.
-Edition IDs, submission IDs, decisions, source text, authors, identifiers, tags,
-source links and coverage records stay intact. It does not merge or create papers
-or submissions, resolve per-paper tracks, or turn reviews into negative decisions.
-The DBLP collector applies the joint mapping only to these two exact main volume
-keys; original booktitle/title remain in source metadata with the membership evidence.
-The shared ingestion entry point also resolves these exact pinned source volumes before
-creating an edition, so refreshing an older normalized file cannot recreate a separate
-IJCAI edition. Other sources, years, workshop volumes and additional tracks retain their existing rules.
-
-## Review a read-only plan
-
-Use a runtime containing this repair command. The configured PostgreSQL writer lock
-is used only for the apply operation; do not replace a newer ingestion runtime with
-an older main-based feature branch.
-
-```sh
-PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain joint-editions \
-  --pg-service foundry-research --pg-schema research --out joint-plan.json
-```
-
-The preflight requires the existing joint venue and exactly its two member aliases,
-one nonempty unambiguous edition per year, and matching pinned DBLP volume evidence
-for every existing submission. A colliding target edition, missing provenance or
-unexpected source scope fails closed. The plan binds current database/schema, IDs,
-counts and full fingerprints of affected papers, identifiers, source records, tags,
-submissions, decisions and their provenance, and coverage rows. Observation time is
-recorded separately from conference year. These are scoped checks, not a final global
-identity or conference completeness audit.
-
-## Apply the exact reviewed plan
-
-Compute the file's SHA256 after review and pass it explicitly:
-
-```sh
-PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain joint-editions \
-  --pg-service foundry-research --pg-schema research --plan joint-plan.json \
-  --plan-sha256 "$PLAN_SHA256" --out joint-applied.json
-```
-
-The plan hash is checked before opening a writer. Current evidence is checked again
-under the dataset writer lock; both edition updates and an audit marker commit in one
-transaction. Protected row hashes are checked again before commit. Changed evidence
-requires a fresh reviewed plan. Existing output reports are preserved; choose a new
-path for another run. SQLite samples use `--db sample.sqlite`; the frozen production
-backup remains protected.
-
-If a response is lost after commit, rerun with the same plan and a new output path.
-When protected evidence is unchanged and both editions already point to the joint
-venue, the result is `already_joint` with zero updates. It does not replay imports or
-regenerate source records. Keep the plan, apply report and independent postcheck
-alongside collection evidence. The audit marker is `joint_ijcai_ecai_2018_2022_v1`.
-
-Publication counts can differ between the DBLP volume and the current publisher
-page. This membership repair preserves all existing rows rather than deleting entries
-to force an aggregate match; section-level main/invited/other classification and
-publication completeness require separate per-paper evidence.

docs/operations.md

@@ -97,13 +97,16 @@ paper-atlas audit arxiv-tags --db /data/paper-index/papers.sqlite \
 采集文件、已入库的来源记录和去重论文实体是不同的计数。来源可能覆盖同一篇论文，同一篇论文也可能有多次投稿；不能把各批文件数量相加作为独立论文总数。实时统计与覆盖报告使用：
 
 ```sh
-paper-atlas query --db /data/paper-index/papers.sqlite --stats
-paper-atlas coverage --db /data/paper-index/papers.sqlite \
+PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas query --pg-service foundry-research --pg-schema research --stats
+PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas coverage \
+  --pg-service foundry-research --pg-schema research \
   --out /data/paper-index/collected/venue-coverage-current.json
 ```
 
 manifest 和采集器状态清单用于确认文件是否齐全及是否缺摘要；`ingest_runs` 和协调器日志用于确认主库导入是否成功。官方程序、书目目录和公开投稿快照分别保留，未出现在某份清单不等于被拒稿；第三方快照也不能代表平台当前全量状态。代码测试成功不能代替最终全库去重、完整性和逐届覆盖核验。
 
+`coverage` 的所有计数来自同一个只读快照。`latest_observed_status_counts` 按每次投稿的最新观察决定统计，`decision_observation_counts` 保留所有历史观察；重复来源不增加投稿数，缺少决定的投稿计入最新状态 unknown，但不凭空制造 unknown 决定。`source_record_counts` 统计来源在该届次的实际记录与缺摘要数，和 `sources` 中逐清单的预期/导入数量分开；身份补充清单与原始论文集清单可以描述同一条记录。来源缺摘要与论文实体从其他来源获得摘要也分别报告。报告仍将历史覆盖标为待独立审计，不能用某个清单 complete 或没有拒稿记录推断全会议覆盖。
+
 当前代码与合并状态以 [Pull Requests](https://gitea.entroia.com/Zayden-Jung/paper-atlas/pulls) 为准；CI 结果以对应提交的 Actions 检查为准。工作分支交付后等待用户手动合并。合并代码不自动部署、不自动启动采集，也不代表数据批次已完成。
 ## 修正已有 PMLR 投稿的轨道
 

docs/usage.md

@@ -83,11 +83,14 @@ paper-atlas query --pg-service foundry-research --query '"language model"' --lim
 paper-atlas query --pg-service foundry-research --tag arxiv:category=cs.AI
 paper-atlas query --pg-service foundry-research --venue ICLR --year 2025 --status rejected
 paper-atlas query --pg-service foundry-research --paper 123
-paper-atlas coverage --db /data/paper-index/papers.sqlite --out /data/paper-index/collected/coverage.json
+PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas coverage \
+  --pg-service foundry-research --pg-schema research --out /data/paper-index/collected/coverage.json
 ```
 
 `--status rejected` 筛选某次投稿的最新已知决定。它并不表示论文在所有会议均未录用；分析转投情况时查看该论文的完整投稿历史。年份表示会议届次，观察时间和决定日期分别保留。
 
+覆盖报告使用只读一致性快照，标明实际数据库和 schema，并列出零入库目标与注册表外的已入库会议。逐届、逐 track 分别列出最新观察状态的投稿数、全部决定观察数、各来源记录数，以及来源摘要和论文实体摘要的缺失数。摘要缺失指 NULL 或空字符串；有文字不代表内容已经完整核验。同一来源的多个清单 URL 可以重叠，不能把清单计数相加当作来源总量；零条拒稿也不证明该会议没有公开拒稿。SQLite 样本仍可显式使用 `--db`，生产报告应查询当前 PostgreSQL。
+
 例如，查找在 ICLR 有拒稿记录、但在其他已知投稿中已有录用的论文：
 
 ```sh

paper_atlas/audits/coverage.py

@@ -1,45 +1,136 @@
-"""Show every target venue, including those with no imported records."""
+"""Describe imported venue coverage without implying a complete census."""
 import argparse
-from paper_atlas.config import DEFAULT_REGISTRY
+from collections import defaultdict
 import json
 from pathlib import Path
 import sqlite3
 
+from paper_atlas.config import DEFAULT_REGISTRY
+from paper_atlas.db.backend import database_arguments, open_read_connection
 from paper_atlas.db.submissions import timestamp
+from paper_atlas.querying import LATEST_DECISION
 
 
-def report(db,registry):
-    registry=json.loads(Path(registry).read_text());targets=registry['venues']
-    names=[row['acronym'].casefold() for row in targets]
-    if len(names)!=len(set(names)):raise ValueError('Duplicate target venue')
-    scope='(v.acronym=? COLLATE NOCASE OR EXISTS(SELECT 1 FROM venue_aliases a WHERE a.venue_id=v.id AND a.alias=? COLLATE NOCASE))'
-    c=sqlite3.connect(Path(db).resolve().as_uri()+'?mode=ro',uri=True);c.row_factory=sqlite3.Row
+STATUSES = ('accepted', 'rejected', 'desk_rejected', 'withdrawn', 'under_review', 'unknown')
+
+
+def _status_counts(connection, *, latest):
+    if latest:
+        query = '''SELECT s.edition_id,COALESCE(d.status,'unknown') AS status,count(*) AS count
+            FROM submissions s LEFT JOIN decisions d ON d.submission_id=s.id AND ''' + LATEST_DECISION
+    else:
+        query = '''SELECT s.edition_id,COALESCE(d.status,'unknown') AS status,count(*) AS count
+            FROM submissions s JOIN decisions d ON d.submission_id=s.id'''
+    counts = defaultdict(dict)
+    for row in connection.execute(query + " GROUP BY s.edition_id,COALESCE(d.status,'unknown')"):
+        counts[row[0]][row[1]] = row[2]
+    return counts
+
+
+def report(db, registry, *, pg_service=None, pg_schema=None):
+    started_at = timestamp()
+    registry = json.loads(Path(registry).read_text())
+    targets = registry['venues']
+    names = [row['acronym'].casefold() for row in targets]
+    if len(names) != len(set(names)):
+        raise ValueError('Duplicate target venue')
+    scope = '''(lower(v.acronym)=lower(?) OR EXISTS(
+        SELECT 1 FROM venue_aliases a WHERE a.venue_id=v.id AND lower(a.alias)=lower(?)))'''
+    connection = open_read_connection(db, pg_service=pg_service, pg_schema=pg_schema)
+    if not pg_service:
+        connection.row_factory = sqlite3.Row
     try:
-        c.execute('BEGIN');venues=[]
+        if pg_service:
+            identity = connection.execute('SELECT current_database(),current_schema()').fetchone()
+            database, schema = identity[0], identity[1]
+        else:
+            database, schema = str(Path(db).resolve()), None
+        latest = _status_counts(connection, latest=True)
+        observations = _status_counts(connection, latest=False)
+        # Aggregate the paper join once for the whole snapshot. Repeating it
+        # for each registry target caused repeated scans on the production DB.
+        edition_counts = {}
+        for row in connection.execute('''SELECT s.edition_id,count(s.id) AS submissions,
+                count(DISTINCT s.paper_id) AS papers,
+                count(DISTINCT CASE WHEN p.abstract IS NULL OR p.abstract='' THEN s.paper_id END)
+                    AS papers_without_abstract_text
+                FROM submissions s LEFT JOIN papers p ON p.id=s.paper_id GROUP BY s.edition_id'''):
+            entry = dict(row)
+            edition_counts[entry.pop('edition_id')] = entry
+        source_counts = defaultdict(list)
+        for row in connection.execute('''SELECT s.edition_id,so.name AS source,
+                count(DISTINCT sr.id) AS source_records,
+                count(DISTINCT CASE WHEN sr.abstract IS NULL OR sr.abstract='' THEN sr.id END)
+                    AS source_records_without_abstract_text
+                FROM submission_sources ss JOIN submissions s ON s.id=ss.submission_id
+                JOIN source_records sr ON sr.id=ss.source_record_id JOIN sources so ON so.id=sr.source_id
+                GROUP BY s.edition_id,so.name ORDER BY s.edition_id,so.name'''):
+            entry = dict(row)
+            source_counts[entry.pop('edition_id')].append(entry)
+        venues, covered_venue_ids = [], set()
         for target in targets:
-            acronym=target['acronym'];params=(acronym,acronym)
-            editions=[]
-            for row in c.execute('SELECT e.id,v.acronym AS canonical_venue,e.year,e.track,count(s.id) AS submissions,count(DISTINCT s.paper_id) AS papers FROM editions e JOIN venues v ON v.id=e.venue_id LEFT JOIN submissions s ON s.edition_id=e.id WHERE '+scope+' GROUP BY e.id ORDER BY e.year,e.track',params):
-                edition=dict(row)
-                edition['sources']=[dict(source) for source in c.execute('SELECT so.name,es.status,es.expected_count,es.imported_count,es.list_url,es.last_success_at,es.last_error FROM edition_sources es JOIN sources so ON so.id=es.source_id WHERE es.edition_id=? ORDER BY so.name',(row['id'],))]
+            acronym = target['acronym']
+            params = (acronym, acronym)
+            editions = []
+            for row in connection.execute('''SELECT e.id,v.id AS venue_id,v.acronym AS canonical_venue,
+                    e.year,e.track FROM editions e JOIN venues v ON v.id=e.venue_id
+                    WHERE ''' + scope + '''
+                    ORDER BY e.year,e.track,v.acronym''', params):
+                edition = dict(row)
+                edition.update(edition_counts.get(row['id'], {
+                    'submissions': 0, 'papers': 0, 'papers_without_abstract_text': 0}))
+                covered_venue_ids.add(edition['venue_id'])
+                edition['latest_observed_status_counts'] = {status: latest[row['id']].get(status, 0) for status in STATUSES}
+                edition['decision_observation_counts'] = {status: observations[row['id']].get(status, 0) for status in STATUSES}
+                edition['source_record_counts'] = source_counts[row['id']]
+                edition['sources'] = [dict(source) for source in connection.execute('''SELECT so.name,
+                    es.status,es.expected_count,es.imported_count,es.list_url,es.last_success_at,es.last_error
+                    FROM edition_sources es JOIN sources so ON so.id=es.source_id
+                    WHERE es.edition_id=? ORDER BY so.name,es.list_url''', (row['id'],))]
                 editions.append(edition)
-            totals=c.execute('SELECT count(s.id),count(DISTINCT s.paper_id) FROM submissions s JOIN editions e ON e.id=s.edition_id JOIN venues v ON v.id=e.venue_id WHERE '+scope,params).fetchone()
-            venues.append({'acronym':acronym,'field':target['field'],'imported_submissions':totals[0],'distinct_papers':totals[1],
-                           'history_status':'partial_history_pending_independent_audit' if totals[0] else 'no_imported_records',
-                           'discovery_sources':target.get('discovery_sources',[]),'editions':editions})
-        return {'checked_at':timestamp(),'target_venue_count':len(targets),'venues_with_imported_records':sum(bool(v['imported_submissions']) for v in venues),
-                'scope_note':registry['scope_note'],'interpretation':'All registry targets are listed. Edition source complete means that source file passed import checks; it does not prove full historical or abstract coverage. Papers can appear in multiple venues/years, so do not sum venue paper counts as a global total.',
-                'venues':venues}
-    finally:c.close()
+            totals = connection.execute('''SELECT count(s.id),count(DISTINCT s.paper_id)
+                FROM submissions s JOIN editions e ON e.id=s.edition_id
+                JOIN venues v ON v.id=e.venue_id WHERE ''' + scope, params).fetchone()
+            venues.append({'acronym': acronym, 'field': target['field'],
+                'imported_submissions': totals[0], 'distinct_papers': totals[1],
+                'history_status': 'partial_history_pending_independent_audit' if totals[0] else 'no_imported_records',
+                'discovery_sources': target.get('discovery_sources', []), 'editions': editions})
+        outside_registry = [dict(row) for row in connection.execute('''SELECT v.id AS venue_id,
+            v.acronym,count(s.id) AS submissions,count(DISTINCT s.paper_id) AS papers
+            FROM submissions s JOIN editions e ON e.id=s.edition_id JOIN venues v ON v.id=e.venue_id
+            GROUP BY v.id,v.acronym ORDER BY v.acronym''') if row[0] not in covered_venue_ids]
+        return {'report_started_at': started_at, 'checked_at': timestamp(),
+            'engine': 'postgresql' if pg_service else 'sqlite', 'database': database, 'schema': schema,
+            'pg_service': pg_service, 'target_venue_count': len(targets),
+            'venues_with_imported_records': sum(bool(v['imported_submissions']) for v in venues),
+            'scope_note': registry['scope_note'],
+            'interpretation': 'All registry targets are listed, including zero imports. '
+                'Latest status counts use the latest observed decision for each attempt; decision observations retain history. '
+                'Source record totals are per source and edition, not per list URL; multiple list entries may describe overlapping records. '
+                'Missing abstract text means NULL or empty string; nonempty text does not prove a complete abstract. '
+                'An edition source marked complete is not a proof of complete conference history or public rejection coverage. '
+                'Zero imported rejections does not establish that no rejections are public. '
+                'Papers can appear in several venues/years; do not sum venue paper counts as a global total.',
+            'venues': venues, 'imported_venues_outside_registry': outside_registry}
+    finally:
+        connection.close()
 
 
 def main(argv=None, *, prog=None):
-    parser=argparse.ArgumentParser(prog=prog);parser.add_argument('--db',required=True)
-    parser.add_argument('--registry',default=str(DEFAULT_REGISTRY));parser.add_argument('--out')
-    args=parser.parse_args(argv);result=report(args.db,args.registry);text=json.dumps(result,ensure_ascii=False,indent=2)
+    parser = argparse.ArgumentParser(prog=prog)
+    database_arguments(parser)
+    parser.add_argument('--registry', default=str(DEFAULT_REGISTRY))
+    parser.add_argument('--out')
+    args = parser.parse_args(argv)
+    result = report(args.db, args.registry, pg_service=args.pg_service, pg_schema=args.pg_schema)
+    text = json.dumps(result, ensure_ascii=False, indent=2)
     if args.out:
-        path=Path(args.out);stage=path.with_suffix('.part');stage.write_text(text+'\n');stage.replace(path)
-    else:print(text)
+        path = Path(args.out)
+        stage = path.with_suffix('.part')
+        stage.write_text(text + '\n')
+        stage.replace(path)
+    else:
+        print(text)
 
 
 if __name__ == '__main__':

paper_atlas/cli.py

@@ -25,7 +25,6 @@ COMMANDS = {
     ('catalogue','dblp'): ('paper_atlas.collectors.dblp', {}, '盘点 DBLP 历史会议目录'),
     ('import','arxiv'): ('paper_atlas.ingestion.arxiv', {}, '导入 arXiv 元数据 ZIP'),
     ('import','records'): ('paper_atlas.ingestion.records', {}, '导入核验过的会议文件'),
-    ('maintain','joint-editions'): ('paper_atlas.ingestion.joint_editions', {}, '按已核验计划修正既有联合会议届次归属'),
     ('maintain','merge'): ('paper_atlas.ingestion.maintenance', {}, '协调单写入合并与恢复'),
     ('maintain','backfill'): ('paper_atlas.ingestion.backfill', {}, '回填既有投稿的来源身份'),
     ('maintain','arxiv-tags'): ('paper_atlas.ingestion.arxiv_tags', {}, '恢复逐来源记录的 arXiv 类别证据'),

paper_atlas/collectors/dblp_records.py

@@ -88,8 +88,6 @@ def normalize_row(row,volumes,targets,include_workshops=False,through_year=2026)
               'scope_rule':edition['rule'],'author_names_raw':fields.get('author',[]),'publtype':row.get('publtype'),
               'bibliography':{key:value for key,value in fields.items() if key in ('booktitle','year','ee','url','pages','volume','number','publisher','series','isbn')},
               'ambiguous_identifiers':ambiguous,'publication_evidence':'inproceedings bibliography; not an original decision note'}
-    if edition.get('joint_edition_evidence'):
-        metadata['joint_edition_evidence'] = edition['joint_edition_evidence']
     if edition.get('track_scope_note'):
         metadata['track_scope_note']=edition['track_scope_note']
     flagged=bool(re.search(r'\b(?:withdrawn|retracted)\b',row.get('publtype') or '',re.I))

paper_atlas/collectors/dblp_scope.py

@@ -96,15 +96,8 @@ def scope(row,volume,through_year=2026):
     elif suffix.casefold() in ('poster','posters') and venue not in ('ICLR','ACL','EMNLP','NAACL','EACL','COLING','CoNLL'):track='poster'
     if venue=='AAAI+IAAI':track='proceedings_unspecified'
     if venue=='AAMAS' and year<2002:return None,'predecessor_not_aamas'
-    joint_evidence = None
-    if (family == 'ijcai' and base == 'IJCAI' and track == 'main'
-            and year in (2018, 2022) and volume['key'] == 'conf/ijcai/' + str(year)):
-        venue, aliases = 'ECAI+IJCAI', ['ECAI', 'IJCAI']
-        joint_evidence = 'https://www.eurai.org/ecai/history'
     result = {'venue':venue,'venue_aliases':aliases,'year':year,'track':track,'bibliographic_year':int(bibliographic[0]),
               'rule':family+':'+base,'proceedings_title':title,'edition_year_basis':year_basis}
-    if joint_evidence:
-        result['joint_edition_evidence'] = joint_evidence
     if venue=='MM' and track=='main':
         # A main MM volume also includes doctoral, tutorials, companion papers
         # and other sections. DBLP XML omits the per-paper TOC section.

paper_atlas/db/store.py

@@ -144,20 +144,6 @@ class Store:
 
     def decision(self,paper,record,venue,year,track,external_id,url,status,raw_value,aliases=None,identities=None,observed_at=None,decision_date=None,source_decision_id='',submitted_at=None,guard_status=False):
         c = self.conn
-        # An already collected file may still call these exact pinned volumes
-        # IJCAI. Resolve membership before finding/creating an edition so a
-        # later refresh cannot split the repaired joint event again.
-        if venue == 'IJCAI' and year in (2018,2022) and track == 'main':
-            from paper_atlas.collectors.dblp import XML_SHA
-            evidence=c.execute('SELECT src.name,r.metadata_json FROM source_records r JOIN sources src ON src.id=r.source_id WHERE r.id=?',(record,)).fetchone()
-            if evidence and evidence[0] == 'dblp':
-                metadata=json.loads(evidence[1])
-                if (metadata.get('snapshot_sha256') == XML_SHA
-                        and metadata.get('proceedings_key') == 'conf/ijcai/'+str(year)
-                        and metadata.get('bibliographic_year') == year
-                        and (metadata.get('bibliography') or {}).get('booktitle') in (['IJCAI'],['IJCAI/ECAI'])):
-                    venue='ECAI+IJCAI'
-                    aliases=sorted(set(aliases or []) | {'ECAI','IJCAI'})
         existing=c.execute('SELECT id FROM venues WHERE lower(acronym)=lower(?)',(venue,)).fetchone()
         if existing:venue_id=existing[0]
         else:venue_id=inserted_id(c,"INSERT INTO venues(acronym,name,field,venue_type) VALUES(?,?,?,'conference')",(venue,venue,'unclassified'))

paper_atlas/ingestion/joint_editions.py

@@ -1,204 +0,0 @@
-"""Evidence-bound repair of two existing IJCAI/ECAI edition memberships."""
-import argparse
-from contextlib import closing
-import hashlib
-import json
-from pathlib import Path
-
-from paper_atlas.db.backend import database_arguments, open_read_connection, open_store
-from paper_atlas.db.submissions import timestamp
-from paper_atlas.collectors.dblp import XML_SHA
-
-YEARS = (2018, 2022)
-TARGET = 'ECAI+IJCAI'
-MEMBERS = {'ECAI', 'IJCAI'}
-EVIDENCE = 'https://www.eurai.org/ecai/history'
-MARKER = 'joint_ijcai_ecai_2018_2022_v1'
-
-
-def _digest(c, sql, params=()):
-    sha = hashlib.sha256(); count = 0
-    for row in c.execute(sql, params):
-        sha.update((json.dumps([row[i] for i in range(len(row))], ensure_ascii=False,
-                              separators=(',', ':')) + '\n').encode())
-        count += 1
-        if count > 100000:
-            raise ValueError('Joint repair evidence scope exceeded bounded row limit')
-    return {'rows': count, 'sha256': sha.hexdigest()}
-
-
-def fingerprints(c, ids):
-    marks = ','.join('?' for _ in ids)
-    submissions = 'SELECT id FROM submissions WHERE edition_id IN (' + marks + ')'
-    papers = 'SELECT paper_id FROM submissions WHERE edition_id IN (' + marks + ')'
-    decisions = 'SELECT id FROM decisions WHERE submission_id IN (' + submissions + ')'
-    checks = {
-        'papers': ('id IN (' + papers + ')', 'id'),
-        'identifiers': ('paper_id IN (' + papers + ')', 'scheme,value'),
-        'source_records': ('paper_id IN (' + papers + ')', 'id'),
-        'paper_tags': ('paper_id IN (' + papers + ')', 'paper_id,tag_id,evidence_key'),
-        'submissions': ('edition_id IN (' + marks + ')', 'id'),
-        'decisions': ('submission_id IN (' + submissions + ')', 'id'),
-        'submission_sources': ('submission_id IN (' + submissions + ')', 'submission_id,source_record_id,source_submission_id'),
-        'submission_identifiers': ('edition_id IN (' + marks + ')', 'edition_id,scheme,value'),
-        'decision_sources': ('decision_id IN (' + decisions + ')', 'decision_id,source_record_id,source_decision_id'),
-        'edition_sources': ('edition_id IN (' + marks + ')', 'edition_id,source_id,list_url'),
-    }
-    return {table: _digest(c, 'SELECT * FROM ' + table + ' WHERE ' + where + ' ORDER BY ' + order, ids)
-            for table, (where, order) in checks.items()}
-
-
-def inspect(c):
-    target = c.execute('SELECT id FROM venues WHERE acronym=?', (TARGET,)).fetchall()
-    if len(target) != 1:
-        raise ValueError('Exactly one existing canonical joint venue required')
-    target = target[0][0]
-    aliases = {r[0] for r in c.execute('SELECT alias FROM venue_aliases WHERE venue_id=?', (target,))}
-    if aliases != MEMBERS:
-        raise ValueError('Joint target must already have exactly ECAI and IJCAI aliases')
-    editions = []
-    for year in YEARS:
-        rows = c.execute('''SELECT e.id,e.venue_id,v.acronym FROM editions e JOIN venues v ON v.id=e.venue_id
-            WHERE v.acronym IN ('IJCAI','ECAI+IJCAI') AND e.year=? AND e.track='main' ''', (year,)).fetchall()
-        if len(rows) != 1:
-            raise ValueError('Expected one existing edition per year; missing or colliding editions require separate review')
-        edition, venue, acronym = (rows[0][i] for i in range(3))
-        count = c.execute('SELECT count(*) FROM submissions WHERE edition_id=?', (edition,)).fetchone()[0]
-        if not 0 < count <= 5000:
-            raise ValueError('Existing edition must have a nonempty bounded submission scope')
-        # Every existing submission must be supported by its own pinned DBLP
-        # volume record. No title/author matching, new attempts or text writes.
-        evidence = c.execute('''SELECT s.id,r.external_id,r.metadata_json FROM submissions s
-            JOIN submission_sources ss ON ss.submission_id=s.id
-            JOIN source_records r ON r.id=ss.source_record_id JOIN sources src ON src.id=r.source_id
-            WHERE s.edition_id=? AND src.name='dblp' ORDER BY s.id,r.id''', (edition,)).fetchall()
-        seen = set()
-        for row in evidence:
-            sid, external, metadata = (row[i] for i in range(3))
-            meta = json.loads(metadata)
-            bibliography = meta.get('bibliography') or {}
-            if (meta.get('snapshot_sha256') != XML_SHA
-                    or meta.get('proceedings_key') != 'conf/ijcai/' + str(year)
-                    or meta.get('bibliographic_year') != year
-                    or bibliography.get('booktitle') not in (['IJCAI'], ['IJCAI/ECAI'])
-                    or not external.startswith('conf/ijcai/')):
-                raise ValueError('Source evidence outside the exact verified IJCAI/ECAI volume')
-            seen.add(sid)
-        if len(seen) != count:
-            raise ValueError('Every submission must have matching original volume evidence')
-        editions.append({'id': edition, 'year': year, 'track': 'main', 'from_venue_id': venue,
-                         'from_venue': acronym, 'target_venue_id': target, 'submissions': count,
-                         'evidence_url': EVIDENCE, 'proceedings_key': 'conf/ijcai/' + str(year)})
-    ids = tuple(e['id'] for e in editions)
-    return {'format_version': 1, 'kind': MARKER, 'checked_at': timestamp(), 'editions': editions,
-            'content_fingerprints': fingerprints(c, ids), 'target_aliases': sorted(aliases),
-            'status': 'already_joint' if all(e['from_venue_id'] == target for e in editions) else 'ready',
-            'scope': 'Only these two existing edition venue references; no source text, paper, submission, decision, track, identifier or tag change.'}
-
-
-def validate_plan(expected):
-    if not isinstance(expected, dict):
-        raise ValueError('Repair plan must be a JSON object')
-    if set(expected) != {'format_version','kind','checked_at','editions','content_fingerprints','target_aliases','status','scope','database'}:
-        raise ValueError('Unexpected repair plan fields')
-    if (type(expected['format_version']) is not int or expected['format_version'] != 1
-            or expected['kind'] != MARKER or expected['status'] not in ('ready','already_joint')):
-        raise ValueError('Unsupported repair plan')
-    if (not isinstance(expected['database'],dict) or not isinstance(expected['content_fingerprints'],dict)
-            or not isinstance(expected['editions'],list) or len(expected['editions']) != 2
-            or expected['target_aliases'] != sorted(MEMBERS)):
-        raise ValueError('Invalid repair plan structure')
-    for edition, year in zip(expected['editions'], YEARS):
-        if not isinstance(edition,dict) or edition.get('year') != year:
-            raise ValueError('Repair plan must name the two exact ordered years')
-        for key in ('id','year','from_venue_id','target_venue_id','submissions'):
-            if type(edition.get(key)) is not int or edition[key] <= 0:
-                raise ValueError('Repair plan IDs/counts must be positive integers')
-
-
-def apply_plan(c, expected):
-    validate_plan(expected)
-    actual = inspect(c)
-    if actual['content_fingerprints'] != expected['content_fingerprints'] or actual['target_aliases'] != expected['target_aliases']:
-        raise ValueError('Stale repair plan: protected content or aliases changed')
-    if len(expected['editions']) != 2:
-        raise ValueError('Exactly two editions required')
-    for old, now in zip(expected['editions'], actual['editions']):
-        if set(old) != set(now):
-            raise ValueError('Unexpected edition fields')
-        for key in ('id','year','track','target_venue_id','submissions','evidence_url','proceedings_key'):
-            if old[key] != now[key]:
-                raise ValueError('Stale repair plan: edition differs')
-        if now['from_venue_id'] != old['from_venue_id'] and now['from_venue_id'] != now['target_venue_id']:
-            raise ValueError('Edition membership differs from plan')
-    if actual['status'] == 'already_joint':
-        return {**actual, 'status':'already_joint', 'updated_editions':0}
-    for edition in actual['editions']:
-        if edition['from_venue_id'] == edition['target_venue_id']:
-            continue
-        changed = c.execute('''UPDATE editions SET venue_id=? WHERE id=? AND venue_id=? AND year=? AND track=?''',
-                            (edition['target_venue_id'],edition['id'],edition['from_venue_id'],edition['year'],edition['track']))
-        if changed.rowcount != 1:
-            raise ValueError('Concurrent edition change; entire repair must roll back')
-    after = inspect(c)
-    if after['status'] != 'already_joint' or after['content_fingerprints'] != actual['content_fingerprints']:
-        raise ValueError('Postcheck changed protected content; entire repair must roll back')
-    c.execute('INSERT INTO schema_migrations(name) VALUES(?) ON CONFLICT DO NOTHING', (MARKER,))
-    return {**after, 'status':'applied', 'updated_editions':sum(e['from_venue_id'] != e['target_venue_id'] for e in actual['editions'])}
-
-
-def database_identity(c, db):
-    if getattr(c, 'postgres', False):
-        row = c.execute('SELECT current_database(),current_schema()').fetchone()
-        return {'engine':'postgresql','database':row[0],'schema':row[1]}
-    return {'engine':'sqlite','path':str(Path(db).resolve())}
-
-
-def run(db, out, *, pg_service=None, pg_schema=None, plan=None, plan_sha256=None):
-    out = Path(out).resolve()
-    if not out.parent.is_dir():
-        raise ValueError('Report parent directory must already exist')
-    if out.exists():
-        raise ValueError('Preserve existing evidence report; choose a new output path')
-    expected = None
-    if plan:
-        raw = Path(plan).read_bytes()
-        if hashlib.sha256(raw).hexdigest() != plan_sha256:
-            raise ValueError('Repair plan hash differs before opening database')
-        expected = json.loads(raw)
-        validate_plan(expected)
-    elif plan_sha256:
-        raise ValueError('Plan hash requires an existing plan')
-    if db and out == Path(db).resolve():
-        raise ValueError('Report must not overwrite database')
-    writer = open_store(db, pg_service=pg_service, pg_schema=pg_schema) if expected is not None else None
-    c = writer.conn if writer else open_read_connection(db, pg_service=pg_service, pg_schema=pg_schema)
-    with closing(c):
-        try:
-            if expected is not None and not getattr(c, 'postgres', False):
-                c.commit(); c.execute('BEGIN IMMEDIATE')
-            identity = database_identity(c, db)
-            if expected is not None and expected['database'] != identity:
-                raise ValueError('Plan belongs to another database/schema')
-            report = apply_plan(c, expected) if expected is not None else inspect(c)
-            report['database'] = identity
-            if expected is not None:
-                report['plan_sha256'] = plan_sha256
-            if expected is not None:
-                c.commit()
-        except BaseException:
-            if expected is not None: c.rollback()
-            raise
-    out.write_text(json.dumps(report,ensure_ascii=False,indent=2)+'\n')
-    print(json.dumps({'status':report['status'],'editions':len(report['editions']),
-                      'updated_editions':report.get('updated_editions',0),'report':str(out)}))
-    return report
-
-
-def main(argv=None, *, prog=None):
-    parser = argparse.ArgumentParser(prog=prog); database_arguments(parser)
-    parser.add_argument('--out', required=True)
-    parser.add_argument('--plan', help='Apply an already reviewed plan; omitted means read-only preflight')
-    parser.add_argument('--plan-sha256', help='Required exact plan SHA256 when applying')
-    args = parser.parse_args(argv)
-    run(args.db,args.out,pg_service=args.pg_service,pg_schema=args.pg_schema,plan=args.plan,plan_sha256=args.plan_sha256)

tests/test_coverage_report.py

@@ -7,6 +7,34 @@ from paper_atlas.audits.coverage import report
 from paper_atlas.db.store import Store
 
 
+def add_coverage_evidence(store):
+    paper, record = store.record('official_metadata', 'known', 'https://example.org/known',
+        'Known paper with two source observations', 'Complete canonical abstract', ['Jane Doe', 'John Smith'],
+        {'openreview': 'coverageForum', 'arxiv': '2401.98765'})
+    edition = store.decision(paper, record, 'COLING+LREC', 2024, 'main', 'known',
+        'https://example.org/accept', 'accepted', 'Accept', aliases=['COLING', 'LREC'],
+        identities=[('openreview', 'coverageForum')], observed_at='2025-05-01T00:00:00Z')
+    same, mirror = store.record('bibliography', 'mirror', 'https://example.org/mirror',
+        'Known paper with two source observations', None, ['Jane Doe', 'John Smith'], {'openreview': 'coverageForum'})
+    store.decision(same, mirror, 'COLING+LREC', 2024, 'main', 'mirror', 'https://example.org/older',
+        'rejected', 'Reject', identities=[('openreview', 'coverageForum')],
+        observed_at='2025-05-01T00:00:00Z', decision_date='2025-04-30T00:00:00Z')
+    unknown, unknown_record = store.record('unknown_catalogue', 'unknown', 'https://example.org/unknown',
+        'Indexed work without a public decision', None, [])
+    store.decision(unknown, unknown_record, 'COLING+LREC', 2024, 'main', 'unknown',
+        'https://example.org/unknown', None, '', identities=[('openreview', 'unknownForum')])
+    same, later = store.record('official_metadata', 'later', 'https://example.org/later',
+        'Known paper with two source observations', 'Complete canonical abstract', ['Jane Doe', 'John Smith'],
+        {'openreview': 'coverageNext', 'arxiv': '2401.98765'})
+    store.decision(same, later, 'COLING+LREC', 2025, 'findings', 'later', 'https://example.org/later',
+        'accepted', 'Accept', identities=[('openreview', 'coverageNext')], observed_at='2025-06-01T00:00:00Z')
+    source = store.source('official_metadata', 'https://example.org')
+    for url, status in [('https://example.org/list', 'complete'), ('https://example.org/identity-refresh', 'partial')]:
+        store.conn.execute('''INSERT INTO edition_sources(edition_id,source_id,list_url,status,expected_count,imported_count)
+            VALUES(?,?,?,?,?,?)''', (edition, source, url, status, 1, 1))
+    store.conn.commit()
+
+
 class CoverageTests(unittest.TestCase):
     def setUp(self):
         self.tmp=tempfile.TemporaryDirectory();self.addCleanup(self.tmp.cleanup)
@@ -31,5 +59,46 @@ class CoverageTests(unittest.TestCase):
         self.store.conn.commit();coling=report(self.db,self.registry)['venues'][0]
         self.assertEqual(coling['imported_submissions'],2);self.assertEqual(coling['distinct_papers'],1)
 
+    def test_observations_attempts_and_missing_source_abstracts_stay_separate(self):
+        add_coverage_evidence(self.store)
+        result = report(self.db, self.registry)
+        self.assertEqual(result['engine'], 'sqlite')
+        self.assertEqual(result['database'], str(self.db))
+        coling = result['venues'][0]
+        self.assertEqual((coling['imported_submissions'], coling['distinct_papers']), (3, 2))
+        edition, findings = coling['editions']
+        self.assertEqual((edition['submissions'], edition['papers'], edition['papers_without_abstract_text']), (2, 2, 1))
+        self.assertEqual(edition['latest_observed_status_counts']['accepted'], 1)
+        self.assertEqual(edition['latest_observed_status_counts']['rejected'], 0)
+        self.assertEqual(edition['latest_observed_status_counts']['unknown'], 1)
+        self.assertEqual(edition['decision_observation_counts']['accepted'], 1)
+        self.assertEqual(edition['decision_observation_counts']['rejected'], 1)
+        self.assertEqual(edition['decision_observation_counts']['unknown'], 0)
+        counts = {row['source']: row for row in edition['source_record_counts']}
+        self.assertEqual(counts['bibliography']['source_records_without_abstract_text'], 1)
+        self.assertEqual(counts['official_metadata']['source_records'], 1)
+        self.assertEqual(len(edition['sources']), 2)
+        self.assertEqual(findings['track'], 'findings')
+        self.assertEqual(result['imported_venues_outside_registry'], [])
+
+    def test_missing_sqlite_audit_does_not_create_a_database(self):
+        path = self.root / 'absent.sqlite'
+        with self.assertRaisesRegex(ValueError, 'Existing database'):
+            report(path, self.registry)
+        self.assertFalse(path.exists())
+
+    def test_empty_editions_remain_visible_after_shared_aggregation(self):
+        self.store.conn.execute("INSERT INTO venues(acronym,name,field,venue_type) VALUES('AAAI','AAAI','general_ai','conference')")
+        venue = self.store.conn.execute("SELECT id FROM venues WHERE acronym='AAAI'").fetchone()[0]
+        self.store.conn.execute("INSERT INTO editions(venue_id,year,track) VALUES(?,2026,'main')", (venue,))
+        self.store.conn.commit()
+        aaai = report(self.db, self.registry)['venues'][1]
+        self.assertEqual(aaai['history_status'], 'no_imported_records')
+        self.assertEqual(len(aaai['editions']), 1)
+        edition = aaai['editions'][0]
+        self.assertEqual((edition['submissions'], edition['papers'], edition['papers_without_abstract_text']), (0, 0, 0))
+        self.assertEqual(sum(edition['latest_observed_status_counts'].values()), 0)
+        self.assertEqual(edition['source_record_counts'], [])
+
 
 if __name__=='__main__':unittest.main()

tests/test_joint_editions.py

@@ -1,187 +0,0 @@
-from contextlib import closing, redirect_stdout
-import hashlib
-import io
-import json
-import os
-from pathlib import Path
-import tempfile
-import unittest
-from unittest.mock import patch
-import uuid
-
-from paper_atlas.collectors.dblp import XML_SHA
-from paper_atlas.collectors.dblp_scope import scope
-from paper_atlas.collectors.dblp_records import normalize_row
-from paper_atlas.db.store import Store
-from paper_atlas.db.backend import open_store
-from paper_atlas.ingestion.joint_editions import (TARGET, MARKER, inspect, apply_plan, run)
-from paper_atlas.postgres.client import connect
-from paper_atlas.postgres.migration import migrate
-from paper_atlas.postgres.schema import quote
-from paper_atlas import querying
-
-
-def seed(store):
-    c = store.conn
-    c.execute("INSERT INTO venues(acronym,name,field,venue_type) VALUES(?,?,'reasoning','conference')", (TARGET,TARGET))
-    target = c.execute('SELECT id FROM venues WHERE acronym=?',(TARGET,)).fetchone()[0]
-    for alias in ['ECAI','IJCAI']:
-        c.execute('INSERT INTO venue_aliases VALUES(?,?)',(target,alias))
-    for year in [2018,2019,2022]:
-        metadata = {'snapshot_sha256':XML_SHA,'proceedings_key':'conf/ijcai/'+str(year),
-                    'bibliographic_year':year,'bibliography':{'booktitle':['IJCAI']}}
-        p,r=store.record('dblp','conf/ijcai/fixture'+str(year),'https://dblp.org/rec/fixture',
-                         'Original publication '+str(year),'Abstract\x00with\x01data',[],
-                         {'doi':'10.24963/ijcai.'+str(year)+'/1'}, {})
-        store.record_tags(p,r,'dblp',[{'namespace':'dblp:booktitle','value':'IJCAI'}])
-        store.decision(p,r,'IJCAI',year,'main','fixture'+str(year),'https://example.org/decision','accepted','listed')
-        stored=json.loads(c.execute('SELECT metadata_json FROM source_records WHERE id=?',(r,)).fetchone()[0]);stored.update(metadata)
-        c.execute('UPDATE source_records SET metadata_json=? WHERE id=?',(json.dumps(stored),r))
-    c.commit()
-
-
-def plan(c):
-    return {**inspect(c),'database':{'engine':'sqlite','path':'fixture'}}
-
-
-class JointEditionTests(unittest.TestCase):
-    def setUp(self):
-        self.tmp=tempfile.TemporaryDirectory(); self.addCleanup(self.tmp.cleanup)
-        self.db=Path(self.tmp.name)/'sample.sqlite'
-        self.store=Store(self.db); self.addCleanup(self.store.conn.close)
-        seed(self.store); self.c=self.store.conn
-
-    def test_apply_preserves_every_protected_row_and_edition_id(self):
-        before=plan(self.c); ids=[r['id']for r in before['editions']]
-        result=apply_plan(self.c,before);self.c.commit()
-        self.assertEqual(result['updated_editions'],2)
-        self.assertEqual(result['content_fingerprints'],before['content_fingerprints'])
-        self.assertEqual([r['id']for r in result['editions']],ids)
-        untouched=self.c.execute('SELECT v.acronym FROM editions e JOIN venues v ON v.id=e.venue_id WHERE e.year=2019').fetchone()[0]
-        self.assertEqual(untouched,'IJCAI')
-        self.assertEqual(self.c.execute('PRAGMA foreign_key_check').fetchall(),[])
-        self.assertEqual(self.c.execute('SELECT count(*) FROM schema_migrations WHERE name=?',(MARKER,)).fetchone()[0],1)
-        with redirect_stdout(io.StringIO())as out:
-            querying.main(['--db',str(self.db),'--venue','ECAI'])
-        self.assertEqual(len(json.loads(out.getvalue())),2)
-        self.assertEqual(apply_plan(self.c,before)['updated_editions'],0)
-
-    def test_refresh_of_old_normalized_source_does_not_recreate_ordinary_edition(self):
-        before=plan(self.c);apply_plan(self.c,before)
-        for year in [2018,2022]:
-            row=self.c.execute('''SELECT r.paper_id,r.id,d.observed_at FROM source_records r
-                JOIN submissions s ON s.source_record_id=r.id JOIN decisions d ON d.submission_id=s.id
-                WHERE r.external_id=?''',('conf/ijcai/fixture'+str(year),)).fetchone()
-            self.store.decision(row[0],row[1],'IJCAI',year,'main','fixture'+str(year),
-                                'https://example.org/decision','accepted','listed',observed_at=row[2])
-        self.assertEqual(inspect(self.c)['content_fingerprints'],before['content_fingerprints'])
-        self.assertEqual(self.c.execute('SELECT count(*) FROM editions').fetchone()[0],3)
-
-    def test_any_changed_source_text_rejects_stale_plan_before_membership_updates(self):
-        before=plan(self.c)
-        self.c.execute("UPDATE source_records SET title='New evidence' WHERE external_id='conf/ijcai/fixture2018'")
-        with self.assertRaisesRegex(ValueError,'Stale repair plan'):
-            apply_plan(self.c,before)
-        self.assertEqual(inspect(self.c)['status'],'ready')
-
-    def test_conflicting_target_edition_is_never_merged_or_deleted(self):
-        target=self.c.execute('SELECT id FROM venues WHERE acronym=?',(TARGET,)).fetchone()[0]
-        self.c.execute("INSERT INTO editions(venue_id,year,track) VALUES(?,2018,'main')",(target,))
-        with self.assertRaisesRegex(ValueError,'colliding'):
-            inspect(self.c)
-        self.assertEqual(self.c.execute('SELECT count(*) FROM editions WHERE year=2018').fetchone()[0],2)
-
-    def test_wrong_source_volume_and_snapshot_fail_closed(self):
-        original=self.c.execute("SELECT metadata_json FROM source_records WHERE external_id='conf/ijcai/fixture2018'").fetchone()[0]
-        for key,value in [('proceedings_key','conf/ijcai/2018w'),('snapshot_sha256','wrong'),('bibliographic_year',2017),('bibliography',{'booktitle':['Other']})]:
-            m=json.loads(original);m[key]=value
-            self.c.execute("UPDATE source_records SET metadata_json=? WHERE external_id='conf/ijcai/fixture2018'",(json.dumps(m),))
-            with self.subTest(key=key),self.assertRaisesRegex(ValueError,'Source evidence'):
-                inspect(self.c)
-
-    def test_missing_source_support_and_wrong_aliases_are_not_guessed(self):
-        self.c.execute("DELETE FROM venue_aliases WHERE alias='ECAI'")
-        with self.assertRaisesRegex(ValueError,'aliases'):
-            inspect(self.c)
-        self.c.rollback()
-        self.c.execute('DELETE FROM submission_sources WHERE source_record_id=(SELECT id FROM source_records WHERE external_id=?)',('conf/ijcai/fixture2022',))
-        with self.assertRaisesRegex(ValueError,'Every submission'):
-            inspect(self.c)
-
-    def test_cli_plan_readonly_hash_binding_identity_and_repeat_apply(self):
-        out=Path(self.tmp.name)/'plan.json'
-        with redirect_stdout(io.StringIO()):run(self.db,out)
-        self.assertEqual(inspect(self.c)['status'],'ready')
-        raw=out.read_bytes();sha=hashlib.sha256(raw).hexdigest()
-        with patch('paper_atlas.ingestion.joint_editions.open_store')as writer:
-            with self.assertRaisesRegex(ValueError,'hash differs'):
-                run(self.db,Path(self.tmp.name)/'bad.json',plan=out,plan_sha256='0'*64)
-            writer.assert_not_called()
-        with redirect_stdout(io.StringIO()):
-            result=run(self.db,Path(self.tmp.name)/'applied.json',plan=out,plan_sha256=sha)
-            again=run(self.db,Path(self.tmp.name)/'retry.json',plan=out,plan_sha256=sha)
-        self.assertEqual(result['updated_editions'],2);self.assertEqual(again['updated_editions'],0)
-        self.assertEqual(result['plan_sha256'],sha);self.assertEqual(again['plan_sha256'],sha)
-        altered=json.loads(raw);altered['database']['path']='another.sqlite'
-        bad=Path(self.tmp.name)/'wrong-db.json';bad.write_text(json.dumps(altered))
-        with self.assertRaisesRegex(ValueError,'another database'):
-            run(self.db,Path(self.tmp.name)/'wrong-result.json',plan=bad,plan_sha256=hashlib.sha256(bad.read_bytes()).hexdigest())
-
-    def test_invalid_empty_plan_fails_before_opening_writer(self):
-        f=Path(self.tmp.name)/'empty.json';f.write_text('{}')
-        with patch('paper_atlas.ingestion.joint_editions.open_store')as writer:
-            with self.assertRaises(ValueError):
-                run(self.db,Path(self.tmp.name)/'empty-result.json',plan=f,plan_sha256=hashlib.sha256(f.read_bytes()).hexdigest())
-            writer.assert_not_called()
-
-    def test_collector_only_changes_the_two_exact_main_volume_keys(self):
-        for year in [2018,2019,2022]:
-            volume={'key':'conf/ijcai/'+str(year),'fields':{'title':['IJCAI '+str(year)],'year':[str(year)]}}
-            row={'record_type':'inproceedings','key':'conf/ijcai/One'+str(year),'fields':{'booktitle':['IJCAI'],
-                 'year':[str(year)],'crossref':[volume['key']],'title':['Original research title'],
-                 'author':['Jane Doe'],'ee':['https://doi.org/10.24963/ijcai.'+str(year)+'/1']}}
-            result,error=normalize_row(row,{volume['key']:volume},{'IJCAI','ECAI'})
-            self.assertIsNone(error)
-            self.assertEqual(result['venue'],TARGET if year in(2018,2022)else'IJCAI')
-            self.assertEqual(result['metadata']['bibliography']['booktitle'],['IJCAI'])
-            self.assertEqual(result['title'],'Original research title')
-            if year in(2018,2022):
-                self.assertIn('joint_edition_evidence',result['metadata'])
-                row['fields']['booktitle']=['IJCAI (Workshop)']
-                scoped,_=scope(row,volume)
-                self.assertEqual(scoped['venue'],'IJCAI')
-                self.assertEqual(scoped['track'],'workshop')
-
-
-@unittest.skipUnless(os.environ.get('PAPER_ATLAS_TEST_PG_SERVICE'),'Requires approved remote PostgreSQL service')
-class PostgreSQLJointTests(unittest.TestCase):
-    def setUp(self):
-        self.service=os.environ['PAPER_ATLAS_TEST_PG_SERVICE'];self.target='pa_joint_'+uuid.uuid4().hex[:12]
-        self.tmp=tempfile.TemporaryDirectory();self.addCleanup(self.tmp.cleanup)
-        self.db=Path(self.tmp.name)/'sample.sqlite';s=Store(self.db);seed(s);s.conn.close()
-        self.addCleanup(self.cleanup)
-        with redirect_stdout(io.StringIO()):migrate(self.db,self.service,Path(self.tmp.name)/'migration.json',schema=self.target,min_free_gib=0)
-
-    def cleanup(self):
-        with connect(self.service)as c:
-            for name, in c.execute('SELECT nspname FROM pg_namespace WHERE starts_with(nspname,%s)',(self.target,)).fetchall():
-                c.execute('DROP SCHEMA '+quote(name)+' CASCADE')
-
-    def test_native_repair_is_idempotent_keeps_codec_and_provenance(self):
-        p=Path(self.tmp.name)/'plan.json'
-        with redirect_stdout(io.StringIO()):
-            before=run(None,p,pg_service=self.service,pg_schema=self.target)
-            after=run(None,Path(self.tmp.name)/'applied.json',pg_service=self.service,pg_schema=self.target,plan=p,plan_sha256=hashlib.sha256(p.read_bytes()).hexdigest())
-        self.assertEqual(after['content_fingerprints'],before['content_fingerprints'])
-        with closing(open_store(pg_service=self.service,pg_schema=self.target).conn)as c:
-            self.assertEqual(apply_plan(c,before)['updated_editions'],0)
-            self.assertEqual(c.execute('SELECT abstract FROM papers ORDER BY id LIMIT 1').fetchone()[0],'Abstract\x00with\x01data')
-
-    def test_native_stale_source_plan_preserves_both_original_memberships(self):
-        with closing(open_store(pg_service=self.service,pg_schema=self.target).conn)as c:
-            before=plan(c)
-            c.execute("UPDATE source_records SET title='Changed' WHERE external_id=?",('conf/ijcai/fixture2022',));c.commit()
-            with self.assertRaisesRegex(ValueError,'Stale repair plan'):
-                apply_plan(c,before)
-            c.rollback()
-            self.assertEqual(inspect(c)['status'],'ready')

tests/test_postgres_ingestion.py

@@ -243,6 +243,50 @@ class PostgreSQLIngestionTests(unittest.TestCase):
         finally:
             reader.close()
 
+    def test_coverage_matches_sqlite_with_history_aliases_and_missing_source_abstracts(self):
+        from paper_atlas.audits.coverage import report
+        from tests.test_coverage_report import add_coverage_evidence
+        registry = self.root / 'venues.json'
+        registry.write_text(json.dumps({'scope_note': 'Candidate scope', 'venues': [
+            {'acronym': 'coling', 'field': 'language'}, {'acronym': 'AAAI', 'field': 'general_ai'}]}))
+        sample = Store(self.root / 'seed.sqlite')
+        try:
+            add_coverage_evidence(sample)
+        finally:
+            sample.conn.close()
+        store = self.store()
+        try:
+            add_coverage_evidence(store)
+        finally:
+            store.conn.close()
+        sqlite_report = report(self.root / 'seed.sqlite', registry)
+        observed = []
+        def reader(*args, **kwargs):
+            connection = open_read_connection(*args, **kwargs)
+            observed.append((connection.execute('SHOW transaction_read_only').fetchone()[0],
+                             connection.execute('SHOW transaction_isolation').fetchone()[0]))
+            return connection
+        with patch('paper_atlas.audits.coverage.open_read_connection', side_effect=reader):
+            actual = report(None, registry, pg_service=self.service, pg_schema=self.schema)
+        self.assertEqual(observed, [('on', 'repeatable read')])
+        self.assertEqual(actual['engine'], 'postgresql')
+        self.assertEqual(actual['schema'], self.schema)
+        self.assertEqual(actual['pg_service'], self.service)
+        def content(result):
+            values = json.loads(json.dumps(result['venues']))
+            for venue in values:
+                for edition in venue['editions']:
+                    for key in ('id', 'venue_id'):
+                        edition.pop(key)
+            return values
+        self.assertEqual(content(actual), content(sqlite_report))
+        self.assertEqual([r['acronym'] for r in actual['imported_venues_outside_registry']], ['ICLR'])
+        edition = actual['venues'][0]['editions'][0]
+        self.assertEqual(edition['latest_observed_status_counts']['accepted'], 1)
+        self.assertEqual(edition['latest_observed_status_counts']['unknown'], 1)
+        self.assertEqual(edition['decision_observation_counts']['rejected'], 1)
+        self.assertEqual(edition['papers_without_abstract_text'], 1)
+
     def test_backfill_merges_evidence_without_reimporting_or_losing_history(self):
         files = []
         for source, external, title in [('official', 'legacy', 'First bibliographic name'), ('openreview_snapshot', 'forum', 'Renamed submission')]: