diff --git a/README.md b/README.md index 205294f..b7ce3cd 100644 --- a/README.md +++ b/README.md @@ -117,6 +117,8 @@ CI 在已授权的远程 runner 上运行测试、构建 wheel,并从源码目 已导入整卷的逐条轨道修正见 [edition-tracks 维护流程](docs/edition-tracks.md):按完整证据生成只读计划,保留原投稿和决定 ID,事务性修正 edition、标识与覆盖,并防止旧整卷刷新覆盖分类。 +完整出版来源的新旧版本导致分轨或摘要变化时,使用 [source-revisions 流程](docs/source-revisions.md):核对同一 DOI 全集,保存完整修正前数据与版本链,支持已有记录的目标 track,按新版本重算分区覆盖数,并阻止旧版元数据刷新。普通增量导入仍禁止缩小预期数量。 + 入库前可使用通用的[来源身份只读预检](docs/source-identity-audit.md),导出候选论文、稳定标识和既有投稿决定证据;候选结果需要审核,不会自动批准导入。 RSS 2026 已独立适配官方论文集与录用表;使用方法和与 2025 demo 分类的区别见 [RSS 2026 来源说明](docs/rss2026-proceedings.md)。 diff --git a/docs/source-revisions.md b/docs/source-revisions.md new file mode 100644 index 0000000..971de5d --- /dev/null +++ b/docs/source-revisions.md @@ -0,0 +1,57 @@ +# 出版来源版本与覆盖修正 + +`paper-atlas maintain source-revisions` 用于同一出版清单的新版本:稳定 DOI 全集、标题、作者和决定不变,但官方分轨、摘要或对应来源证据已经过重新核对。它不导入新论文、不合并实体、不推断拒稿,不把出版分类当成研究类型。 + +例如,同一清单的“未分类”由 192 条减少到 166 条,而主会等分区增加。直接放松增量导入的计数保护会隐藏清单丢失。本命令先核验完整的新旧清单及数据库快照,再单独修正版本;普通 `import records` 的单调计数规则保持原样。 + +## 准备完整证据 + +在工作区外保存 inventory JSON,包含以下字段: + +| 字段 | 内容 | +| --- | --- | +| `format` | `1` | +| `source`, `venue`, `year`, `event_key`, `list_url` | 明确的已有来源、会议缩写、年份、事件键和 HTTPS 出版清单地址 | +| `previous_rows`, `rows` | 完整旧版、新版规范化来源记录数组,每版最多 10,000 条;每条包含稳定 DOI、来源字段、作者、摘要、track、决定、标签、原始来源 metadata | +| `evidence` | 一个或多个 `{ "url": "https://…", "sha256": "…" }`,绑定已独立核验的原始证据文件 | + +两版必须有完全相同的 DOI 集合;仅允许 `abstract`、`track`、`tags`、`metadata` 的有据变化。必须保留 collector 原始字段,不能将工作身份核查产生的额外锚点混入原始来源版本。只有明确 `accepted` 的出版观察可进入本流程。证据 URL/hash 是审核输入,本命令不联网确认它们;执行者必须另行验证文件哈希、完整性及对应关系。 + +数据库可以部分导入,甚至各分区的预期数暂时来自不同版本。现有覆盖必须为 `partial`、无错误,导入数与实际来源记录一致;预期数必须恰好属于旧版或新版的同一分区。已有目标 edition 和对应覆盖记录必须存在,目标中可以有其他来源及既有投稿。本版本不会新建 edition,也不会处理共享来源投稿、多次既有 scope curation 的再迁移,或成员集合变化;遇到这些情况会拒绝并要求单独核查。 + +## 先生成只读计划 + +```sh +paper-atlas maintain source-revisions \ + --pg-service foundry-research --pg-schema research \ + --inventory /data/paper-index/validation/reviewed-source-version.json \ + --out /data/paper-index/validation/source-revision-plan.json +``` + +计划包含明确数据库身份、两个完整来源版本、当前版本父节点、覆盖表原值和修正值、逐条变更,以及受影响会议 scope 内所有论文的规范记录、稳定标识、全部跨会议投稿和决定、来源关系、标签与已存在的 annotation。拒稿后录用的历史也在保护范围内,不能只校验当前会议的录用观察。 + +计划和原始证据应保存到服务器与独立恢复位置,核对完整差异及计划文件 SHA-256。执行前重新核对生产保护基线与单写入安排。输出文件必须是不存在的新路径,禁止覆盖旧报告。 + +## 按文件哈希应用并验收 + +```sh +paper-atlas maintain source-revisions \ + --pg-service foundry-research --pg-schema research \ + --plan /data/paper-index/validation/source-revision-plan.json \ + --plan-sha256 <已审核的计划文件SHA256> \ + --out /data/paper-index/validation/source-revision-result.json +``` + +应用时重新生成计划并要求完全一致;PostgreSQL 使用已有 writer 锁及相关表写锁,SQLite 使用立即事务。任何保护数据、当前版本或目标标识冲突都使操作失败。操作保留投稿和决定 ID、原始决定证据、规范论文文本、全部稳定标识、来源关系和现有标签;只修改计划列出的投稿/投稿标识 edition,以及来源记录的摘要与 metadata 当前投影。旧投影完整保存在版本历史中,来源原始获取时间不会伪造为修正时间。 + +覆盖只调整明确分区的预期数和来源观察数,保留 `partial`、cursor、成功时间和错误字段原值。尚未导入的分区不产生虚构投稿或来源观察。覆盖数不能被解读为科学作品数量,或所有 track、摘要、负面决定已完整。 + +同一事务显式增加可共享的可选表 `source_scope_revisions`,保持原 21 张业务表的身份。每个版本存储新来源全集、父版本、完整计划及修正前数据;应用接口只追加版本,不更新或删除旧版本。没有新建 PostgreSQL 实例。该表纳入 SQLite→PostgreSQL 的可选表迁移、全量内容哈希、约束与计数验收。`version_sha256` 是规范化计划内容哈希;CLI 的 `--plan-sha256` 是计划文件字节哈希,二者用途不同。 + +提交前在同一事务比较全部保护数据与明确预期的变更;失败则全部回滚,包括新增表和版本记录。相同计划的重放还会重新验收当前状态;如后续数据已变化则拒绝,不会重复迁移或恢复旧状态。完整计划与 before-image 用于审查和恢复准备,本命令没有自动回滚历史版本的入口;不得手工删除版本来绕开刷新保护。 + +## 后续刷新及限制 + +`import records` 检查当前版本的逐条完整来源字段;`Store.record` 也检查原始文本、作者、来源 metadata 和基础标识,防止直接调用绕回旧版摘要。来源字段必须与当前版本相符,可以额外携带已经核验的工作身份/curation 证据,不能替换来源自身字段。相同来源、会议事件和清单地址下未登记的成员必须另行核验新版本。 + +版本表通过包内接口追加,未声明数据库管理员的任意 SQL 为安全接口。新来源或新版本在导入前仍需要独立工作身份核查;姓名、标题或无候选都不证明作品身份。实现测试和版本修正验收也不替代所有导入结束后的全库去重、标识冲突、近似标题/作者、摘要、来源覆盖与查询审计。 diff --git a/paper_atlas/cli.py b/paper_atlas/cli.py index e4f8590..83c3668 100644 --- a/paper_atlas/cli.py +++ b/paper_atlas/cli.py @@ -46,6 +46,7 @@ COMMANDS = { ('import','arxiv'): ('paper_atlas.ingestion.arxiv', {}, '导入 arXiv 元数据 ZIP'), ('import','records'): ('paper_atlas.ingestion.records', {}, '导入核验过的会议文件'), ('maintain','edition-tracks'): ('paper_atlas.ingestion.edition_tracks', {}, '按逐条来源证据修正既有投稿轨道并保护刷新'), + ('maintain','source-revisions'): ('paper_atlas.ingestion.source_revisions', {}, '按完整来源版本核验并修正出版分轨与覆盖数,保留历史并阻止旧版刷新'), ('maintain','joint-editions'): ('paper_atlas.ingestion.joint_editions', {}, '按已核验计划修正既有联合会议届次归属'), ('maintain','event-identity'): ('paper_atlas.ingestion.event_identity', {}, '按核验计划增加同年会议事件身份'), ('maintain','merge'): ('paper_atlas.ingestion.maintenance', {}, '协调单写入合并与恢复'), diff --git a/paper_atlas/db/source_revisions.py b/paper_atlas/db/source_revisions.py new file mode 100644 index 0000000..e6e71b2 --- /dev/null +++ b/paper_atlas/db/source_revisions.py @@ -0,0 +1,128 @@ +"""Versioned publication-source scopes and guards against stale source refreshes.""" +import json +import re + +TABLE = 'source_scope_revisions' +MARKER = 'source_scope_revisions_v1' +COLUMNS = ('version_sha256', 'source_id', 'venue_id', 'year', 'event_key', 'list_url', + 'parent_sha256', 'records_json', 'plan_json', 'applied_at') + + +def available(c): + if getattr(c, 'postgres', False): + return bool(c.execute("SELECT 1 FROM information_schema.tables WHERE table_schema=current_schema() AND table_name=?", (TABLE,)).fetchone()) + return bool(c.execute("SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (TABLE,)).fetchone()) + + +def initialize(c): + exists = available(c) + marker = bool(c.execute('SELECT 1 FROM schema_migrations WHERE name=?', (MARKER,)).fetchone()) + if exists != marker: + raise ValueError('Source revision table/marker mismatch') + if exists: + result = c.execute('SELECT * FROM source_scope_revisions LIMIT 0') + description = result.cursor.description if getattr(c, 'postgres', False) else result.description + if tuple(x[0] for x in description) != COLUMNS: + raise ValueError('Unexpected source revision schema') + return + ddl = '''CREATE TABLE source_scope_revisions ( + version_sha256 TEXT PRIMARY KEY, + source_id INTEGER NOT NULL REFERENCES sources(id), + venue_id INTEGER NOT NULL REFERENCES venues(id), + year INTEGER NOT NULL, event_key TEXT NOT NULL, list_url TEXT NOT NULL, + parent_sha256 TEXT UNIQUE REFERENCES source_scope_revisions(version_sha256), + records_json TEXT NOT NULL, plan_json TEXT NOT NULL, + applied_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP)''' + if getattr(c, 'postgres', False): + ddl = re.sub(r'\bINTEGER\b', 'BIGINT', ddl) + ddl = re.sub(r'\bTEXT\b', 'TEXT COLLATE "C"', ddl) + c.execute(ddl) + c.execute('CREATE INDEX source_scope_revisions_scope ON source_scope_revisions(source_id,venue_id,year,event_key,list_url)') + c.execute('INSERT INTO schema_migrations(name) VALUES(?)', (MARKER,)) + + +def heads(c, source_id): + if not available(c): + return [] + if not c.execute('SELECT 1 FROM schema_migrations WHERE name=?', (MARKER,)).fetchone(): + raise ValueError('Source revision table/marker mismatch') + return c.execute('''SELECT v.version_sha256,v.venue_id,v.year,v.event_key,v.list_url + FROM source_scope_revisions v WHERE v.source_id=? AND NOT EXISTS + (SELECT 1 FROM source_scope_revisions child WHERE child.parent_sha256=v.version_sha256) + ORDER BY v.version_sha256''', (source_id,)).fetchall() + + +def current_rows(c, source_id, cache): + active = heads(c, source_id) + signatures = tuple(r[0] for r in active) + # A new head invalidates the cache, even when maintenance uses the same Store. + if cache.get('signatures') != (source_id, signatures): + records = {}; scopes = {} + for signature in signatures: + raw = c.execute('SELECT records_json FROM source_scope_revisions WHERE version_sha256=?', (signature,)).fetchone()[0] + scopes[signature] = json.loads(raw) + for row in scopes[signature]: + ext = row['external_id'] + if ext in records and records[ext] != row: + raise ValueError('Conflicting active source snapshots') + records[ext] = row + cache.clear() + cache.update(signatures=(source_id, signatures), records=records, scopes=scopes) + return cache['records'] + + +def subset(expected, actual): + """Reviewed source fields must survive additional work-identity/curation evidence.""" + return isinstance(actual, dict) and all(k in actual and actual[k] == v for k, v in expected.items()) + + +def require_record(c, source_id, external, url, title, abstract, authors, identifiers, metadata, cache): + expected = current_rows(c, source_id, cache).get(external) + if expected is None: + return + actual = {'url': url, 'title': title, 'abstract': abstract, 'authors': authors} + if any(actual[k] != expected.get(k) for k in actual): + raise ValueError('Stale source snapshot: reconcile the source version before refresh') + if not subset(expected.get('metadata', {}), metadata or {}) or not subset(expected.get('identifiers', {}), identifiers or {}): + raise ValueError('Stale source snapshot metadata or identifiers') + + +def scope_head(c, source_id, manifest): + venue = c.execute('SELECT id FROM venues WHERE acronym=?', (manifest.get('venue'),)).fetchone() + if not venue: + return None + matches = [r for r in heads(c, source_id) + if [r[i] for i in range(1, 5)] == [venue[0], manifest.get('year'), manifest.get('event_key', ''), manifest.get('list_url')]] + if len(matches) > 1: + raise ValueError('Forked source revision history') + return matches[0][0] if matches else None + + +def require_manifest(c, source_id, manifest, cache): + current_rows(c, source_id, cache) + signature = scope_head(c, source_id, manifest) + if signature is None: + return + expected = sum(r['track'] == manifest.get('track') for r in cache['scopes'][signature]) + bound = manifest.get('coverage_expected_count') if manifest.get('coverage_mode') == 'incremental' else manifest.get('count') + if expected < 1 or bound != expected or manifest.get('status') != 'partial': + raise ValueError('Source coverage manifest differs from current source snapshot') + + +def require_row(c, source_id, row, manifest, cache): + expected = current_rows(c, source_id, cache).get(row['external_id']) + scoped_manifest = {**manifest, 'venue': row['venue'], 'year': row['year'], 'event_key': row.get('event_key', '')} + signature = scope_head(c, source_id, scoped_manifest) + if expected is None: + if signature is not None: + raise ValueError('Publication absent from current source snapshot; reconcile source membership first') + return + if signature is None or not any(r['external_id'] == row['external_id'] for r in cache['scopes'][signature]): + raise ValueError('Source listing differs from current source snapshot') + for key, value in expected.items(): + if key in ('metadata', 'identifiers'): + okay = subset(value, row.get(key)) + else: + okay = row.get(key) == value + if not okay: + raise ValueError('Stale source snapshot field: ' + key) diff --git a/paper_atlas/db/store.py b/paper_atlas/db/store.py index e625ca1..545c8bb 100644 --- a/paper_atlas/db/store.py +++ b/paper_atlas/db/store.py @@ -96,6 +96,10 @@ class Store: if found: matches.add(found[0]) old = c.execute('SELECT paper_id,title,abstract,retrieved_at,metadata_json FROM source_records WHERE source_id=? AND external_id=?',(source_id:=self.source(source,url),external_id)).fetchone() + from paper_atlas.db.source_revisions import require_record + if not hasattr(self, '_source_revision_cache'): + self._source_revision_cache = {} + require_record(c, source_id, external_id, url, title, abstract, authors, identifiers, metadata, self._source_revision_cache) from paper_atlas.db.edition_scopes import preserve metadata = preserve(dict(metadata or {}), old[4] if old else None) if old: diff --git a/paper_atlas/ingestion/records.py b/paper_atlas/ingestion/records.py index 5b03604..ec49d64 100644 --- a/paper_atlas/ingestion/records.py +++ b/paper_atlas/ingestion/records.py @@ -101,6 +101,9 @@ def ingest_file(db,path, *, pg_service=None, pg_schema=None): c.commit() if incremental is not None: require_incremental_parent(c, source_id, manifest, incremental) + from paper_atlas.db.source_revisions import require_manifest + store._source_revision_cache = {} + require_manifest(c, source_id, manifest, store._source_revision_cache) digest = hashlib.sha256() with path.open('rb') as stream: for line in stream: @@ -113,6 +116,10 @@ def ingest_file(db,path, *, pg_service=None, pg_schema=None): require_track_record(row, manifest) from paper_atlas.db.events import validate_event_key validate_event_key(row.get('event_key')) + from paper_atlas.db.source_revisions import require_row + if not hasattr(store, '_source_revision_cache'): + store._source_revision_cache = {} + require_row(c, source_id, row, manifest, store._source_revision_cache) external = row['external_id'] if external in seen: raise ValueError('Duplicate external ID in one collected file: '+external) diff --git a/paper_atlas/ingestion/source_revisions.py b/paper_atlas/ingestion/source_revisions.py new file mode 100644 index 0000000..28017dc --- /dev/null +++ b/paper_atlas/ingestion/source_revisions.py @@ -0,0 +1,327 @@ +"""Plan-bound publication scope revisions with complete before-images, never reimport.""" +import argparse +from collections import Counter +from contextlib import closing +import copy +import hashlib +import json +from pathlib import Path +import re + +from paper_atlas.db import source_revisions as versions +from paper_atlas.db.backend import database_arguments, open_read_connection, open_store +from paper_atlas.db.edition_scopes import KEY +from paper_atlas.ingestion.edition_tracks import curation +from paper_atlas.ingestion.event_identity import database_identity, digest, values + +MAX_ROWS = 10000 + + +def validate(inventory): + required = {'format', 'source', 'venue', 'year', 'event_key', 'list_url', + 'previous_rows', 'rows', 'evidence'} + if not isinstance(inventory, dict) or set(inventory) != required or inventory['format'] != 1: + raise ValueError('Unsupported source revision inventory') + if (type(inventory['year']) is not int or inventory['year'] < 2000 + or not all(isinstance(inventory[k], str) and inventory[k] for k in ('source', 'venue', 'list_url')) + or not isinstance(inventory['event_key'], str)): + raise ValueError('Explicit publication source scope required') + if not inventory['list_url'].startswith('https://'): + raise ValueError('HTTPS source listing required') + evidence = inventory['evidence'] + if not isinstance(evidence, list) or not 1 <= len(evidence) <= 100: + raise ValueError('Bounded source evidence required') + for item in evidence: + if (not isinstance(item, dict) or set(item) != {'url', 'sha256'} + or not isinstance(item['url'], str) or not item['url'].startswith('https://') + or not isinstance(item['sha256'], str) or not re.fullmatch('[0-9a-f]{64}', item['sha256'])): + raise ValueError('Source evidence URL/hash required') + snapshots = [] + for key in ('previous_rows', 'rows'): + items = inventory[key] + if not isinstance(items, list) or not 1 <= len(items) <= MAX_ROWS: + raise ValueError('Complete bounded source snapshots required') + records = {} + for row in items: + if not isinstance(row, dict) or any(row.get(k) != inventory[k] for k in ('source', 'venue', 'year')): + raise ValueError('Source snapshot scope mismatch') + ext = row.get('external_id') + if (not isinstance(ext, str) or ext in records or row.get('status') != 'accepted' + or not re.fullmatch(r'10\.\d{4,9}/\S+', ext) + or row.get('identifiers', {}).get('doi') != ext + or not isinstance(row.get('track'), str) or not row['track'] + or row.get('event_key', '') != inventory['event_key'] + or not isinstance(row.get('title'), str) or not row['title'] + or not isinstance(row.get('authors'), list) + or not isinstance(row.get('metadata'), dict)): + raise ValueError('Unique DOI-bearing accepted publication records required') + records[ext] = row + snapshots.append(records) + previous, current = snapshots + if set(previous) != set(current): + raise ValueError('Source membership changed; this operation cannot add or remove publications') + for ext, row in current.items(): + before = previous[ext] + stable = (set(before) | set(row)) - {'abstract', 'track', 'tags', 'metadata'} + if any(before.get(k) != row.get(k) for k in stable): + raise ValueError('Stable publication identity or decision changed') + return previous, current + + +def bounded(c, sql, params): + result = values(c.execute(sql + ' LIMIT 100001', params)) + if len(result) > 100000: + raise ValueError('Source revision protection exceeds bounded scope') + return result + + +def capture(c, inventory): + previous, current = validate(inventory) + source = c.execute('SELECT id FROM sources WHERE name=?', (inventory['source'],)).fetchone() + venue = c.execute('SELECT id FROM venues WHERE acronym=?', (inventory['venue'],)).fetchone() + if not source or not venue: + raise ValueError('Existing source and venue required') + sid, vid = source[0], venue[0] + editions = bounded(c, 'SELECT * FROM editions WHERE venue_id=? AND year=? AND event_key=? ORDER BY id', + (vid, inventory['year'], inventory['event_key'])) + by_track = {r[3]: r[0] for r in editions} + # The reviewed scope includes all its existing target attempts, even when + # they originate from another source. No empty-target shortcut is used. + scope_subs = 'SELECT s.id FROM submissions s JOIN editions e ON e.id=s.edition_id WHERE e.venue_id=? AND e.year=? AND e.event_key=?' + scope_params = (vid, inventory['year'], inventory['event_key']) + scoped = bounded(c, 'SELECT * FROM submissions WHERE id IN (' + scope_subs + ') ORDER BY id', scope_params) + own_records = bounded(c, '''SELECT DISTINCT r.* FROM source_records r JOIN submission_sources ss ON ss.source_record_id=r.id + WHERE r.source_id=? AND ss.submission_id IN (''' + scope_subs + ') ORDER BY r.id', (sid, *scope_params)) + if not own_records: + raise ValueError('Existing source observations required') + if any(r[2] not in current for r in own_records): + raise ValueError('Snapshot omits an existing source observation') + pids = sorted({r[1] for r in scoped}) + if not pids or len(pids) > MAX_ROWS: + raise ValueError('Bounded nonempty protected paper scope required') + marks = ','.join('?' for _ in pids) + papers = 'SELECT id FROM papers WHERE id IN (' + marks + ')' + subs = 'SELECT id FROM submissions WHERE paper_id IN (' + papers + ')' + decisions = 'SELECT id FROM decisions WHERE submission_id IN (' + subs + ')' + queries = { + 'papers': 'SELECT * FROM papers WHERE id IN (' + papers + ') ORDER BY id', + 'identifiers': 'SELECT * FROM identifiers WHERE paper_id IN (' + papers + ') ORDER BY scheme,value', + 'source_records': 'SELECT * FROM source_records WHERE paper_id IN (' + papers + ') ORDER BY id', + 'submissions': 'SELECT * FROM submissions WHERE paper_id IN (' + papers + ') ORDER BY id', + 'decisions': 'SELECT * FROM decisions WHERE submission_id IN (' + subs + ') ORDER BY id', + 'submission_sources': 'SELECT * FROM submission_sources WHERE submission_id IN (' + subs + ') ORDER BY submission_id,source_record_id,source_submission_id', + 'submission_identifiers': 'SELECT * FROM submission_identifiers WHERE submission_id IN (' + subs + ') ORDER BY edition_id,scheme,value', + 'decision_sources': 'SELECT * FROM decision_sources WHERE decision_id IN (' + decisions + ') ORDER BY decision_id,source_record_id,source_decision_id', + 'paper_tags': 'SELECT * FROM paper_tags WHERE paper_id IN (' + papers + ') ORDER BY paper_id,tag_id,evidence_key', + } + from paper_atlas.curation.annotations import available + if available(c): + queries['annotation_events'] = 'SELECT * FROM annotation_events WHERE paper_id IN (' + papers + ') ORDER BY id' + packet = {k: bounded(c, q, pids) for k, q in queries.items()} + coverage = bounded(c, '''SELECT es.* FROM edition_sources es JOIN editions e ON e.id=es.edition_id + WHERE es.source_id=? AND e.venue_id=? AND e.year=? AND e.event_key=? ORDER BY es.edition_id,es.list_url''', (sid, *scope_params)) + if not coverage or any(r[2] != inventory['list_url'] for r in coverage): + raise ValueError('Unambiguous source listing required') + if len({r[0] for r in coverage}) != len(coverage): + raise ValueError('One source coverage entry per edition required') + heads = [[r[i] for i in range(len(r))] for r in versions.heads(c, sid)] + own_heads = [r for r in heads if r[1:] == [vid, inventory['year'], inventory['event_key'], inventory['list_url']]] + if len(own_heads) > 1: + raise ValueError('Forked source revision history') + parent = own_heads[0][0] if own_heads else None + return {'source_id': sid, 'venue_id': vid, 'editions': editions, 'coverage': coverage, + 'packet': packet, 'own_records': own_records, 'revision_heads': heads, + 'parent_sha256': parent, 'by_track': by_track, + 'venue_names': sorted({inventory['venue']} | {r[0] for r in c.execute('SELECT alias FROM venue_aliases WHERE venue_id=?', (vid,))})} + + +def changes(c, inventory, before): + previous, current = validate(inventory) + packet = before['packet']; result = []; sid = before['source_id'] + editions = {r[0]: r for r in before['editions']} + for raw in before['own_records']: + ext = raw[2]; old = previous[ext]; new = current[ext] + metadata = json.loads(raw[8]) + candidates = [old, new] + if not any(raw[4] == r['url'] and raw[5] == r['title'] and raw[6] == r.get('abstract') + and json.loads(raw[7]) == r['authors'] and versions.subset(r['metadata'], metadata) for r in candidates): + raise ValueError('Current raw source differs from both reviewed snapshots') + if metadata.get('record_identifiers', {}).get('doi') != ext: + raise ValueError('Current stable DOI differs from source evidence') + attempts = [s for s in packet['submissions'] if s[3] == raw[0] and s[2] in editions] + if len(attempts) != 1: + raise ValueError('Exactly one publication attempt per source record required') + sub = attempts[0]; origin = editions[sub[2]] + if origin[3] not in {old['track'], new['track']} or sub[4] != ext or sub[1] != raw[3]: + raise ValueError('Current source attempt identity/scope differs') + if [x for x in packet['submission_sources'] if x[0] == sub[0]] != [[sub[0], raw[0], ext]]: + raise ValueError('Shared-source attempt requires separate reconciliation') + decisions = [d for d in packet['decisions'] if d[1] == sub[0]] + if not decisions or any(d[2] != 'accepted' for d in decisions): + raise ValueError('Only accepted publication observations can be revised') + target = before['by_track'].get(new['track']) + if target is None: + raise ValueError('Reviewed target edition must already exist') + identities = [x for x in packet['submission_identifiers'] if x[3] == sub[0]] + if not identities or any(x[0] != sub[2] for x in identities): + raise ValueError('Inconsistent scoped submission identifiers') + for _, scheme, value, _ in identities: + conflict = c.execute('SELECT submission_id FROM submission_identifiers WHERE edition_id=? AND scheme=? AND value=?', (target, scheme, value)).fetchone() + if conflict and conflict[0] != sub[0]: + raise ValueError('Target submission identifier collision') + replacement = copy.deepcopy(metadata); replacement.update(new['metadata']) + if sub[2] != target and KEY in metadata: + raise ValueError('An already curated attempt needs explicit scope-chain reconciliation') + if sub[2] != target or raw[6] != new.get('abstract') or replacement != metadata: + result.append({'source_record_id': raw[0], 'paper_id': raw[3], 'submission_id': sub[0], + 'external_id': ext, 'from_edition_id': sub[2], 'target_edition_id': target, + 'from_track': origin[3], 'track': new['track'], + 'new_abstract': new.get('abstract'), 'metadata': replacement}) + old_expected = Counter(r['track'] for r in previous.values()); new_expected = Counter(r['track'] for r in current.values()) + observed = Counter() + own_ids = {r[0] for r in before['own_records']} + for s in packet['submissions']: + if s[3] in own_ids and s[2] in editions: + observed[s[2]] += 1 + coverage = [] + for raw in before['coverage']: + track = editions[raw[0]][3] + if raw[3] != 'partial' or raw[8] is not None or raw[5] != observed[raw[0]]: + raise ValueError('Coverage parent is not the verified partial source observation count') + if raw[4] not in {old_expected[track], new_expected[track]}: + raise ValueError('Coverage expected count differs from both source versions') + count = raw[5] + sum((r['target_edition_id'] == raw[0]) - (r['from_edition_id'] == raw[0]) for r in result) + if count < 0 or count > new_expected[track]: + raise ValueError('Corrected coverage exceeds current source membership') + coverage.append({'before': raw, 'expected_count': new_expected[track], 'imported_count': count}) + covered = {r['before'][0] for r in coverage} + if any(r['target_edition_id'] not in covered for r in result): + raise ValueError('Target source coverage must already exist') + return result, coverage + + +def proposal(c, inventory, identity): + before = capture(c, inventory) + if before['parent_sha256']: + saved = json.loads(c.execute('SELECT records_json FROM source_scope_revisions WHERE version_sha256=?', (before['parent_sha256'],)).fetchone()[0]) + if sorted(saved, key=lambda r: r['external_id']) != sorted(inventory['previous_rows'], key=lambda r: r['external_id']): + raise ValueError('Previous snapshot differs from active source version') + records, coverage = changes(c, inventory, before) + return {'format': 1, 'database': identity, 'inventory': inventory, 'before': before, + 'changes': records, 'coverage': coverage} + + +def verify(c, plan, signature): + after = capture(c, plan['inventory']); expected = copy.deepcopy(plan['before']['packet']) + for change in plan['changes']: + for row in expected['submissions']: + if row[0] == change['submission_id']: + row[2] = change['target_edition_id'] + for row in expected['submission_identifiers']: + if row[3] == change['submission_id']: + row[0] = change['target_edition_id'] + for row in expected['source_records']: + if row[0] == change['source_record_id']: + row[6] = change['new_abstract'] + row[8] = encoded_metadata(plan, change, signature) + expected['submission_identifiers'].sort(key=lambda r: (r[0], r[1], r[2])) + if after['packet'] != expected: + raise ValueError('Protected profiles/history/tags/identifiers differ; roll back') + if after['editions'] != plan['before']['editions']: + raise ValueError('Edition identities changed') + expected_heads = [r for r in plan['before']['revision_heads'] if r[0] != plan['before']['parent_sha256']] + inventory = plan['inventory'] + expected_heads.append([signature, plan['before']['venue_id'], inventory['year'], inventory['event_key'], inventory['list_url']]) + if after['revision_heads'] != sorted(expected_heads): + raise ValueError('Active source version differs') + expected_coverage = [] + for item in plan['coverage']: + row = item['before'][:]; row[4] = item['expected_count']; row[5] = item['imported_count'] + expected_coverage.append(row) + if after['coverage'] != expected_coverage: + raise ValueError('Reconciled coverage differs; roll back') + stored = c.execute('SELECT records_json,plan_json,parent_sha256 FROM source_scope_revisions WHERE version_sha256=?', (signature,)).fetchone() + if (not stored or json.loads(stored[0]) != plan['inventory']['rows'] + or json.loads(stored[1]) != plan or stored[2] != plan['before']['parent_sha256']): + raise ValueError('Versioned source or complete before-images changed') + return {'records_changed': len(plan['changes']), 'events_moved': sum(r['from_edition_id'] != r['target_edition_id'] for r in plan['changes']), + 'protected_papers': len(expected['papers']), 'protected_decisions': len(expected['decisions'])} + + +def encoded_metadata(plan, change, signature): + metadata = copy.deepcopy(change['metadata']) + if change['from_edition_id'] != change['target_edition_id']: + inventory = plan['inventory'] + evidence = inventory['evidence'][0] + row = {**change, 'source': inventory['source'], 'doi': change['external_id'], + 'venue_id': plan['before']['venue_id'], 'year': inventory['year'], + 'event_key': inventory['event_key'], 'venue_names': plan['before']['venue_names'], + 'section': change['track'], 'evidence_url': evidence['url'], 'evidence_sha256': evidence['sha256']} + metadata[KEY] = [curation(row, change['target_edition_id'], signature)] + return json.dumps(metadata, ensure_ascii=False) + + +def apply_plan(c, plan): + if plan.get('format') != 1: + raise ValueError('Unsupported source revision plan') + if not getattr(c, 'postgres', False) and not c.in_transaction: + c.execute('BEGIN IMMEDIATE') + signature = digest(plan) + if versions.available(c) and c.execute('SELECT 1 FROM source_scope_revisions WHERE version_sha256=?', (signature,)).fetchone(): + return {'status': 'already_applied', 'version_sha256': signature, **verify(c, plan, signature)} + if getattr(c, 'postgres', False): + c.execute('LOCK TABLE sources,papers,identifiers,source_records,venues,venue_aliases,editions,edition_sources,submissions,decisions,submission_sources,submission_identifiers,decision_sources,tags,paper_tags IN SHARE ROW EXCLUSIVE MODE') + if proposal(c, plan['inventory'], plan['database']) != plan: + raise ValueError('Stale source revision plan; create and review a fresh plan') + versions.initialize(c) + for row in plan['changes']: + if row['from_edition_id'] != row['target_edition_id']: + c.execute('UPDATE submission_identifiers SET edition_id=? WHERE submission_id=?', (row['target_edition_id'], row['submission_id'])) + c.execute('UPDATE submissions SET edition_id=? WHERE id=? AND edition_id=?', (row['target_edition_id'], row['submission_id'], row['from_edition_id'])) + c.execute('UPDATE source_records SET abstract=?,metadata_json=? WHERE id=?', + (row['new_abstract'], encoded_metadata(plan, row, signature), row['source_record_id'])) + for row in plan['coverage']: + before = row['before'] + c.execute('UPDATE edition_sources SET expected_count=?,imported_count=? WHERE edition_id=? AND source_id=? AND list_url=?', + (row['expected_count'], row['imported_count'], before[0], before[1], before[2])) + inventory = plan['inventory'] + c.execute('''INSERT INTO source_scope_revisions(version_sha256,source_id,venue_id,year,event_key,list_url,parent_sha256,records_json,plan_json) + VALUES(?,?,?,?,?,?,?,?,?)''', (signature, plan['before']['source_id'], plan['before']['venue_id'], inventory['year'], inventory['event_key'], + inventory['list_url'], plan['before']['parent_sha256'], json.dumps(inventory['rows'], ensure_ascii=False), json.dumps(plan, ensure_ascii=False))) + return {'status': 'applied', 'version_sha256': signature, **verify(c, plan, signature)} + + +def run(db, out, *, pg_service=None, pg_schema=None, inventory=None, plan=None, plan_sha256=None): + out = Path(out) + if out.exists() or not out.parent.is_dir(): + raise ValueError('New output in existing directory required') + if bool(inventory) == bool(plan) or bool(plan) != bool(plan_sha256): + raise ValueError('Choose inventory preflight or plan plus exact file hash') + if inventory: + with closing(open_read_connection(db, pg_service=pg_service, pg_schema=pg_schema)) as c: + result = proposal(c, json.loads(Path(inventory).read_bytes()), database_identity(c, db)) + else: + raw = Path(plan).read_bytes() + if hashlib.sha256(raw).hexdigest() != plan_sha256: + raise ValueError('Plan file hash differs') + proposed = json.loads(raw) + with closing(open_store(db, pg_service=pg_service, pg_schema=pg_schema).conn) as c: + try: + if proposed['database'] != database_identity(c, db): + raise ValueError('Plan belongs to another database') + if not getattr(c, 'postgres', False): + c.commit(); c.execute('BEGIN IMMEDIATE') + result = apply_plan(c, proposed); c.commit() + except BaseException: + c.rollback(); raise + out.write_text(json.dumps(result, ensure_ascii=False, indent=2) + '\n') + return result + + +def main(argv=None, *, prog=None): + parser = argparse.ArgumentParser(prog=prog); database_arguments(parser) + parser.add_argument('--inventory'); parser.add_argument('--plan'); parser.add_argument('--plan-sha256'); parser.add_argument('--out', required=True) + args = parser.parse_args(argv) + report = run(args.db, args.out, pg_service=args.pg_service, pg_schema=args.pg_schema, + inventory=args.inventory, plan=args.plan, plan_sha256=args.plan_sha256) + print(json.dumps({'status': report.get('status', 'planned'), 'out': args.out}), flush=True) diff --git a/paper_atlas/postgres/schema.py b/paper_atlas/postgres/schema.py index 0120746..dcc29c6 100644 --- a/paper_atlas/postgres/schema.py +++ b/paper_atlas/postgres/schema.py @@ -8,7 +8,7 @@ TABLES = ('sources', 'papers', 'identifiers', 'source_records', 'venues', 'editi 'submission_sources', 'submission_identifiers', 'decision_sources', 'submission_merges', 'decision_merges', 'schema_migrations', 'tags', 'paper_tags', 'match_candidates', 'paper_merges', 'ingest_runs') -OPTIONAL_TABLES = ('annotation_events',) +OPTIONAL_TABLES = ('annotation_events', 'source_scope_revisions') CHECKS = { 'annotation_events': ["kind IN ('tag','research_type')", 'retracts_id IS NULL OR retracts_id<>id'], 'venues': ["venue_type IN ('conference','journal','review_platform')"], diff --git a/tests/test_source_revisions.py b/tests/test_source_revisions.py new file mode 100644 index 0000000..7d7b982 --- /dev/null +++ b/tests/test_source_revisions.py @@ -0,0 +1,209 @@ +"""Whole-source scope correction, stale refresh, full history and atomicity on both engines.""" +from contextlib import closing, redirect_stdout +import copy +import hashlib +import io +import json +import os +from pathlib import Path +import tempfile +import unittest +from unittest.mock import patch +import uuid + +from paper_atlas.db.backend import open_store +from paper_atlas.db.edition_scopes import KEY +from paper_atlas.db.source_revisions import available, require_row, require_manifest +from paper_atlas.db.store import Store +from paper_atlas.ingestion.source_revisions import proposal, apply_plan, run, validate +from paper_atlas.ingestion.event_identity import database_identity, digest, values +from paper_atlas.ingestion.records import coverage_options, ingest_file +from paper_atlas.postgres.client import connect +from paper_atlas.postgres.migration import migrate +from paper_atlas.postgres.schema import quote + +URL = 'https://example.org/volume' + + +def seed(path): + old = [] + for i in range(4): + doi = '10.1234/publication'+str(i) + old.append({'source': 'fixture', 'venue': 'MM', 'year': 2026, 'track': 'main' if i == 0 else 'unclassified', + 'external_id': doi, 'url': 'https://doi.org/'+doi, 'title': 'Distinct publication '+str(i), + 'authors': ['Alice Author', 'Bob Writer'], 'abstract': None if i == 1 else 'Source abstract '+str(i), + 'identifiers': {'doi': doi}, 'submission_identifiers': {'publication_doi': doi}, + 'status': 'accepted', 'raw_decision': 'Publication observation', 'evidence_url': URL, + 'observed_at': '2026-06-01T00:00:00Z', 'tags': [{'namespace': 'fixture:track', 'value': 'main' if i == 0 else 'unclassified'}], + 'metadata': {'original': i, 'program': None, 'scientific_work_identity_approved': False}}) + current = copy.deepcopy(old) + current[1].update(track='main', abstract='Reviewed program abstract with NUL\x00 and SOH\x01') + current[1]['metadata']['program'] = {'id': 'p1', 'evidence_url': URL} + current[1]['tags'][0]['value'] = 'main' + store = Store(path) + for row in old[:3]: + p, r = store.record(row['source'], row['external_id'], row['url'], row['title'], row['abstract'], row['authors'], row['identifiers'], row['metadata']) + store.decision(p, r, 'MM', 2026, row['track'], row['external_id'], URL, 'accepted', row['raw_decision'], + identities=[('publication_doi', row['external_id'])], observed_at=row['observed_at']) + store.record_tags(p, r, 'fixture', row['tags']) + if row == old[1]: + store.conn.execute('UPDATE papers SET abstract=? WHERE id=?', ('Canonical preprint abstract', p)) + p, other = store.record('prior', 'old-forum', 'https://example.org/prior', row['title'], 'Canonical preprint abstract', row['authors'], row['identifiers']) + store.decision(p, other, 'PRIOR', 2025, 'main', 'old-forum', 'https://example.org/prior', 'rejected', 'Reject', observed_at='2025-01-01T00:00:00Z') + # An unrelated source already has an attempt in the populated target edition. + p, r = store.record('other', 'other', 'https://example.org/other', 'Unrelated target paper', 'Other abstract', ['Someone Else']) + store.decision(p, r, 'MM', 2026, 'main', 'other', 'https://example.org/other', 'accepted', 'Published') + sid = store.conn.execute("SELECT id FROM sources WHERE name='fixture'").fetchone()[0] + for track, expected, imported in [('main', 1, 1), ('unclassified', 3, 2)]: + eid = store.conn.execute('SELECT id FROM editions WHERE year=2026 AND track=?', (track,)).fetchone()[0] + store.conn.execute("INSERT INTO edition_sources VALUES(?,?,?,'partial',?,?,?,'2026-06-01',NULL)", (eid, sid, URL, expected, imported, 'checkpoint')) + store.conn.commit(); store.conn.close() + return {'format': 1, 'source': 'fixture', 'venue': 'MM', 'year': 2026, 'event_key': '', 'list_url': URL, + 'previous_rows': old, 'rows': current, 'evidence': [{'url': URL, 'sha256': 'a'*64}]} + + +class RevisionCases: + def store(self): + return open_store(**self.target) + + def plan(self, c): + return proposal(c, self.inventory, database_identity(c, self.target.get('db'))) + + def test_move_to_populated_target_preserves_full_history_and_before_images(self): + with closing(self.store().conn) as c: + plan = self.plan(c); self.assertEqual(len(plan['changes']), 1) + self.assertEqual(len(plan['before']['packet']['submissions']), 5) + before_papers = plan['before']['packet']['papers'] + before_tags = plan['before']['packet']['paper_tags'] + result = apply_plan(c, plan); c.commit() + self.assertEqual((result['events_moved'], result['records_changed']), (1, 1)) + self.assertEqual(result['protected_decisions'], 5) + self.assertEqual(values(c.execute('SELECT * FROM papers ORDER BY id')), before_papers) + self.assertEqual(values(c.execute('SELECT * FROM paper_tags ORDER BY paper_id,tag_id,evidence_key')), before_tags) + self.assertEqual(c.execute("SELECT count(*) FROM decisions WHERE status='rejected'").fetchone()[0], 1) + self.assertEqual(values(c.execute('SELECT status,expected_count,imported_count,cursor,last_success_at,last_error FROM edition_sources ORDER BY edition_id')), + [['partial', 2, 2, 'checkpoint', '2026-06-01', None], ['partial', 2, 1, 'checkpoint', '2026-06-01', None]]) + saved = json.loads(c.execute('SELECT plan_json FROM source_scope_revisions').fetchone()[0]) + self.assertEqual(saved, plan) + self.assertEqual(apply_plan(c, plan)['status'], 'already_applied') + self.assertEqual(c.execute('SELECT count(*) FROM source_scope_revisions').fetchone()[0], 1) + + def test_stale_direct_store_and_jsonl_refresh_fail_but_current_refresh_succeeds(self): + with closing(self.store().conn) as c: + plan = self.plan(c); apply_plan(c, plan); c.commit() + with closing(self.store().conn) as c: + store = object.__new__(Store); store.conn = c; store.event_identity = True + old = self.inventory['previous_rows'][1] + with self.assertRaisesRegex(ValueError, 'Stale source'): + store.record('fixture', old['external_id'], old['url'], old['title'], old['abstract'], old['authors'], old['identifiers'], old['metadata']) + c.rollback() + row = self.inventory['rows'][1] + p, r = store.record('fixture', row['external_id'], row['url'], row['title'], row['abstract'], row['authors'], row['identifiers'], row['metadata']) + self.assertEqual(c.execute('SELECT abstract FROM papers WHERE id=?', (p,)).fetchone()[0], 'Canonical preprint abstract') + self.assertEqual(json.loads(c.execute('SELECT metadata_json FROM source_records WHERE id=?', (r,)).fetchone()[0])[KEY][0]['target']['track'], 'main') + c.rollback() + path = self.root/'old.jsonl'; raw = (json.dumps(old)+'\n').encode(); path.write_bytes(raw) + path.with_suffix('.manifest.json').write_text(json.dumps({'source': 'fixture', 'venue': 'MM', 'year': 2026, + 'track': 'unclassified', 'list_url': URL, 'status': 'partial', 'count': 1, 'file_sha256': hashlib.sha256(raw).hexdigest()})) + with self.assertRaisesRegex(ValueError, 'source snapshot'): + ingest_file(self.target.get('db'), path, **{k: v for k, v in self.target.items() if k != 'db'}) + + def test_unregistered_member_and_changed_program_metadata_are_rejected(self): + with closing(self.store().conn) as c: + apply_plan(c, self.plan(c)); c.commit() + sid = c.execute("SELECT id FROM sources WHERE name='fixture'").fetchone()[0] + cache = {}; unknown = copy.deepcopy(self.inventory['rows'][0]); unknown['external_id'] = '10.1234/unlisted' + with self.assertRaisesRegex(ValueError, 'absent from current'): + require_row(c, sid, unknown, {'list_url': URL}, cache) + bad = copy.deepcopy(self.inventory['rows'][1]); bad['metadata']['program'] = None + with self.assertRaisesRegex(ValueError, 'Stale source'): + require_row(c, sid, bad, {'list_url': URL}, cache) + good = copy.deepcopy(self.inventory['rows'][1]); good['metadata']['verified_work_anchor'] = {'reason': 'extra evidence'} + require_row(c, sid, good, {'list_url': URL}, cache) + with self.assertRaisesRegex(ValueError, 'coverage manifest'): + require_manifest(c, sid, {'venue': 'MM', 'year': 2026, 'track': 'main', 'list_url': URL, 'count': 1, 'status': 'partial'}, cache) + require_manifest(c, sid, {'venue': 'MM', 'year': 2026, 'track': 'main', 'list_url': URL, 'count': 2, 'status': 'partial'}, cache) + + def test_stale_plan_and_target_identifier_collision_prevent_writes(self): + with closing(self.store().conn) as c: + plan = self.plan(c) + c.execute("UPDATE decisions SET raw_value='Additional evidence' WHERE status='rejected'") + with self.assertRaisesRegex(ValueError, 'Stale'): + apply_plan(c, plan) + c.rollback() + move = plan['changes'][0] + target_sub = c.execute('SELECT id FROM submissions WHERE edition_id=? LIMIT 1', (move['target_edition_id'],)).fetchone()[0] + c.execute('INSERT INTO submission_identifiers VALUES(?,?,?,?)', (move['target_edition_id'], 'publication_doi', move['external_id'], target_sub)) + with self.assertRaisesRegex(ValueError, 'collision'): + self.plan(c) + c.rollback() + self.assertFalse(available(c)) + + def test_failed_postcheck_rolls_back_rows_extension_and_coverage(self): + with closing(self.store().conn) as c: + plan = self.plan(c) + with patch('paper_atlas.ingestion.source_revisions.verify', side_effect=ValueError('Injected verification failure')): + with self.assertRaisesRegex(ValueError, 'Injected'): + apply_plan(c, plan) + c.rollback() + self.assertFalse(available(c)) + self.assertEqual(self.plan(c), plan) + + def test_cli_binds_database_and_file_hash_without_writing_during_preflight(self): + inventory = self.root/'inventory.json'; inventory.write_text(json.dumps(self.inventory)) + path = self.root/'plan.json' + args = {k: v for k, v in self.target.items() if k != 'db'} + run(self.target.get('db'), path, inventory=inventory, **args) + with closing(self.store().conn) as c: + self.assertFalse(available(c)) + with self.assertRaisesRegex(ValueError, 'hash'): + run(self.target.get('db'), self.root/'bad.json', plan=path, plan_sha256='0'*64, **args) + report = run(self.target.get('db'), self.root/'applied.json', plan=path, plan_sha256=hashlib.sha256(path.read_bytes()).hexdigest(), **args) + self.assertEqual(report['events_moved'], 1) + + def test_membership_identity_changes_and_incremental_shrink_remain_forbidden(self): + for mutate in [lambda x: x['rows'].pop(), lambda x: x['rows'][0].update(title='Different work'), + lambda x: x['rows'][0].update(status='rejected')]: + bad = copy.deepcopy(self.inventory); mutate(bad) + with self.assertRaises(ValueError): + validate(bad) + with self.assertRaisesRegex(ValueError, 'incremental'): + coverage_options({'coverage_mode': 'incremental', 'venue': 'MM', 'year': 2026, 'track': 'unclassified', + 'status': 'partial', 'count': 1, 'coverage_previous_count': 2, + 'coverage_previous_expected_count': 3, 'coverage_expected_count': 2}) + + +class SQLiteSourceRevisionTests(RevisionCases, unittest.TestCase): + def setUp(self): + temp = tempfile.TemporaryDirectory(); self.addCleanup(temp.cleanup); self.root = Path(temp.name) + db = self.root/'source.sqlite'; self.inventory = seed(db); self.target = {'db': db} + + @unittest.skipUnless(os.environ.get('PAPER_ATLAS_TEST_PG_SERVICE'), 'Requires approved remote PostgreSQL service') + def test_optional_revision_table_migrates_with_full_history(self): + with closing(self.store().conn) as c: + plan = self.plan(c); apply_plan(c, plan); c.commit() + service = os.environ['PAPER_ATLAS_TEST_PG_SERVICE']; schema = 'pa_rev_copy_'+uuid.uuid4().hex[:10] + self.addCleanup(drop_schema, service, schema) + with redirect_stdout(io.StringIO()): + migrate(self.target['db'], service, self.root/'copied.json', schema=schema, min_free_gib=0) + with closing(open_store(pg_service=service, pg_schema=schema).conn) as c: + self.assertEqual(json.loads(c.execute('SELECT plan_json FROM source_scope_revisions').fetchone()[0]), plan) + self.assertEqual(c.execute('SELECT count(*) FROM source_scope_revisions').fetchone()[0], 1) + + +def drop_schema(service, schema): + with connect(service) as c: + for name, in c.execute('SELECT nspname FROM pg_namespace WHERE starts_with(nspname,%s)', (schema,)): + c.execute('DROP SCHEMA '+quote(name)+' CASCADE') + + +@unittest.skipUnless(os.environ.get('PAPER_ATLAS_TEST_PG_SERVICE'), 'Requires approved remote PostgreSQL service') +class PostgreSQLSourceRevisionTests(RevisionCases, unittest.TestCase): + def setUp(self): + temp = tempfile.TemporaryDirectory(); self.addCleanup(temp.cleanup); self.root = Path(temp.name) + db = self.root/'source.sqlite'; self.inventory = seed(db) + service = os.environ['PAPER_ATLAS_TEST_PG_SERVICE']; schema = 'pa_rev_'+uuid.uuid4().hex[:12] + self.addCleanup(drop_schema, service, schema) + with redirect_stdout(io.StringIO()): + migrate(db, service, self.root/'migration.json', schema=schema, min_free_gib=0) + self.target = {'pg_service': service, 'pg_schema': schema}