Compare commits
10 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| afdc0c4021 | |||
| cac842e5e5 | |||
| b1429cd3d8 | |||
| 2988e4e42b | |||
| 4c834dc10e | |||
| b535cc556b | |||
| dc4f61d2f6 | |||
| 04b2ed9e95 | |||
| 5d858ab8e2 | |||
| bf4d0cd38a |
@@ -134,6 +134,7 @@ jobs:
|
||||
assert all(r['status'] == 'unknown' and not r['identifiers'] and not r['contribution_type_verified'] and not r['publication_identity_verified'] and not r['final_decision_verified'] for r in mm_preflight['entries'])
|
||||
subprocess.run([str(venv/'bin/paper-atlas'),'--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','cvf','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','rss-proceedings','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','rss2026-proceedings','--help'],cwd=outside,check=True)
|
||||
eccv_fixture = json.loads(gzip.decompress((src/'tests/fixtures/eccv2026-source.json.gz').read_bytes()))
|
||||
eccv_official = root/'eccv2026-official'
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
Paper Atlas 将 arXiv、AI 会议论文集及公开投稿决定整理到可追溯的数据库中,用于查询和后续研究分析。生产数据使用 PostgreSQL,支持从既有 SQLite 索引完整迁移;SQLite 保留离线核验和小型样本用途。它按批次运行,不需要常驻 Web 服务;采集标题、摘要、标识、标签及决定证据,不下载论文全文。
|
||||
|
||||
项目仍在建设中。候选范围包含 43 个会议(含 ACM MM、MLSys),尚未完成全部历史数据和最终去重复核。实际进度以数据库、来源清单和覆盖报告为准,而不是仓库文件数量。
|
||||
历史目录包含 43 个会议。主动维护以核心 AI 会议及其已公开的决定为主,继续跟踪 ACM MM 和 MLSys。此前入库的 CIKM 数据与证据保留,后续主动补漏范围排除 CIKM。实际覆盖以数据库和来源清单为准。
|
||||
|
||||
当前主动补漏从 **2020 年**起;2000 年起已发现的名单、缺口与待办保存在[补漏恢复清单](docs/coverage/backfill-handoff-2026-11-10.md),下次从这里接续。已有旧数据保留,范围排除不算补齐。
|
||||
|
||||
@@ -108,6 +108,8 @@ AAMAS 的[官方论文集采集器](docs/aamas-proceedings.md)支持已核验的
|
||||
|
||||
CI 在已授权的远程 runner 上运行测试、构建 wheel,并从源码目录外验证安装后的命令及随包资源。数据库全量去重和覆盖率是独立的数据验收步骤,不因代码测试通过而自动完成。
|
||||
|
||||
一批数据全部验收后,运行 [`audit global-identity`](docs/operations.md#导入后的全库身份核验) 核对 PostgreSQL 的稳定标识、同题与近似标题候选、摘要完整性和全文检索。审计报告和候选文件对应一次快照,后续导入完成后重新生成。使用 `audit histories` 检查跨会议投稿决定及其来源关系。
|
||||
|
||||
开发通过工作分支和 PR 交付,由用户手动合并。
|
||||
|
||||
早期WWW档案的可重放元数据采集、论文/海报分轨与待核查条目见 [WWW档案采集](docs/www-archive.md)。
|
||||
@@ -123,6 +125,8 @@ CI 在已授权的远程 runner 上运行测试、构建 wheel,并从源码目
|
||||
|
||||
入库前可使用通用的[来源身份只读预检](docs/source-identity-audit.md),导出候选论文、稳定标识和既有投稿决定证据;候选结果需要审核,不会自动批准导入。
|
||||
|
||||
RSS 2025 的 `collect rss-proceedings` 读取官方逐篇标题、完整作者、摘要和 DOI,并核对录用与演示节目单;不下载 PDF。15 个演示 ID 有明确来源,其余逐篇投稿类型保持未明确,不用标题或会话主题代替 track。运行、证据恢复和限制见 [RSS 元数据说明](docs/rss-proceedings.md)。
|
||||
|
||||
RSS 2026 已独立适配官方论文集与录用表;使用方法和与 2025 demo 分类的区别见 [RSS 2026 来源说明](docs/rss2026-proceedings.md)。
|
||||
|
||||
ICAPS 2026 的[官方名单采集器](docs/icaps2026-program.md)区分 90 篇主会轨道组录用与 10 篇既有论文展示;逐篇 track、摘要和稳定标识尚未解决,输出均待审核,不能直接视作已完成入库。
|
||||
|
||||
@@ -1,5 +1,21 @@
|
||||
# 运行维护与数据核验
|
||||
|
||||
## 导入后的全库身份核验
|
||||
|
||||
在服务器准备一个至少有 2 GiB 可用空间的审计目录。使用生产读取角色,在计划中的导入全部验收后运行以下命令。命令在 PostgreSQL 的同一个可重复读快照内检查库内记录。SQLite 临时文件包含用来比较的标题键与作者指纹,扫描结束后清理。
|
||||
|
||||
```bash
|
||||
paper-atlas audit global-identity \
|
||||
--pg-service foundry-research --pg-schema research \
|
||||
--scratch-root /data/paper-index/validation \
|
||||
--candidate-output /data/paper-index/validation/identity-candidates-2026-11-22.jsonl \
|
||||
> /data/paper-index/validation/identity-report-2026-11-22.json
|
||||
```
|
||||
|
||||
JSON 报告记录快照时间、论文与缺摘要计数、稳定标识冲突、约束状态、会议年度与轨道的来源覆盖,以及一次实际的全文搜索和执行计划。JSONL 保存同题且规范化作者集合相同的论文组,以及相同作者集合中标题相似度至少为 0.90 的论文对。超过 100 条记录的作者组会列入报告,留作后续逐组核对。缺作者、作者增减或换名的版本需要结合 DOI、论坛或作者说明,使用 `audit source-identities` 和原始来源复核。
|
||||
|
||||
将报告、候选文件、运行版本与 SHA-256 一起保留。使用 `audit histories` 核对拒稿、撤稿、后续录用和来源链接。需要合并候选时先确认论文版本关系并保存投稿、决定、标签及来源证据,再按已授权的流程操作。新一批导入完成后更新快照和审计报告。
|
||||
|
||||
## PostgreSQL 迁移
|
||||
|
||||
共享实例的容器名为 `foundry-postgres`,底层镜像为 `postgres:17-alpine`;Compose 服务名仍为 `postgres`。数据库和 schema 是可配置的逻辑边界,表按研究数据域组织,不按调用项目复制。其他项目可获授 `research` 的 USAGE 及必要表权限,复用相同论文 ID 和表。未来换库只更新 libpq service;换 schema 可更新角色 search_path 或查询的 `--pg-schema`。
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
# RSS 官方论文元数据
|
||||
|
||||
这是长期维护、按需运行的 package collector,不是定时服务或一次性导入脚本。目前仅接受独立核对过的 2025 年版式;不把 2026 年或其他年份套用同一分类规则。
|
||||
|
||||
## 数据范围和证据
|
||||
|
||||
- [RSS XXI 官方论文集](https://www.roboticsproceedings.org/rss21/)列出 163 个不同的逐篇 HTML 链接。每篇有 citation 元数据、BibTeX 和摘要区;PDF 链接作为原始元数据保存,不请求全文。
|
||||
- [官方 Accepted Papers](https://roboticsconference.org/2025/program/papers/)按论文 ID、标题和 session 对账。网页作者列包含隐藏的重复展示,逐篇有序 citation_author 与论文集目录作者完整对比后才采用;不截断作者、不将重复展示当新论文。
|
||||
- [官方 Demos](https://roboticsconference.org/2025/program/demos/)明确列出 15 个论文 ID,包括标题不以 Demonstrating 开头的 ID 93。这些条目标为 `demo`,保留整行节目证据。其他 148 条使用 `proceedings_unspecified`:没有逐篇类型依据时,不因缺席演示表就自动推断 main。
|
||||
- [作者说明](https://roboticsconference.org/2025/information/authorinfo/)描述 Science/Systems 与 Demo 两类投稿,两类都进入论文集。分类属于该来源的投稿/发表语义,不自动生成通用 research_type 标签。
|
||||
|
||||
`rss_proceedings:session` 保留官方 session,`rss_proceedings:program_role` 保留明确的 demonstration。二者与其他平台标签隔离;session 主题不等于投稿 track。单篇来源 ID 是 `2025/pNNN`,DOI 同时保留原始 BibTeX 大小写和标准化身份键,显式的 `publication_doi` 供原生导入器在同一会议年度/track 中关联其他正式发表来源;不能据此合并不同 track 或不同会议的投稿经历。
|
||||
|
||||
只有已发表、被官方录用表列出的条目才产生 accepted 记录。采集观察时间来自响应回执;出版日期单独留在 metadata,不伪造原始决定日期。来源未列出的论文不能推断为 rejected、desk rejected 或 withdrawn;公开负面决定是另一个待核范围。
|
||||
|
||||
## 运行
|
||||
|
||||
构建、安装和项目测试仅使用已确认的 `ces-control:host` 远程 runner。本机不能执行隐式构建的验证。服务器已安装环境运行示例:
|
||||
|
||||
```sh
|
||||
paper-atlas collect rss-proceedings --year 2025 \
|
||||
--snapshots /data/paper-index/collected/rss2025-full-evidence-v1 \
|
||||
--out /data/paper-index/collected/rss2025-source-v1
|
||||
```
|
||||
|
||||
也可使用 `--cache /data/paper-index/raw/rss-http` 请求限定官方 HTML 元数据。两种输入互斥。冻结快照的 `receipts.json` 每项必须包含 `file`、准确 `url/final_url`、`http_status: 200`、`content_type: text/html`、`bytes`、`sha256` 和含时区的观察时间 `at`。路径必须位于快照内,不能通过符号链接逃逸。所有输入逐个核验字节数与哈希,UTF-8 解码;重定向、未知版式和异常 ID 需要先核查。
|
||||
|
||||
采集只输出 JSONL、每个 track 的 manifest、完整 HTML 证据和 `rss-2025.json` 来源报告,不连接生产数据库。两份清单状态保持 `partial`;即使该卷 163 个链接全部解析,也不表示独立作品去重、逐篇类型或公开负面决定全集已完成。明确为空的摘要保存 `null` 和原因;缺失或重复摘要边界属于版式异常,停止输出。
|
||||
|
||||
## 冻结与恢复
|
||||
|
||||
2026-11-09 在服务器冻结 167 个官方 HTML 响应:目录、录用表、演示表、作者说明和 163 个逐篇页面。回执 SHA256:`f427d93a113702030864e789efe1176d69f71d2c567e81536d155e53d9513a93`。六个未改写样例位于 `tests/fixtures/rss2025/`,附其真实响应回执;完整原件保存在服务器来源目录与独立审阅备份,不作为运行时下载依赖。
|
||||
|
||||
整个采集持有输出目录的排他锁。全部解析和交叉检查通过后才发布 JSONL、manifest 和完成报告;完成报告最后发布。已有完成报告或 JSONL 时拒绝覆盖。若发布中断,先核对目录、回执和所有文件哈希,保留现场,再换新输出目录重采;不能看到日志安静就重启或重复导入。新的官方页面内容或计数与冻结样本不同,必须另留快照、解释变化并验证后再更新采集器。
|
||||
|
||||
生产导入前必须另外做来源独立核验和 PostgreSQL 只读身份预检。DOI、同标题候选和既有 arXiv 实体需要逐项核查,保留跨会的所有投稿和决定关系;发现冲突时隔离,不能只按标题强合并。已导入队列不得重放。生产独立验收通过后才重新统计年度缺口;导入成功和 CI 不能替代最终全库去重、摘要/标签及约束和实际查询审计。
|
||||
|
||||
本说明中的 163/15 是冻结官方页面的来源清单,不是当前生产库导入结果。历史缺口在本批生产验收前仍为 47。
|
||||
@@ -48,6 +48,8 @@ paper-atlas maintain source-revisions \
|
||||
|
||||
同一事务显式增加可共享的可选表 `source_scope_revisions`,保持原 21 张业务表的身份。每个版本存储新来源全集、父版本、完整计划及修正前数据;应用接口只追加版本,不更新或删除旧版本。没有新建 PostgreSQL 实例。该表纳入 SQLite→PostgreSQL 的可选表迁移、全量内容哈希、约束与计数验收。`version_sha256` 是规范化计划内容哈希;CLI 的 `--plan-sha256` 是计划文件字节哈希,二者用途不同。
|
||||
|
||||
源快照和计划各自在 UTF-8 JSON 达到 1 MiB 时由客户端压缩后写入,避免数据库在接收大参数时出现内存峰值。读取兼容既有纯 JSON 和带 `gzip+base64:v1:` 前缀的压缩值,完整内容和计划哈希均可按原口径核验。使用该表执行写入或刷新时,运行时需要支持两种存储格式。
|
||||
|
||||
提交前在同一事务比较全部保护数据与明确预期的变更;失败则全部回滚,包括新增表和版本记录。相同计划的重放还会重新验收当前状态;如后续数据已变化则拒绝,不会重复迁移或恢复旧状态。完整计划与 before-image 用于审查和恢复准备,本命令没有自动回滚历史版本的入口;不得手工删除版本来绕开刷新保护。
|
||||
|
||||
## 后续刷新及限制
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
"""Read-only PostgreSQL identity and integrity audit for an existing snapshot."""
|
||||
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from contextlib import closing
|
||||
from difflib import SequenceMatcher
|
||||
from hashlib import sha256
|
||||
from itertools import combinations, groupby
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import sqlite3
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
from paper_atlas.db.store import author_keys
|
||||
from paper_atlas.postgres.client import ReadConnection, SEARCH_VECTOR
|
||||
|
||||
|
||||
ROSTER_GROUP_LIMIT = 100
|
||||
TITLE_SCORE = .90
|
||||
|
||||
|
||||
def _scalar(connection, query, params=()):
|
||||
return connection.execute(query, params).fetchone()[0]
|
||||
|
||||
|
||||
def _checks(connection):
|
||||
counts = {}
|
||||
for table, columns in (
|
||||
('identifiers', 'scheme,value'),
|
||||
('source_records', 'source_id,external_id'),
|
||||
('submission_identifiers', 'edition_id,scheme,value'),
|
||||
):
|
||||
counts[f'duplicate_{table}_keys'] = _scalar(connection, f'''
|
||||
SELECT count(*) FROM (
|
||||
SELECT {columns} FROM {table} GROUP BY {columns} HAVING count(*)>1
|
||||
) duplicates''')
|
||||
counts['normalized_doi_collisions'] = _scalar(connection, '''
|
||||
SELECT count(*) FROM (
|
||||
SELECT lower(regexp_replace(btrim(value),
|
||||
?, '', 'i')) AS doi
|
||||
FROM identifiers WHERE lower(scheme)='doi' GROUP BY 1
|
||||
HAVING count(DISTINCT paper_id)>1
|
||||
) duplicates''', (r'^(https?://(dx\.)?doi\.org/|doi:)[[:space:]]*',))
|
||||
counts['multiple_papers_per_base_arxiv_id'] = _scalar(connection, r'''
|
||||
SELECT count(*) FROM (
|
||||
SELECT regexp_replace(btrim(value),'v[0-9]+$','','i') AS arxiv_id
|
||||
FROM identifiers WHERE lower(scheme)='arxiv' GROUP BY 1
|
||||
HAVING count(DISTINCT paper_id)>1
|
||||
) duplicates''')
|
||||
counts['submission_identifier_wrong_edition'] = _scalar(connection, '''
|
||||
SELECT count(*) FROM submission_identifiers i
|
||||
JOIN submissions s ON s.id=i.submission_id WHERE s.edition_id<>i.edition_id''')
|
||||
counts['unvalidated_foreign_key_or_check_constraints'] = _scalar(connection, '''
|
||||
SELECT count(*) FROM pg_constraint k JOIN pg_namespace n ON n.oid=k.connamespace
|
||||
WHERE n.nspname=current_schema() AND k.contype IN ('f','c') AND NOT k.convalidated''')
|
||||
counts['tag_wrong_source_paper'] = _scalar(connection, '''
|
||||
SELECT count(*) FROM paper_tags t JOIN source_records r ON r.id=t.source_record_id
|
||||
WHERE t.paper_id<>r.paper_id''')
|
||||
return counts
|
||||
|
||||
|
||||
def _coverage(connection):
|
||||
return [dict(row) for row in connection.execute('''
|
||||
SELECT v.acronym AS venue,e.year,e.track,e.event_key,o.name AS source,
|
||||
l.list_url,l.status,l.expected_count,l.imported_count
|
||||
FROM edition_sources l JOIN editions e ON e.id=l.edition_id
|
||||
JOIN venues v ON v.id=e.venue_id JOIN sources o ON o.id=l.source_id
|
||||
ORDER BY v.acronym,e.year,e.track,e.event_key,o.name,l.list_url''')]
|
||||
|
||||
|
||||
def _fts(connection):
|
||||
indexed = bool(_scalar(connection, '''
|
||||
SELECT count(*) FROM pg_index i JOIN pg_class t ON t.oid=i.indrelid
|
||||
JOIN pg_class x ON x.oid=i.indexrelid JOIN pg_am a ON a.oid=x.relam
|
||||
JOIN pg_namespace n ON n.oid=t.relnamespace
|
||||
WHERE n.nspname=current_schema() AND t.relname='papers' AND a.amname='gin'
|
||||
AND i.indisvalid AND i.indisready AND x.relname='papers_search_gin'
|
||||
AND strpos(pg_get_indexdef(i.indexrelid),'to_tsvector')>0'''))
|
||||
query = 'SELECT id FROM papers WHERE ' + SEARCH_VECTOR + " @@ websearch_to_tsquery('english',?) LIMIT 5"
|
||||
phrase = 'convolutional vision transformer'
|
||||
plan = [row[0] for row in connection.execute('EXPLAIN (COSTS OFF) '+query, (phrase,))]
|
||||
ids = [row[0] for row in connection.execute(query, (phrase,))]
|
||||
return {'gin_index_ready': indexed, 'plan': plan, 'matched_paper_ids': ids, 'query': phrase}
|
||||
|
||||
|
||||
def _author_fingerprint(raw):
|
||||
authors = sorted(author_keys(json.loads(raw)))
|
||||
if not authors:
|
||||
return None
|
||||
return sha256(json.dumps(authors, ensure_ascii=False).encode()).digest()
|
||||
|
||||
|
||||
def _stage_titles(connection, scratch):
|
||||
scratch.execute('CREATE TABLE titles (id INTEGER PRIMARY KEY, title_key TEXT NOT NULL, authors BLOB)')
|
||||
counts = Counter(papers=0, missing_abstract=0, missing_author_keys=0, invalid_authors_json=0)
|
||||
with connection.connection.cursor(name='paper_atlas_global_identity_scan') as titles:
|
||||
titles.itersize = 5000
|
||||
titles.execute('SELECT id,title_key,authors_json,abstract IS NULL OR btrim(abstract)=\'\' FROM papers ORDER BY id')
|
||||
while rows := titles.fetchmany(5000):
|
||||
batch = []
|
||||
for row in rows:
|
||||
paper_id,title_key,authors,missing = (row[index] for index in range(4))
|
||||
try:
|
||||
fingerprint = _author_fingerprint(authors)
|
||||
except (TypeError, ValueError):
|
||||
fingerprint = None
|
||||
counts['invalid_authors_json'] += 1
|
||||
counts['papers'] += 1
|
||||
counts['missing_abstract'] += int(missing)
|
||||
counts['missing_author_keys'] += int(fingerprint is None)
|
||||
batch.append((paper_id,title_key,fingerprint))
|
||||
scratch.executemany('INSERT INTO titles VALUES (?,?,?)', batch)
|
||||
scratch.commit()
|
||||
scratch.execute('CREATE INDEX keyed_titles ON titles (title_key,authors,id)')
|
||||
scratch.execute('CREATE INDEX title_rosters ON titles (authors,title_key,id) WHERE authors IS NOT NULL')
|
||||
return dict(counts)
|
||||
|
||||
|
||||
def _candidates(scratch, output):
|
||||
counts = Counter(exact_title_groups=0, exact_title_and_roster_groups=0,
|
||||
exact_title_and_roster_papers=0, near_title_and_roster_pairs=0,
|
||||
oversized_rosters=0, papers_in_oversized_rosters=0)
|
||||
def emit(data):
|
||||
output.write(json.dumps(data, ensure_ascii=False, separators=(',',':'))+'\n')
|
||||
counts['exact_title_groups'] = scratch.execute('''
|
||||
SELECT count(*) FROM (SELECT title_key FROM titles WHERE title_key<>''
|
||||
GROUP BY title_key HAVING count(*)>1)''').fetchone()[0]
|
||||
for title,roster,size in scratch.execute('''
|
||||
SELECT title_key,authors,count(*) FROM titles WHERE title_key<>'' AND authors IS NOT NULL
|
||||
GROUP BY title_key,authors HAVING count(*)>1 ORDER BY title_key,authors'''):
|
||||
counts['exact_title_and_roster_groups'] += 1
|
||||
counts['exact_title_and_roster_papers'] += size
|
||||
ids = [row[0] for row in scratch.execute('''
|
||||
SELECT id FROM titles WHERE title_key=? AND authors=? ORDER BY id''', (title,roster))]
|
||||
emit({'kind':'exact_title_and_roster','title_key':title,'paper_ids':ids})
|
||||
cursor = scratch.execute('''
|
||||
SELECT authors,title_key,id FROM titles WHERE authors IS NOT NULL AND length(title_key)>=25
|
||||
ORDER BY authors,title_key,id''')
|
||||
for _, group in groupby(cursor, key=lambda row:row[0]):
|
||||
entries = []
|
||||
size = 0
|
||||
for _, title, paper_id in group:
|
||||
size += 1
|
||||
if size <= ROSTER_GROUP_LIMIT:
|
||||
entries.append((title,paper_id))
|
||||
if size > ROSTER_GROUP_LIMIT:
|
||||
counts['oversized_rosters'] += 1
|
||||
counts['papers_in_oversized_rosters'] += size
|
||||
emit({'kind':'oversized_roster','size':size,'sample_paper_ids':[p for _,p in entries[:5]]})
|
||||
continue
|
||||
for (left,left_id),(right,right_id) in combinations(entries,2):
|
||||
if left==right or 2*min(len(left),len(right))/(len(left)+len(right)) < TITLE_SCORE:
|
||||
continue
|
||||
matcher = SequenceMatcher(None,left,right,autojunk=False)
|
||||
if matcher.quick_ratio()>=TITLE_SCORE and (score:=matcher.ratio())>=TITLE_SCORE:
|
||||
counts['near_title_and_roster_pairs'] += 1
|
||||
emit({'kind':'near_title_and_roster','paper_ids':[left_id,right_id],'score':round(score,5)})
|
||||
return dict(counts)
|
||||
|
||||
|
||||
def report(connection, scratch_root, candidates):
|
||||
transaction = dict(connection.execute('''
|
||||
SELECT current_database() AS database,current_schema() AS schema,
|
||||
current_setting('transaction_read_only') AS read_only,
|
||||
current_setting('transaction_isolation') AS isolation,
|
||||
transaction_timestamp()::text AS snapshot_at''').fetchone())
|
||||
if transaction['read_only']!='on' or transaction['isolation']!='repeatable read':
|
||||
raise ValueError('A read-only repeatable-read snapshot is required')
|
||||
if shutil.disk_usage(scratch_root).free < 2*1024**3:
|
||||
raise ValueError('The scratch filesystem must have at least 2 GiB free')
|
||||
connection.execute("SET LOCAL statement_timeout='180s'")
|
||||
integrity = _checks(connection)
|
||||
coverage = _coverage(connection)
|
||||
search = _fts(connection)
|
||||
with TemporaryDirectory(prefix='paper-atlas-audit-',dir=scratch_root) as directory:
|
||||
with closing(sqlite3.connect(Path(directory)/'titles.sqlite')) as scratch:
|
||||
scratch.execute('PRAGMA temp_store=FILE')
|
||||
counts = _stage_titles(connection,scratch)
|
||||
candidate_counts = _candidates(scratch,candidates)
|
||||
return {'snapshot':transaction,'papers':counts,'integrity':integrity,
|
||||
'candidate_counts':candidate_counts,'edition_sources':coverage,'search':search,
|
||||
'candidate_rules':{'minimum_normalized_title_length':25,'near_title_score':TITLE_SCORE,
|
||||
'max_papers_per_near_title_author_group':ROSTER_GROUP_LIMIT,
|
||||
'requires_exact_normalized_author_set':True},
|
||||
'canonical_data_changed':False}
|
||||
|
||||
|
||||
def main(argv=None, *, prog=None):
|
||||
parser = argparse.ArgumentParser(prog=prog,description='只读核验 PostgreSQL 全库身份候选、约束、来源覆盖和全文检索')
|
||||
parser.add_argument('--pg-service',required=True)
|
||||
parser.add_argument('--pg-schema')
|
||||
parser.add_argument('--scratch-root',type=Path,required=True)
|
||||
parser.add_argument('--candidate-output',type=Path,required=True)
|
||||
args = parser.parse_args(argv)
|
||||
with closing(ReadConnection(args.pg_service,args.pg_schema)) as connection:
|
||||
with args.candidate_output.open('x',encoding='utf-8') as candidates:
|
||||
result = report(connection,args.scratch_root,candidates)
|
||||
print(json.dumps(result,ensure_ascii=False))
|
||||
@@ -15,6 +15,7 @@ COMMANDS = {
|
||||
('collect','acl'): ('paper_atlas.collectors.acl', {}, 'ACL Anthology XML'),
|
||||
('collect','springer-selected'): ('paper_atlas.collectors.springer_selected', {}, '核验限定的会后选集章节与原会议关系'),
|
||||
('collect','iclr-archive'): ('paper_atlas.collectors.iclr_archive', {}, '核对ICLR 2014–2016官方历史录用档案'),
|
||||
('collect','rss-proceedings'): ('paper_atlas.collectors.rss', {}, '核对RSS 2025官方论文摘要与演示证据'),
|
||||
('collect','iclr-program'): ('paper_atlas.collectors.iclr', {}, 'ICLR 官方公开程序'),
|
||||
('collect','iclr-snapshot'): ('paper_atlas.collectors.openreview', {}, 'ICLR 公开投稿与决定快照'),
|
||||
('collect','public-openreview'): ('paper_atlas.collectors.public_openreview', {}, '核验其他会议的公开拒稿及撤稿标签'),
|
||||
@@ -66,6 +67,7 @@ COMMANDS = {
|
||||
('audit','public-openreview'): ('paper_atlas.audits.public_openreview', {}, '核对公开负面决定与已有投稿历史'),
|
||||
('audit','source-identities'): ('paper_atlas.audits.source_identity', {}, '只读核对来源文件的身份候选与既有投稿历史'),
|
||||
('audit','histories'): ('paper_atlas.audits.histories', {}, '只读核对跨会议投稿历史及决定来源关系'),
|
||||
('audit','global-identity'): ('paper_atlas.audits.global_identity', {}, '只读核对 PostgreSQL 身份候选与完整性'),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -103,6 +103,8 @@ def collect_volume(volume,cache,out,allowed=None):
|
||||
path = Path(out)/f'pmlr-v{volume}.jsonl'
|
||||
path.parent.mkdir(parents=True,exist_ok=True)
|
||||
mixed_icml = venue == 'ICML' and year >= 2024
|
||||
corl_blue_sky_volume = volume == 164 and venue == 'CORL' and year == 2021
|
||||
corl_section_tracks = {'contributed papers': 'main', 'blue sky papers': 'blue_sky'}
|
||||
mapping, track_program = collect_track_evidence(year, cache) if mixed_icml else ({}, None)
|
||||
records = {}
|
||||
prefaces = set()
|
||||
@@ -120,7 +122,8 @@ def collect_volume(volume,cache,out,allowed=None):
|
||||
raise ValueError('Paper metadata has no publication ID: '+member_name)
|
||||
external = f"v{volume}/{row['id']}"
|
||||
# PMLR also gives volume prefaces the inproceedings layout.
|
||||
if str(row.get('section') or '').strip().casefold() == 'preface':
|
||||
section = str(row.get('section') or '').strip()
|
||||
if section.casefold() == 'preface':
|
||||
prefaces.add(external)
|
||||
continue
|
||||
ids = {}
|
||||
@@ -134,6 +137,12 @@ def collect_volume(volume,cache,out,allowed=None):
|
||||
value = value.split('id=')[-1].split('&')[0]
|
||||
ids[key] = value
|
||||
record = {'source':'pmlr','external_id':external,'url':f"https://proceedings.mlr.press/v{volume}/{row.get('id')}.html",'title':row['title'],'abstract':row.get('abstract'),'authors':row.get('author',[]),'identifiers':ids,'venue':venue,'year':year,'track':track,'status':'accepted','list_url':f'https://proceedings.mlr.press/v{volume}/','metadata':{key:row[key] for key in ('id','volume','date','page','doi','openreview','arxiv','publisher') if key in row}}
|
||||
if corl_blue_sky_volume:
|
||||
section_track = corl_section_tracks.get(section.casefold())
|
||||
if section_track is None:
|
||||
raise ValueError(f'Unrecognized CoRL 2021 section for {external}: {section!r}')
|
||||
record['track'] = section_track
|
||||
record['metadata']['pmlr_section'] = section
|
||||
version={'path':member_name,'date':row.get('date')}
|
||||
if external in records:
|
||||
previous=records[external]
|
||||
@@ -163,7 +172,7 @@ def collect_volume(volume,cache,out,allowed=None):
|
||||
with tempfile.TemporaryDirectory(prefix=f'.pmlr-v{volume}-', dir=path.parent) as temporary:
|
||||
stage = Path(temporary)
|
||||
for current_track, rows in sorted(grouped.items()):
|
||||
filename = f'pmlr-v{volume}-{current_track}.jsonl' if mixed_icml else path.name
|
||||
filename = f'pmlr-v{volume}-{current_track}.jsonl' if mixed_icml or current_track != track else path.name
|
||||
output = stage / filename
|
||||
output.write_text(''.join(json.dumps(row, ensure_ascii=False, default=str)+'\n' for row in rows))
|
||||
absent = sum(not (row.get('abstract') or '').strip() for row in rows)
|
||||
|
||||
@@ -0,0 +1,279 @@
|
||||
"""Reviewed RSS 2025 proceedings HTML; no PDF or private review access."""
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from datetime import date
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import tempfile
|
||||
import unicodedata
|
||||
|
||||
from paper_atlas.db.submissions import timestamp
|
||||
from paper_atlas.io.html import Document, Node, clean
|
||||
from paper_atlas.io.http import fetch
|
||||
|
||||
SOURCE = 'rss_proceedings'
|
||||
ROOT = 'https://www.roboticsproceedings.org/rss21/'
|
||||
HEADING = 'Robotics: Science and Systems XXI'
|
||||
PAGES = {'index': ROOT,
|
||||
'accepted': 'https://roboticsconference.org/2025/program/papers/',
|
||||
'demos': 'https://roboticsconference.org/2025/program/demos/',
|
||||
'authorinfo': 'https://roboticsconference.org/2025/information/authorinfo/'}
|
||||
EXPECTED = 163
|
||||
DEMO_IDS = {2, 20, 21, 33, 58, 59, 62, 84, 86, 92, 93, 107, 129, 132, 141}
|
||||
|
||||
|
||||
def one(values, description):
|
||||
if len(values) != 1:
|
||||
raise ValueError('Expected one '+description)
|
||||
return values[0]
|
||||
|
||||
|
||||
def comparable(text):
|
||||
return unicodedata.normalize('NFC', clean(text))
|
||||
|
||||
|
||||
def parse_index(html):
|
||||
document = Document(html)
|
||||
nodes = list(document.root.walk())
|
||||
if [clean(n.text()) for n in nodes if n.tag == 'h1'] != [HEADING]:
|
||||
raise ValueError('Unexpected RSS proceedings edition')
|
||||
result = {}
|
||||
for row in (n for n in nodes if n.tag == 'tr'):
|
||||
links = [n for n in row.walk() if n.tag == 'a'
|
||||
and re.fullmatch(r'p[0-9]{3}\.html', n.attrs.get('href', ''))]
|
||||
if not links:
|
||||
continue
|
||||
link = one(links, 'proceedings entry link')
|
||||
pid = int(link.attrs['href'][1:4])
|
||||
title = clean(link.text())
|
||||
authors = clean(one([n for n in row.walk() if n.tag == 'i'], 'index authors').text())
|
||||
if pid in result or pid < 1 or not title or not authors:
|
||||
raise ValueError('Empty or repeated RSS index entry')
|
||||
result[pid] = {'title': title, 'authors_display': authors,
|
||||
'url': ROOT+link.attrs['href']}
|
||||
occurrences = sum(n.tag == 'a' and bool(re.fullmatch(r'p[0-9]{3}\.html', n.attrs.get('href', '')))
|
||||
for n in nodes)
|
||||
if not result or len(result) != occurrences:
|
||||
raise ValueError('Unparsed or duplicate proceedings links')
|
||||
return result
|
||||
|
||||
|
||||
def parse_program(html, heading, columns):
|
||||
nodes = list(Document(html).root.walk())
|
||||
if [clean(n.text()) for n in nodes if n.tag == 'h1'] != [heading]:
|
||||
raise ValueError('Unexpected RSS program page')
|
||||
headers = [[clean(n.text()) for n in row.children if isinstance(n, Node) and n.tag == 'th']
|
||||
for row in nodes if row.tag == 'tr']
|
||||
if list(columns) not in headers:
|
||||
raise ValueError('Unexpected RSS program columns')
|
||||
result = {}
|
||||
for row in (n for n in nodes if n.tag == 'tr'):
|
||||
cells = [n for n in row.children if isinstance(n, Node) and n.tag == 'td']
|
||||
if not cells:
|
||||
continue
|
||||
if len(cells) != len(columns):
|
||||
raise ValueError('Unexpected RSS program row width')
|
||||
raw_id = clean(cells[0].text())
|
||||
if not re.fullmatch('[1-9][0-9]*', raw_id):
|
||||
raise ValueError('Invalid program ID')
|
||||
pid = int(raw_id)
|
||||
if pid in result:
|
||||
raise ValueError('Duplicate program ID')
|
||||
result[pid] = {key: clean(node.text()) for key, node in zip(columns, cells)}
|
||||
if not result[pid]['Title']:
|
||||
raise ValueError('Missing program title')
|
||||
if not result:
|
||||
raise ValueError('No RSS program rows')
|
||||
return result
|
||||
|
||||
|
||||
def validate_program(index, accepted, demos, authorinfo):
|
||||
if set(index) != set(accepted) or len(index) != EXPECTED or set(demos) != DEMO_IDS:
|
||||
raise ValueError('RSS program and reviewed proceedings census differ')
|
||||
text = clean(Document(authorinfo).root.text())
|
||||
if ('There are two types of submissions to RSS' not in text
|
||||
or 'Both types of papers appear in the RSS proceedings.' not in text):
|
||||
raise ValueError('Submission category evidence changed')
|
||||
for pid, item in index.items():
|
||||
variants = [accepted[pid]['Title']] + ([demos[pid]['Title']] if pid in demos else [])
|
||||
if any(comparable(title) != comparable(item['title']) for title in variants):
|
||||
raise ValueError('Conflicting official program title for '+str(pid))
|
||||
|
||||
|
||||
def parse_paper(html, pid, entry, accepted, demo, evidence):
|
||||
nodes = list(Document(html).root.walk())
|
||||
meta = {}
|
||||
for node in nodes:
|
||||
if node.tag == 'meta' and node.attrs.get('name', '').startswith('citation_'):
|
||||
meta.setdefault(node.attrs['name'], []).append(node.attrs.get('content', ''))
|
||||
title = one(meta.get('citation_title', []), 'citation title')
|
||||
authors = meta.get('citation_author', [])
|
||||
publication_date = one(meta.get('citation_publication_date', []), 'publication date')
|
||||
if not re.fullmatch(r'2025/[0-9]{2}/[0-9]{2}', publication_date):
|
||||
raise ValueError('Unexpected publication year/date')
|
||||
date.fromisoformat(publication_date.replace('/', '-'))
|
||||
if (one(meta.get('citation_conference_title', []), 'conference') != HEADING
|
||||
or one(meta.get('citation_volume', []), 'volume') != '21'
|
||||
or [clean(n.text()) for n in nodes if n.tag == 'h1'] != [HEADING]
|
||||
or [comparable(n.text()) for n in nodes if n.tag == 'h3'] != [comparable(title)]
|
||||
or comparable(title) != comparable(entry['title']) or not authors
|
||||
or any(not clean(author) for author in authors)
|
||||
or comparable(', '.join(authors)) != comparable(entry['authors_display'])):
|
||||
raise ValueError('Proceedings/citation title, authors or edition disagree')
|
||||
pdf = one(meta.get('citation_pdf_url', []), 'citation PDF URL')
|
||||
if pdf not in (ROOT+f'p{pid:03d}.pdf', ROOT.replace('https:', 'http:')+f'p{pid:03d}.pdf'):
|
||||
raise ValueError('Citation PDF ID differs; URL is metadata only')
|
||||
bibtex = one([n.text() for n in nodes if n.tag == 'pre'], 'original BibTeX')
|
||||
doi = one(re.findall(r'^\s*DOI\s*=\s*\{([^{}]+)\}', bibtex, re.I | re.M), 'BibTeX DOI')
|
||||
year = one(re.findall(r'^\s*YEAR\s*=\s*\{([^{}]+)\}', bibtex, re.I | re.M), 'BibTeX year')
|
||||
if doi != f'10.15607/RSS.2025.XXI.{pid:03d}' or year != '2025':
|
||||
raise ValueError('BibTeX DOI/year differs from volume and paper ID')
|
||||
abstract_parts = []
|
||||
inside = False
|
||||
markers = []
|
||||
for node in (n for n in nodes if n.tag == 'p'):
|
||||
labels = [clean(n.text()) for n in node.walk() if n.tag == 'b']
|
||||
if 'Abstract:' in labels:
|
||||
markers.append('start')
|
||||
inside = True
|
||||
value = clean(node.text()).removeprefix('Abstract:').strip()
|
||||
if value:
|
||||
abstract_parts.append(value)
|
||||
elif 'Download:' in labels:
|
||||
markers.append('end')
|
||||
inside = False
|
||||
elif inside:
|
||||
abstract_parts.append(clean(node.text()))
|
||||
if markers != ['start', 'end']:
|
||||
raise ValueError('Abstract boundaries missing or repeated')
|
||||
abstract = clean(' '.join(abstract_parts)) or None
|
||||
track = 'demo' if demo else 'proceedings_unspecified'
|
||||
tags = [{'namespace': SOURCE+':session', 'value': accepted['Session']}]
|
||||
if demo:
|
||||
tags.append({'namespace': SOURCE+':program_role', 'value': 'demonstration'})
|
||||
return {'source': SOURCE, 'external_id': f'2025/p{pid:03d}', 'url': entry['url'],
|
||||
'title': title, 'abstract': abstract, 'authors': authors,
|
||||
'identifiers': {'doi': doi.lower()},
|
||||
'submission_identifiers': [{'scheme': 'publication_doi', 'value': doi.lower()}],
|
||||
'venue': 'RSS', 'venue_aliases': [], 'year': 2025, 'track': track,
|
||||
'status': 'accepted', 'observed_at': evidence['paper']['observed_at'],
|
||||
'raw_decision': 'listed in official RSS 2025 proceedings and accepted program',
|
||||
'tags': tags,
|
||||
'metadata': {'publisher_paper_id': pid, 'citation_metadata': meta,
|
||||
'publication_date': publication_date, 'original_bibtex': bibtex,
|
||||
'index_entry': entry, 'accepted_program_entry': accepted,
|
||||
'demo_program_entry': demo, 'source_evidence': evidence,
|
||||
'missing_abstract_reason': None if abstract else 'empty_official_abstract_section',
|
||||
'track_scope_note': ('Explicit official demo program ID; not inferred from title.' if demo else
|
||||
'Proceedings and accepted list do not individually identify submission type; session is not a track.'),
|
||||
'decision_date_precision': 'not_supplied',
|
||||
'publication_evidence': 'Published proceedings and accepted program; not an original decision note.'}}
|
||||
|
||||
|
||||
class Pages:
|
||||
def __init__(self, *, cache=None, snapshots=None):
|
||||
if bool(cache) == bool(snapshots):
|
||||
raise ValueError('Choose exactly one cache or frozen snapshot directory')
|
||||
self.cache = Path(cache) if cache else None
|
||||
self.root = Path(snapshots).resolve() if snapshots else None
|
||||
self.receipts = json.loads((self.root/'receipts.json').read_bytes())['receipts'] if self.root else []
|
||||
self.evidence = {}
|
||||
self.data = {}
|
||||
|
||||
def read(self, url):
|
||||
if self.root:
|
||||
receipt = one([r for r in self.receipts if r['url'] == url], 'exact frozen response receipt')
|
||||
filename = Path(receipt['file'])
|
||||
path = self.root/filename
|
||||
if (filename.is_absolute() or '..' in filename.parts or not path.resolve().is_relative_to(self.root)
|
||||
or receipt['http_status'] != 200 or receipt['content_type'] != 'text/html'
|
||||
or receipt.get('final_url', url) != url):
|
||||
raise ValueError('Unsafe or unsuccessful frozen response')
|
||||
data = path.read_bytes()
|
||||
observed = receipt['at']
|
||||
else:
|
||||
data = fetch(url, self.cache, 2*1024**2, delay=1)
|
||||
receipt = json.loads((self.cache/hashlib.sha256(url.encode()).hexdigest()).with_suffix('.json').read_bytes())
|
||||
observed = receipt['retrieved_at']
|
||||
if (len(data) > 2*1024**2 or len(data) != receipt['bytes']
|
||||
or hashlib.sha256(data).hexdigest() != receipt['sha256'] or receipt['url'] != url):
|
||||
raise ValueError('Response checksum, URL or size differs')
|
||||
evidence = {'url': url, 'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data),
|
||||
'file': 'evidence/'+hashlib.sha256(data).hexdigest()+'.html',
|
||||
'observed_at': timestamp(observed), 'http_status': 200, 'response_receipt': receipt}
|
||||
self.evidence[url] = evidence
|
||||
self.data[url] = data
|
||||
return data.decode('utf-8'), evidence
|
||||
|
||||
|
||||
def collect(year, out, *, cache=None, snapshots=None):
|
||||
if year != 2025:
|
||||
raise ValueError('Only the independently reviewed RSS 2025 edition is supported')
|
||||
out = Path(out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
with (out/'rss.collect.lock').open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
report_path = out/'rss-2025.json'
|
||||
if report_path.exists() or list(out.glob('rss-2025-*.jsonl')):
|
||||
raise FileExistsError('Inspect existing output; never overwrite a completed or partial queue')
|
||||
pages = Pages(cache=cache, snapshots=snapshots)
|
||||
context = {key: pages.read(url) for key, url in PAGES.items()}
|
||||
index = parse_index(context['index'][0])
|
||||
accepted = parse_program(context['accepted'][0], 'Accepted Papers', ('ID', 'Session', 'Title', 'Authors'))
|
||||
demos = parse_program(context['demos'][0], 'Demos', ('ID', 'Location', 'Title', 'Day', 'Time'))
|
||||
validate_program(index, accepted, demos, context['authorinfo'][0])
|
||||
records = []
|
||||
for pid, entry in sorted(index.items()):
|
||||
html, evidence = pages.read(entry['url'])
|
||||
sources = {key: item[1] for key, item in context.items()}
|
||||
sources['paper'] = evidence
|
||||
records.append(parse_paper(html, pid, entry, accepted[pid], demos.get(pid), sources))
|
||||
counts = Counter(r['track'] for r in records)
|
||||
if counts != {'demo': len(DEMO_IDS), 'proceedings_unspecified': EXPECTED-len(DEMO_IDS)}:
|
||||
raise ValueError('Track census differs; review official source changes')
|
||||
files = []
|
||||
with tempfile.TemporaryDirectory(prefix='.rss-', dir=out) as temporary:
|
||||
stage = Path(temporary)
|
||||
(stage/'evidence').mkdir()
|
||||
for url, data in pages.data.items():
|
||||
(stage/pages.evidence[url]['file']).write_bytes(data)
|
||||
for track in sorted(counts):
|
||||
group = [r for r in records if r['track'] == track]
|
||||
path = stage/f'rss-2025-{track}.jsonl'
|
||||
path.write_text(''.join(json.dumps(r, ensure_ascii=False)+'\n' for r in group))
|
||||
manifest = {'source': SOURCE, 'parser_version': 1, 'venue': 'RSS', 'year': 2025,
|
||||
'track': track, 'count': len(group), 'file': str(out/path.name),
|
||||
'file_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
'list_url': ROOT, 'status': 'partial',
|
||||
'missing_abstract_count': sum(r['abstract'] is None for r in group)}
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps(manifest, ensure_ascii=False, indent=2)+'\n')
|
||||
files.append(manifest)
|
||||
result = {'source': SOURCE, 'year': 2025, 'records': len(records), 'files': files,
|
||||
'source_evidence': list(pages.evidence.values()), 'database_written': False,
|
||||
'pdfs_downloaded': False, 'status': 'collected_pending_identity_review',
|
||||
'note': 'Only the published proceedings. Demo IDs have explicit program evidence; '
|
||||
'other individual submission types remain unknown. No absent-based negative decisions. '
|
||||
'Separate source identity review and independent production acceptance are required.'}
|
||||
(stage/report_path.name).write_text(json.dumps(result, ensure_ascii=False, indent=2)+'\n')
|
||||
(out/'evidence').mkdir(exist_ok=True)
|
||||
for path in (stage/'evidence').iterdir():
|
||||
path.replace(out/'evidence'/path.name)
|
||||
for path in stage.iterdir():
|
||||
if path.is_file() and path.name != report_path.name:
|
||||
path.replace(out/path.name)
|
||||
(stage/report_path.name).replace(report_path)
|
||||
return result
|
||||
|
||||
|
||||
def main(argv=None, *, prog=None):
|
||||
parser = argparse.ArgumentParser(prog=prog)
|
||||
parser.add_argument('--year', type=int, choices=[2025], required=True)
|
||||
parser.add_argument('--out', required=True)
|
||||
group = parser.add_mutually_exclusive_group(required=True)
|
||||
group.add_argument('--cache')
|
||||
group.add_argument('--snapshots')
|
||||
args = parser.parse_args(argv)
|
||||
print(json.dumps(collect(args.year, args.out, cache=args.cache, snapshots=args.snapshots), ensure_ascii=False))
|
||||
@@ -1,4 +1,6 @@
|
||||
"""Versioned publication-source scopes and guards against stale source refreshes."""
|
||||
import base64
|
||||
import gzip
|
||||
import json
|
||||
import re
|
||||
|
||||
@@ -6,6 +8,24 @@ TABLE = 'source_scope_revisions'
|
||||
MARKER = 'source_scope_revisions_v1'
|
||||
COLUMNS = ('version_sha256', 'source_id', 'venue_id', 'year', 'event_key', 'list_url',
|
||||
'parent_sha256', 'records_json', 'plan_json', 'applied_at')
|
||||
COMPRESSED_PREFIX = 'gzip+base64:v1:'
|
||||
COMPRESSION_THRESHOLD = 1024 * 1024
|
||||
|
||||
|
||||
def encode_payload(value):
|
||||
"""Compress large revision payloads before sending them to the database."""
|
||||
text = json.dumps(value, ensure_ascii=False, separators=(',', ':'))
|
||||
raw = text.encode('utf-8')
|
||||
if len(raw) < COMPRESSION_THRESHOLD:
|
||||
return text
|
||||
return COMPRESSED_PREFIX + base64.b64encode(gzip.compress(raw, compresslevel=1, mtime=0)).decode('ascii')
|
||||
|
||||
|
||||
def decode_payload(text):
|
||||
if text.startswith(COMPRESSED_PREFIX):
|
||||
packed = base64.b64decode(text[len(COMPRESSED_PREFIX):], validate=True)
|
||||
return json.loads(gzip.decompress(packed))
|
||||
return json.loads(text)
|
||||
|
||||
|
||||
def available(c):
|
||||
@@ -60,7 +80,7 @@ def current_rows(c, source_id, cache):
|
||||
records = {}; scopes = {}
|
||||
for signature in signatures:
|
||||
raw = c.execute('SELECT records_json FROM source_scope_revisions WHERE version_sha256=?', (signature,)).fetchone()[0]
|
||||
scopes[signature] = json.loads(raw)
|
||||
scopes[signature] = decode_payload(raw)
|
||||
for row in scopes[signature]:
|
||||
ext = row['external_id']
|
||||
if ext in records and records[ext] != row:
|
||||
|
||||
@@ -203,7 +203,7 @@ def changes(c, inventory, before):
|
||||
def proposal(c, inventory, identity):
|
||||
before = capture(c, inventory)
|
||||
if before['parent_sha256']:
|
||||
saved = json.loads(c.execute('SELECT records_json FROM source_scope_revisions WHERE version_sha256=?', (before['parent_sha256'],)).fetchone()[0])
|
||||
saved = versions.decode_payload(c.execute('SELECT records_json FROM source_scope_revisions WHERE version_sha256=?', (before['parent_sha256'],)).fetchone()[0])
|
||||
if sorted(saved, key=lambda r: r['external_id']) != sorted(inventory['previous_rows'], key=lambda r: r['external_id']):
|
||||
raise ValueError('Previous snapshot differs from active source version')
|
||||
records, coverage = changes(c, inventory, before)
|
||||
@@ -241,8 +241,8 @@ def verify(c, plan, signature):
|
||||
if after['coverage'] != expected_coverage:
|
||||
raise ValueError('Reconciled coverage differs; roll back')
|
||||
stored = c.execute('SELECT records_json,plan_json,parent_sha256 FROM source_scope_revisions WHERE version_sha256=?', (signature,)).fetchone()
|
||||
if (not stored or json.loads(stored[0]) != plan['inventory']['rows']
|
||||
or json.loads(stored[1]) != plan or stored[2] != plan['before']['parent_sha256']):
|
||||
if (not stored or versions.decode_payload(stored[0]) != plan['inventory']['rows']
|
||||
or versions.decode_payload(stored[1]) != plan or stored[2] != plan['before']['parent_sha256']):
|
||||
raise ValueError('Versioned source or complete before-images changed')
|
||||
return {'records_changed': len(plan['changes']), 'events_moved': sum(r['from_edition_id'] != r['target_edition_id'] for r in plan['changes']),
|
||||
'protected_papers': len(expected['papers']), 'protected_decisions': len(expected['decisions'])}
|
||||
@@ -269,6 +269,9 @@ def apply_plan(c, plan):
|
||||
signature = digest(plan)
|
||||
if versions.available(c) and c.execute('SELECT 1 FROM source_scope_revisions WHERE version_sha256=?', (signature,)).fetchone():
|
||||
return {'status': 'already_applied', 'version_sha256': signature, **verify(c, plan, signature)}
|
||||
inventory = plan['inventory']
|
||||
records_payload = versions.encode_payload(inventory['rows'])
|
||||
plan_payload = versions.encode_payload(plan)
|
||||
if getattr(c, 'postgres', False):
|
||||
c.execute('LOCK TABLE sources,papers,identifiers,source_records,venues,venue_aliases,editions,edition_sources,submissions,decisions,submission_sources,submission_identifiers,decision_sources,tags,paper_tags IN SHARE ROW EXCLUSIVE MODE')
|
||||
if proposal(c, plan['inventory'], plan['database']) != plan:
|
||||
@@ -284,10 +287,10 @@ def apply_plan(c, plan):
|
||||
before = row['before']
|
||||
c.execute('UPDATE edition_sources SET expected_count=?,imported_count=? WHERE edition_id=? AND source_id=? AND list_url=?',
|
||||
(row['expected_count'], row['imported_count'], before[0], before[1], before[2]))
|
||||
inventory = plan['inventory']
|
||||
c.execute('''INSERT INTO source_scope_revisions(version_sha256,source_id,venue_id,year,event_key,list_url,parent_sha256,records_json,plan_json)
|
||||
VALUES(?,?,?,?,?,?,?,?,?)''', (signature, plan['before']['source_id'], plan['before']['venue_id'], inventory['year'], inventory['event_key'],
|
||||
inventory['list_url'], plan['before']['parent_sha256'], json.dumps(inventory['rows'], ensure_ascii=False), json.dumps(plan, ensure_ascii=False)))
|
||||
inventory['list_url'], plan['before']['parent_sha256'],
|
||||
records_payload, plan_payload))
|
||||
return {'status': 'applied', 'version_sha256': signature, **verify(c, plan, signature)}
|
||||
|
||||
|
||||
|
||||
Vendored
+4356
File diff suppressed because it is too large
Load Diff
+1501
File diff suppressed because it is too large
Load Diff
Vendored
+1646
File diff suppressed because it is too large
Load Diff
Vendored
+1585
File diff suppressed because it is too large
Load Diff
Vendored
+148
@@ -0,0 +1,148 @@
|
||||
<html>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
||||
<head>
|
||||
<meta name="citation_title" content="Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation" />
|
||||
<meta name="citation_author" content="Pascal Roth" />
|
||||
<meta name="citation_author" content="Jonas Frey" />
|
||||
<meta name="citation_author" content="Cesar Cadena" />
|
||||
<meta name="citation_author" content="Marco Hutter" />
|
||||
<meta name="citation_publication_date" content="2025/06/21" />
|
||||
<meta name="citation_conference_title" content="Robotics: Science and Systems XXI" />
|
||||
<meta name="citation_isbn" content="979-8-9902848-1-4" />
|
||||
<meta name="citation_volume" content="21" />
|
||||
<meta name="citation_pdf_url" content="http://www.roboticsproceedings.org/rss21/p001.pdf" />
|
||||
<title>Robotics: Science and Systems XXI - Online Proceedings</title>
|
||||
<link type="text/css" rel="stylesheet" href="../style.css">
|
||||
<!--[if IE 5]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie5.css">
|
||||
<![endif]-->
|
||||
<!--[if IE 6]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie6.css">
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div class="menu">
|
||||
<ul>
|
||||
<li class="here"><a class="menu" href="../index.html">Home</a></li>
|
||||
<li class="label">RSS XXII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XXI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/authors.html">Authors</a></li>
|
||||
<li class="label">RSS X</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/authors.html">Authors</a></li>
|
||||
<li class="label">RSS V</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/authors.html">Authors</a></li>
|
||||
<li class="label">RSS III</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/authors.html">Authors</a></li>
|
||||
<li class="label">RSS II</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/authors.html">Authors</a></li>
|
||||
<li class="label">RSS I</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/authors.html">Authors</a></li>
|
||||
<li class="label">Ethics and Malpractice</span></li>
|
||||
<li class="menu"><a class="menu" href="../statement.html">Statement</a></li>
|
||||
<li class="label">Contact</span></li>
|
||||
<li class="last"><a class="menu" href="../contact.html">Webmaster</a></li>
|
||||
<li class="no-style"><center><img border="0" src="../robot-small.jpg"></center>
|
||||
<br><br><br>
|
||||
<center>We are using google analytics for statistics!</center>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="content">
|
||||
|
||||
<h1>Robotics: Science and Systems XXI</h1>
|
||||
<h3>Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation</h3>
|
||||
<i>Pascal Roth, Jonas Frey, Cesar Cadena, Marco Hutter</i><br>
|
||||
<p>
|
||||
<b>Abstract:</b>
|
||||
</p><p style="text-align: justify;">
|
||||
Ensuring safe navigation in complex environments requires accurate real-time traversability assessment and understanding of environmental interactions relative to the robot's capabilities. Traditional methods, which assume simplified dynamics, often require designing and tuning cost functions to safely guide paths or actions toward the goal. This process is tedious, environment-dependent, and not generalizable. To overcome these issues, we propose a novel learned perceptive Forward Dynamics Model (FDM) that predicts the robot's future state conditioned on the surrounding geometry and history of proprioceptive measurements, proposing a more scalable, safer, and heuristic-free solution. The FDM is trained on multiple years of simulated navigation experience, including high-risk maneuvers, and real-world interactions to incorporate the full system dynamics beyond rigid body simulation. We integrate our perceptive FDM into a zero-shot Model Predictive Path Integral (MPPI) planning framework, leveraging the learned mapping between actions, future states, and failure probability. This allows for optimizing a simplified cost function, eliminating the need for extensive cost-tuning to ensure safety. On the legged robot ANYmal, the proposed perceptive FDM improves the position estimation by on average 41% over competitive baselines, which translates into a 27% higher navigation success rate in rough simulation environments. Moreover, we demonstrate effective sim-to-real transfer and showcase the benefit of training on synthetic and real data. Code and models are made publicly available under https://github.com/leggedrobotics/fdm.
|
||||
</p>
|
||||
<p>
|
||||
<b>Download:</b> <a href="p001.pdf" target="_blank" onclick="_gaq.push(['_trackEvent','rss10/p001.pdf','PDF',this.href]);"><img src="../icon-pdf.png" border=0></a>
|
||||
</p>
|
||||
<p>
|
||||
<b>Bibtex:</b>
|
||||
<pre>
|
||||
@INPROCEEDINGS{RothP-RSS-25,
|
||||
AUTHOR = {Pascal Roth AND Jonas Frey AND Cesar Cadena AND Marco Hutter},
|
||||
TITLE = {{Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation}},
|
||||
BOOKTITLE = {Proceedings of Robotics: Science and Systems},
|
||||
YEAR = {2025},
|
||||
ADDRESS = {LosAngeles, CA, USA},
|
||||
MONTH = {June},
|
||||
DOI = {10.15607/RSS.2025.XXI.001}
|
||||
}
|
||||
|
||||
</pre>
|
||||
</p>
|
||||
|
||||
</div>
|
||||
<div class="analyt">
|
||||
<script type="text/javascript">
|
||||
|
||||
var _gaq = _gaq || [];
|
||||
_gaq.push(['_setAccount', 'UA-24631810-1']);
|
||||
_gaq.push(['_trackPageview']);
|
||||
|
||||
(function() {
|
||||
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
|
||||
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
||||
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
|
||||
})();
|
||||
|
||||
</script>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Vendored
+155
@@ -0,0 +1,155 @@
|
||||
<html>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
||||
<head>
|
||||
<meta name="citation_title" content="Demonstrating ViSafe: Vision-enabled Safety for High-speed Detect and Avoid" />
|
||||
<meta name="citation_author" content="Parv Kapoor" />
|
||||
<meta name="citation_author" content="Ian Higgins" />
|
||||
<meta name="citation_author" content="Nikhil Varma Keetha" />
|
||||
<meta name="citation_author" content="Jay Patrikar" />
|
||||
<meta name="citation_author" content="Brady Moon" />
|
||||
<meta name="citation_author" content="Zelin Ye" />
|
||||
<meta name="citation_author" content="Yao He" />
|
||||
<meta name="citation_author" content="Ivan Cisneros" />
|
||||
<meta name="citation_author" content="Changliu Liu" />
|
||||
<meta name="citation_author" content="Eunsuk Kang" />
|
||||
<meta name="citation_author" content="Sebastian Scherer" />
|
||||
<meta name="citation_publication_date" content="2025/06/21" />
|
||||
<meta name="citation_conference_title" content="Robotics: Science and Systems XXI" />
|
||||
<meta name="citation_isbn" content="979-8-9902848-1-4" />
|
||||
<meta name="citation_volume" content="21" />
|
||||
<meta name="citation_pdf_url" content="http://www.roboticsproceedings.org/rss21/p002.pdf" />
|
||||
<title>Robotics: Science and Systems XXI - Online Proceedings</title>
|
||||
<link type="text/css" rel="stylesheet" href="../style.css">
|
||||
<!--[if IE 5]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie5.css">
|
||||
<![endif]-->
|
||||
<!--[if IE 6]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie6.css">
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div class="menu">
|
||||
<ul>
|
||||
<li class="here"><a class="menu" href="../index.html">Home</a></li>
|
||||
<li class="label">RSS XXII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XXI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/authors.html">Authors</a></li>
|
||||
<li class="label">RSS X</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/authors.html">Authors</a></li>
|
||||
<li class="label">RSS V</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/authors.html">Authors</a></li>
|
||||
<li class="label">RSS III</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/authors.html">Authors</a></li>
|
||||
<li class="label">RSS II</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/authors.html">Authors</a></li>
|
||||
<li class="label">RSS I</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/authors.html">Authors</a></li>
|
||||
<li class="label">Ethics and Malpractice</span></li>
|
||||
<li class="menu"><a class="menu" href="../statement.html">Statement</a></li>
|
||||
<li class="label">Contact</span></li>
|
||||
<li class="last"><a class="menu" href="../contact.html">Webmaster</a></li>
|
||||
<li class="no-style"><center><img border="0" src="../robot-small.jpg"></center>
|
||||
<br><br><br>
|
||||
<center>We are using google analytics for statistics!</center>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="content">
|
||||
|
||||
<h1>Robotics: Science and Systems XXI</h1>
|
||||
<h3>Demonstrating ViSafe: Vision-enabled Safety for High-speed Detect and Avoid</h3>
|
||||
<i>Parv Kapoor, Ian Higgins, Nikhil Varma Keetha, Jay Patrikar, Brady Moon, Zelin Ye, Yao He, Ivan Cisneros, Changliu Liu, Eunsuk Kang, Sebastian Scherer</i><br>
|
||||
<p>
|
||||
<b>Abstract:</b>
|
||||
</p><p style="text-align: justify;">
|
||||
Assured safe-separation is essential for achieving seamless high-density operation of airborne vehicles in a shared airspace. To equip resource-constrained aerial systems with this safety-critical capability, we present ViSafe, a high-speed vision-only airborne collision avoidance system. ViSafe offers a full-stack solution to the Detect and Avoid (DAA) problem by tightly integrating a learning-based edge-AI framework with a custom multi-camera hardware prototype designed under SWaP-C constraints. By leveraging perceptual input-focused control barrier functions (CBF) to design, encode, and enforce safety thresholds, ViSafe can provide provably safe runtime guarantees for self-separation in high-speed aerial operations. We evaluate ViSafe’s performance through an extensive test campaign involving both simulated digital twins and real-world flight scenarios. By independently varying agent types, closure rates, interaction geometries, and environmental conditions (e.g., weather and lighting), we demonstrate that ViSafe consistently ensures self-separation across diverse scenarios. In first-of-its-kind real-world high-speed collision avoidance tests with closure rates reaching 144 km/h, ViSafe sets a new benchmark for vision-only autonomous collision avoidance, establishing a new standard for safety in high-speed aerial navigation.
|
||||
</p>
|
||||
<p>
|
||||
<b>Download:</b> <a href="p002.pdf" target="_blank" onclick="_gaq.push(['_trackEvent','rss10/p002.pdf','PDF',this.href]);"><img src="../icon-pdf.png" border=0></a>
|
||||
</p>
|
||||
<p>
|
||||
<b>Bibtex:</b>
|
||||
<pre>
|
||||
@INPROCEEDINGS{KapoorP-RSS-25,
|
||||
AUTHOR = {Parv Kapoor AND Ian Higgins AND Nikhil Varma Keetha AND Jay Patrikar AND Brady Moon AND Zelin Ye AND Yao He AND Ivan Cisneros AND Changliu Liu AND Eunsuk Kang AND Sebastian Scherer},
|
||||
TITLE = {{Demonstrating ViSafe: Vision-enabled Safety for High-speed Detect and Avoid}},
|
||||
BOOKTITLE = {Proceedings of Robotics: Science and Systems},
|
||||
YEAR = {2025},
|
||||
ADDRESS = {LosAngeles, CA, USA},
|
||||
MONTH = {June},
|
||||
DOI = {10.15607/RSS.2025.XXI.002}
|
||||
}
|
||||
|
||||
</pre>
|
||||
</p>
|
||||
|
||||
</div>
|
||||
<div class="analyt">
|
||||
<script type="text/javascript">
|
||||
|
||||
var _gaq = _gaq || [];
|
||||
_gaq.push(['_setAccount', 'UA-24631810-1']);
|
||||
_gaq.push(['_trackPageview']);
|
||||
|
||||
(function() {
|
||||
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
|
||||
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
||||
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
|
||||
})();
|
||||
|
||||
</script>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Vendored
+65
@@ -0,0 +1,65 @@
|
||||
{
|
||||
"scope": "Six unmodified official HTML fixtures; exact observations from full167 server freeze. No research PDFs.",
|
||||
"receipts": [
|
||||
{
|
||||
"file": "index.html",
|
||||
"url": "https://www.roboticsproceedings.org/rss21/",
|
||||
"final_url": "https://www.roboticsproceedings.org/rss21/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 77753,
|
||||
"sha256": "5d8e530b5ce7184795c14a7dca615f9c1a8b0b189661e0e6e795167b68b8516d",
|
||||
"at": "2026-11-09T09:53:09.121733+00:00"
|
||||
},
|
||||
{
|
||||
"file": "accepted.html",
|
||||
"url": "https://roboticsconference.org/2025/program/papers/",
|
||||
"final_url": "https://roboticsconference.org/2025/program/papers/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 197708,
|
||||
"sha256": "fa1a5bce496a85eca8cec79aed8dbc148cda791fe6859375ba0c294983a18683",
|
||||
"at": "2026-11-09T09:53:10.171450+00:00"
|
||||
},
|
||||
{
|
||||
"file": "demos.html",
|
||||
"url": "https://roboticsconference.org/2025/program/demos/",
|
||||
"final_url": "https://roboticsconference.org/2025/program/demos/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 43542,
|
||||
"sha256": "0b154ed110f6d49ebe2ba87f2c9c498df5fc1ea00bfe508b548522dfe4779698",
|
||||
"at": "2026-11-09T09:53:11.145288+00:00"
|
||||
},
|
||||
{
|
||||
"file": "authorinfo.html",
|
||||
"url": "https://roboticsconference.org/2025/information/authorinfo/",
|
||||
"final_url": "https://roboticsconference.org/2025/information/authorinfo/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 54508,
|
||||
"sha256": "24d023daf9a074434c6e004001b863dfd260330e523671d614fa120b052927ab",
|
||||
"at": "2026-11-09T09:53:12.170246+00:00"
|
||||
},
|
||||
{
|
||||
"file": "p001.html",
|
||||
"url": "https://www.roboticsproceedings.org/rss21/p001.html",
|
||||
"final_url": "https://www.roboticsproceedings.org/rss21/p001.html",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 9048,
|
||||
"sha256": "973172348096a5cbf3e311bdc1ce33b27447761fa1058a050e611aa17547d821",
|
||||
"at": "2026-11-09T09:53:13.437943+00:00"
|
||||
},
|
||||
{
|
||||
"file": "p002.html",
|
||||
"url": "https://www.roboticsproceedings.org/rss21/p002.html",
|
||||
"final_url": "https://www.roboticsproceedings.org/rss21/p002.html",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 9335,
|
||||
"sha256": "30b5b68e7ed6a809ca51282c5efd1416513417ba91eba2845f2e73ed18b90ba4",
|
||||
"at": "2026-11-09T09:53:14.697857+00:00"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
import io
|
||||
import json
|
||||
import sqlite3
|
||||
import unittest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
from paper_atlas.audits import global_identity
|
||||
|
||||
|
||||
class GlobalIdentityCandidatesTest(unittest.TestCase):
|
||||
def test_exact_same_title_and_near_title_require_author_evidence(self):
|
||||
output = io.StringIO()
|
||||
with sqlite3.connect(':memory:') as scratch:
|
||||
scratch.execute('CREATE TABLE titles (id INTEGER, title_key TEXT, authors BLOB)')
|
||||
scratch.executemany('INSERT INTO titles VALUES (?,?,?)', [
|
||||
(1, 'astronomicaltransientcatalogue', b'first'),
|
||||
(2, 'astronomicaltransientcatalogue', b'first'),
|
||||
(3, 'astronomicaltransientcatalogues', b'first'),
|
||||
(4, 'astronomicaltransientcatalogue', b'second'),
|
||||
(5, 'astronomicaltransientcatalogue', None),
|
||||
])
|
||||
counts = global_identity._candidates(scratch, output)
|
||||
|
||||
records = [json.loads(row) for row in output.getvalue().splitlines()]
|
||||
self.assertEqual(counts['exact_title_groups'], 1)
|
||||
self.assertEqual(counts['exact_title_and_roster_papers'], 2)
|
||||
self.assertEqual(counts['near_title_and_roster_pairs'], 2)
|
||||
self.assertEqual(records[0]['paper_ids'], [1, 2])
|
||||
self.assertEqual({tuple(r['paper_ids']) for r in records[1:]}, {(1, 3), (2, 3)})
|
||||
|
||||
def test_author_roster_limit_is_visible_in_receipt(self):
|
||||
output = io.StringIO()
|
||||
with sqlite3.connect(':memory:') as scratch:
|
||||
scratch.execute('CREATE TABLE titles (id INTEGER, title_key TEXT, authors BLOB)')
|
||||
scratch.executemany('INSERT INTO titles VALUES (?,?,?)', [
|
||||
(1, 'astronomicaltransientcatalogue', b'first'),
|
||||
(2, 'astronomicaltransientcatalogues', b'first'),
|
||||
(3, 'astronomicaltransientcatalogued', b'first'),
|
||||
])
|
||||
with patch.object(global_identity, 'ROSTER_GROUP_LIMIT', 2):
|
||||
counts = global_identity._candidates(scratch, output)
|
||||
|
||||
self.assertEqual(counts['oversized_rosters'], 1)
|
||||
self.assertEqual(counts['papers_in_oversized_rosters'], 3)
|
||||
self.assertEqual(counts['near_title_and_roster_pairs'], 0)
|
||||
self.assertEqual(json.loads(output.getvalue())['kind'], 'oversized_roster')
|
||||
|
||||
def test_fingerprint_normalizes_supported_author_encodings(self):
|
||||
strings = json.dumps(['Alice Doe', 'Chloë Li'])
|
||||
records = json.dumps([['Li', 'Chloë'], {'given':'Alice','family':'Doe'}])
|
||||
self.assertEqual(global_identity._author_fingerprint(strings),
|
||||
global_identity._author_fingerprint(records))
|
||||
self.assertIsNone(global_identity._author_fingerprint('[]'))
|
||||
|
||||
def test_doi_pattern_is_bound_as_a_parameter(self):
|
||||
connection = Mock()
|
||||
|
||||
def execute(query, params):
|
||||
self.assertEqual(query.count('?'), len(params))
|
||||
if params:
|
||||
pattern, = params
|
||||
self.assertIn('https?', pattern)
|
||||
result = Mock()
|
||||
result.fetchone.return_value = (0,)
|
||||
return result
|
||||
|
||||
connection.execute.side_effect = execute
|
||||
self.assertEqual(global_identity._checks(connection)['normalized_doi_collisions'], 0)
|
||||
@@ -60,6 +60,24 @@ class MetadataTests(unittest.TestCase):
|
||||
self.assertEqual(result['files'][0]['count'], 1)
|
||||
self.assertEqual([row['external_id'] for row in rows], ['v244/paper1'])
|
||||
|
||||
def test_corl_2021_sections_keep_blue_sky_apart_from_contributed_papers(self):
|
||||
config = b'shortname: CoRL\nyear: 2021\nbooktitle: Proceedings of the 5th Conference on Robot Learning\n'
|
||||
paper = '---\nid: {id}\nlayout: inproceedings\ntitle: Paper\nabstract: Abstract\nsection: {section}\n---\n'
|
||||
posts = [('one.md', paper.format(id='one', section='Contributed Papers')),
|
||||
('two.md', paper.format(id='two', section='Blue Sky Papers'))]
|
||||
with tempfile.TemporaryDirectory() as tmp, patch('paper_atlas.collectors.pmlr.fetch', return_value=config), patch(
|
||||
'paper_atlas.collectors.pmlr.metadata_texts', return_value=iter(posts),
|
||||
):
|
||||
result = collect_volume(164, tmp, tmp)
|
||||
self.assertEqual(result['track'], 'mixed')
|
||||
files = {manifest['track']: Path(manifest['file']) for manifest in result['files']}
|
||||
self.assertEqual({key: path.name for key, path in files.items()},
|
||||
{'main': 'pmlr-v164.jsonl', 'blue_sky': 'pmlr-v164-blue_sky.jsonl'})
|
||||
rows = {track: json.loads(path.read_text()) for track, path in files.items()}
|
||||
self.assertEqual([(rows[t]['external_id'], rows[t]['track'], rows[t]['metadata']['pmlr_section'])
|
||||
for t in ('main', 'blue_sky')],
|
||||
[('v164/one', 'main', 'Contributed Papers'), ('v164/two', 'blue_sky', 'Blue Sky Papers')])
|
||||
|
||||
def test_conflicting_duplicate_posts_do_not_replace_previous_collection(self):
|
||||
config=b'shortname: CoRL\nyear: 2023\nbooktitle: CoRL\n'
|
||||
paper='---\nid: paper1\nlayout: inproceedings\ntitle: {title}\nabstract: Abstract\n---\n'
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
"""Frozen publisher examples, conflicting evidence and preserved attempt identity."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from paper_atlas.collectors import rss
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
FIXTURES = Path(__file__).parent/'fixtures/rss2025'
|
||||
OBSERVED = '2026-11-09T10:00:00Z'
|
||||
|
||||
|
||||
def fixture(name):
|
||||
return (FIXTURES/name).read_text()
|
||||
|
||||
|
||||
class RssProceedingsTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.index = rss.parse_index(fixture('index.html'))
|
||||
self.accepted = rss.parse_program(fixture('accepted.html'), 'Accepted Papers', ('ID', 'Session', 'Title', 'Authors'))
|
||||
self.demos = rss.parse_program(fixture('demos.html'), 'Demos', ('ID', 'Location', 'Title', 'Day', 'Time'))
|
||||
|
||||
def paper(self, pid=1, html=None, entry=None):
|
||||
return rss.parse_paper(html or fixture(f'p{pid:03d}.html'), pid, entry or self.index[pid],
|
||||
self.accepted[pid], self.demos.get(pid), {'paper': {'observed_at': OBSERVED}})
|
||||
|
||||
def test_frozen_official_census_and_all_fixture_hashes(self):
|
||||
receipts = json.loads((FIXTURES/'receipts.json').read_bytes())['receipts']
|
||||
self.assertEqual(len(receipts), 6)
|
||||
for r in receipts:
|
||||
data = (FIXTURES/r['file']).read_bytes()
|
||||
self.assertEqual((len(data), hashlib.sha256(data).hexdigest()), (r['bytes'], r['sha256']))
|
||||
self.assertEqual(len(self.index), 163)
|
||||
self.assertEqual(set(self.index), set(self.accepted))
|
||||
self.assertEqual(len(self.demos), 15)
|
||||
rss.validate_program(self.index, self.accepted, self.demos, fixture('authorinfo.html'))
|
||||
self.assertFalse(self.demos[93]['Title'].startswith('Demonstrating'))
|
||||
|
||||
def test_complete_authors_abstract_and_source_metadata_are_preserved(self):
|
||||
row = self.paper()
|
||||
self.assertEqual(row['authors'], ['Pascal Roth', 'Jonas Frey', 'Cesar Cadena', 'Marco Hutter'])
|
||||
self.assertTrue(row['abstract'].startswith('Ensuring safe navigation'))
|
||||
self.assertTrue(row['abstract'].endswith('https://github.com/leggedrobotics/fdm.'))
|
||||
self.assertNotIn('Download:', row['abstract'])
|
||||
self.assertEqual(row['identifiers'], {'doi': '10.15607/rss.2025.xxi.001'})
|
||||
self.assertEqual(row['metadata']['publication_date'], '2025/06/21')
|
||||
self.assertIn('AUTHOR', row['metadata']['original_bibtex'])
|
||||
self.assertEqual(row['status'], 'accepted')
|
||||
self.assertNotIn('decision_date', row)
|
||||
self.assertEqual(row['metadata']['decision_date_precision'], 'not_supplied')
|
||||
|
||||
def test_demo_evidence_is_independent_of_title_and_session_topic(self):
|
||||
row = self.paper(2)
|
||||
self.assertEqual(row['track'], 'demo')
|
||||
self.assertEqual(row['metadata']['demo_program_entry']['ID'], '2')
|
||||
self.assertEqual(self.paper()['track'], 'proceedings_unspecified')
|
||||
self.assertEqual({t['namespace'] for t in row['tags']}, {'rss_proceedings:session', 'rss_proceedings:program_role'})
|
||||
self.assertFalse(any('research_type' in t['namespace'] for t in row['tags']))
|
||||
html = fixture('p001.html').replace('Learned Perceptive Forward Dynamics Model', 'Demonstrating a Learned Model')
|
||||
entry = {**self.index[1], 'title': self.index[1]['title'].replace('Learned Perceptive Forward Dynamics Model', 'Demonstrating a Learned Model')}
|
||||
self.assertEqual(self.paper(html=html, entry=entry)['track'], 'proceedings_unspecified')
|
||||
|
||||
def test_missing_abstract_is_explicit_and_multiple_paragraphs_are_complete(self):
|
||||
html = fixture('p001.html')
|
||||
beginning = html.index('<p style="text-align: justify;">')
|
||||
end = html.index('</p>', beginning)
|
||||
empty = html[:beginning]+'<p style="text-align: justify;">'+html[end:]
|
||||
row = self.paper(html=empty)
|
||||
self.assertIsNone(row['abstract'])
|
||||
self.assertEqual(row['metadata']['missing_abstract_reason'], 'empty_official_abstract_section')
|
||||
multi = html.replace('</p> \n<p> \n<b>Download:', '</p><p>Second abstract paragraph.</p><p><b>Download:', 1)
|
||||
self.assertTrue(self.paper(html=multi)['abstract'].endswith('Second abstract paragraph.'))
|
||||
|
||||
def test_metadata_drift_author_truncation_and_wrong_paper_doi_fail(self):
|
||||
original = fixture('p001.html')
|
||||
for altered in [original.replace('XXI.001}', 'XXI.002}'),
|
||||
original.replace('content="2025/06/21"', 'content="2026/06/21"'),
|
||||
original.replace('<meta name="citation_author" content="Marco Hutter" />', ''),
|
||||
original.replace('<b>Abstract:</b>', '<b>Summary:</b>'),
|
||||
original.replace('rss21/p001.pdf', 'rss21/p002.pdf'),
|
||||
original.replace('citation_volume" content="21', 'citation_volume" content="22')]:
|
||||
with self.subTest(altered=altered[:60]), self.assertRaises(ValueError):
|
||||
self.paper(html=altered)
|
||||
|
||||
def test_duplicate_and_unparsed_index_entries_fail(self):
|
||||
for html in [fixture('index.html').replace('p002.html', 'p001.html'),
|
||||
fixture('index.html')+'<a href="p999.html">Stray paper</a>',
|
||||
fixture('index.html').replace( rss.HEADING, 'A different volume')]:
|
||||
with self.subTest(html=html[:40]), self.assertRaises(ValueError):
|
||||
rss.parse_index(html)
|
||||
|
||||
def test_program_title_id_category_and_header_changes_require_review(self):
|
||||
for change in ['title', 'id', 'category']:
|
||||
index = {k: dict(v) for k, v in self.index.items()}
|
||||
demos = {k: dict(v) for k, v in self.demos.items()}
|
||||
if change == 'title':
|
||||
demos[2]['Title'] = 'Another paper'
|
||||
elif change == 'id':
|
||||
del index[1]
|
||||
else:
|
||||
del demos[93]
|
||||
with self.subTest(change=change), self.assertRaises(ValueError):
|
||||
rss.validate_program(index, self.accepted, demos, fixture('authorinfo.html'))
|
||||
with self.assertRaises(ValueError):
|
||||
rss.parse_program(fixture('demos.html').replace('>Location<', '>Other<'), 'Demos', ('ID', 'Location', 'Title', 'Day', 'Time'))
|
||||
|
||||
def snapshot(self, root):
|
||||
root.mkdir()
|
||||
data = (FIXTURES/'p001.html').read_bytes()
|
||||
(root/'paper.html').write_bytes(data)
|
||||
receipt = {'file': 'paper.html', 'url': self.index[1]['url'], 'final_url': self.index[1]['url'],
|
||||
'http_status': 200, 'content_type': 'text/html', 'bytes': len(data),
|
||||
'sha256': hashlib.sha256(data).hexdigest(), 'at': OBSERVED}
|
||||
(root/'receipts.json').write_text(json.dumps({'receipts': [receipt]}))
|
||||
return receipt
|
||||
|
||||
def test_frozen_receipt_hash_status_redirect_and_path_guards(self):
|
||||
for failure in ['hash', 'status', 'redirect', 'path', 'duplicate', 'symlink']:
|
||||
with self.subTest(failure=failure), tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)/'source'
|
||||
r = self.snapshot(root)
|
||||
receipts = [r]
|
||||
if failure == 'hash': r['sha256'] = '0'*64
|
||||
elif failure == 'status': r['http_status'] = 404
|
||||
elif failure == 'redirect': r['final_url'] = 'https://example.org/p001.html'
|
||||
elif failure == 'path': r['file'] = '../outside.html'
|
||||
elif failure == 'duplicate': receipts.append(dict(r))
|
||||
else:
|
||||
(root.parent/'outside.html').write_bytes((root/'paper.html').read_bytes())
|
||||
(root/'paper.html').unlink()
|
||||
(root/'paper.html').symlink_to(root.parent/'outside.html')
|
||||
(root/'receipts.json').write_text(json.dumps({'receipts': receipts}))
|
||||
with self.assertRaises(ValueError): rss.Pages(snapshots=root).read(self.index[1]['url'])
|
||||
|
||||
def test_unreviewed_edition_and_existing_output_refuse_work(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with self.assertRaises(ValueError): rss.collect(2026, tmp, snapshots=FIXTURES)
|
||||
(Path(tmp)/'rss-2025.json').write_text('{}')
|
||||
with self.assertRaises(FileExistsError): rss.collect(2025, tmp, snapshots=FIXTURES)
|
||||
|
||||
def test_native_roundtrip_is_idempotent_with_same_doi_and_distinct_tracks(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
original = self.paper()
|
||||
dblp = {**original, 'source': 'dblp', 'external_id': 'conf/rss/example', 'metadata': {}, 'tags': []}
|
||||
variant = {**original, 'external_id': '2025/other-track', 'track': 'workshop'}
|
||||
for n, record in enumerate([dblp, original, variant, original]):
|
||||
path = root/f'{n}.jsonl'
|
||||
path.write_text(json.dumps(record)+'\n')
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps({'source': record['source'], 'list_url': rss.ROOT, 'count': 1}))
|
||||
ingest_file(root/'sample.sqlite', path)
|
||||
with sqlite3.connect(root/'sample.sqlite') as c:
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0], 1)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM source_records').fetchone()[0], 3)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM submissions').fetchone()[0], 2)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM decisions').fetchone()[0], 2)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM submission_sources').fetchone()[0], 3)
|
||||
self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
|
||||
def test_two_real_papers_collection_commits_evidence_only_after_success(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
index = {pid: self.index[pid] for pid in [1, 2]}
|
||||
accepted = {pid: self.accepted[pid] for pid in [1, 2]}
|
||||
demos = {2: self.demos[2]}
|
||||
with (patch.object(rss, 'EXPECTED', 2), patch.object(rss, 'DEMO_IDS', {2}),
|
||||
patch.object(rss, 'parse_index', return_value=index),
|
||||
patch.object(rss, 'parse_program', side_effect=[accepted, demos])):
|
||||
result = rss.collect(2025, root/'out', snapshots=FIXTURES)
|
||||
self.assertEqual((result['records'], len(result['files'])), (2, 2))
|
||||
self.assertFalse(result['database_written'])
|
||||
self.assertFalse(result['pdfs_downloaded'])
|
||||
self.assertEqual(len(result['source_evidence']), 6)
|
||||
for item in result['source_evidence']:
|
||||
self.assertEqual(hashlib.sha256((root/'out'/item['file']).read_bytes()).hexdigest(), item['sha256'])
|
||||
for item in result['files']:
|
||||
self.assertEqual(item['missing_abstract_count'], 0)
|
||||
self.assertEqual(hashlib.sha256(Path(item['file']).read_bytes()).hexdigest(), item['file_sha256'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -13,7 +13,7 @@ import uuid
|
||||
|
||||
from paper_atlas.db.backend import open_store
|
||||
from paper_atlas.db.edition_scopes import KEY
|
||||
from paper_atlas.db.source_revisions import available, require_row, require_manifest
|
||||
from paper_atlas.db.source_revisions import available, decode_payload, require_row, require_manifest
|
||||
from paper_atlas.db.store import Store
|
||||
from paper_atlas.ingestion.source_revisions import proposal, apply_plan, run, validate
|
||||
from paper_atlas.ingestion.event_identity import database_identity, digest, values
|
||||
@@ -69,6 +69,28 @@ class RevisionCases:
|
||||
def plan(self, c):
|
||||
return proposal(c, self.inventory, database_identity(c, self.target.get('db')))
|
||||
|
||||
def test_large_payload_round_trips_and_protects_next_revision(self):
|
||||
for key in ('previous_rows', 'rows'):
|
||||
self.inventory[key][-1]['metadata']['evidence'] = '来源原文' * 100000
|
||||
with closing(self.store().conn) as c:
|
||||
plan = self.plan(c)
|
||||
applied = apply_plan(c, plan)
|
||||
c.commit()
|
||||
saved = c.execute('SELECT records_json,plan_json FROM source_scope_revisions').fetchone()
|
||||
expected = (self.inventory['rows'], plan)
|
||||
for i, value in enumerate(expected):
|
||||
self.assertTrue(saved[i].startswith('gzip+base64:v1:'))
|
||||
self.assertLess(len(saved[i]), len(json.dumps(value, ensure_ascii=False).encode()) // 10)
|
||||
self.assertEqual(decode_payload(saved[i]), value)
|
||||
self.assertEqual(apply_plan(c, plan)['status'], 'already_applied')
|
||||
sid = c.execute("SELECT id FROM sources WHERE name='fixture'").fetchone()[0]
|
||||
require_row(c, sid, self.inventory['rows'][-1], {'list_url': URL}, {})
|
||||
next_inventory = copy.deepcopy(self.inventory)
|
||||
next_inventory['previous_rows'] = copy.deepcopy(self.inventory['rows'])
|
||||
next_inventory['rows'][-1]['metadata']['reviewed'] = True
|
||||
next_plan = proposal(c, next_inventory, database_identity(c, self.target.get('db')))
|
||||
self.assertEqual(next_plan['before']['parent_sha256'], applied['version_sha256'])
|
||||
|
||||
def test_move_to_populated_target_preserves_full_history_and_before_images(self):
|
||||
with closing(self.store().conn) as c:
|
||||
plan = self.plan(c); self.assertEqual(len(plan['changes']), 1)
|
||||
|
||||
Reference in New Issue
Block a user