Compare commits
13 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 codex-bot
|
5d858ab8e2 | ||
|
codex-bot
|
bf4d0cd38a | ||
 Zayden-Jung
|
ec9edd88d4 | ||
|
codex-bot
|
b5c0618be5 | ||
|
codex-bot
|
81864e48ae | ||
|
Zayden-Jung
|
8ad052f577 | ||
|
codex-bot
|
348ebe0695 | ||
|
codex-bot
|
aa696bd7b5 | ||
|
codex-bot
|
90bd534edd | ||
|
Zayden-Jung
|
02a408ba8f | ||
|
codex-bot
|
714f745bd5 | ||
|
Zayden-Jung
|
cc78e837d2 | ||
|
codex-bot
|
c0b2146ae8 |
@@ -50,6 +50,8 @@ jobs:
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','cvf','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','www-archive','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','aamas-proceedings','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','iclr-archive','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','rss-proceedings','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','postgres','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','edition-tracks','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','joint-editions','--help'],cwd=outside,check=True)
|
||||
|
||||
@@ -6,6 +6,10 @@ Paper Atlas 将 arXiv、AI 会议论文集及公开投稿决定整理到可追
|
||||
|
||||
项目仍在建设中。候选范围包含 43 个会议（含 ACM MM、MLSys），尚未完成全部历史数据和最终去重复核。实际进度以数据库、来源清单和覆盖报告为准，而不是仓库文件数量。
|
||||
|
||||
历史 ICLR 2014–2016 官方录用页由 `collect iclr-archive` 适配，保留主会/workshop及重复展示证据；运行和身份核验边界见 [历史档案说明](docs/coverage/iclr-archive-2014-2016.md)。
|
||||
|
||||
RSS 2025 的 `collect rss-proceedings` 读取官方逐篇标题、完整作者、摘要和 DOI，并核对录用与演示节目单；不下载 PDF。15 个演示 ID 有明确来源，其余逐篇投稿类型保持未明确，不用标题或会话主题代替 track。运行、证据恢复和限制见 [RSS 元数据说明](docs/rss-proceedings.md)。
|
||||
|
||||
## 项目结构
|
||||
|
||||
```text
|
||||
@@ -62,6 +66,8 @@ paper-atlas collect colm --cache /data/paper-index/raw/http-cache \
|
||||
|
||||
早期主系列的Workshop名称按[ICML/IROS精确届次证据](docs/coverage/historical-primary-series-2026-11-09.md)识别，保留原名与范围证据；代码支持不代表已补齐生产数据。
|
||||
|
||||
EWSL、EPS、ECP、EWSP 的[具体前身卷与完整快照要求](docs/coverage/historical-predecessors-2026-11-09.md)单独核验，保留原始系列及事件/出版年份；旧候选文件缺少 ECP/EWSP 时会明确提示，避免静默遗漏。
|
||||
|
||||
AAMAS 的[官方论文集采集器](docs/aamas-proceedings.md)目前核验了 2012 版式，可从缓存或已校验快照采集标题、作者及页面提供的摘要；发表形式单独保存，全文/扩展摘要中未明确的投稿 track 保持待明确，不根据会话主题推定主会轨道。
|
||||
|
||||
`paper-atlas collect --help` 查看来源；`paper-atlas import --help` 查看导入方式。完整批次步骤见[使用指南](docs/usage.md)。
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
# EWSL 与规划会议前身的来源范围
|
||||
|
||||
本项扩展 DBLP 元数据适配：核对具体历史卷，保留 EWSL、EPS、ECP、EWSP 原始系列名称，并使用已有会议别名关系关联 ECML-PKDD 或 ICAPS。它不把历史论文的原始会议名称替换成后来的会议，也不从卷的存在推断有逐篇数据。
|
||||
|
||||
## 证据与已核验边界
|
||||
|
||||
[ECML-PKDD 2011 官方历史页](https://ecmlpkdd2011.org/pastConferences.php)与 [ICAPS 官方系列历史](https://www.icaps-conference.org/conference-series/)分别提供系列前身关系。2026-11-09 冻结页面 SHA-256 为：
|
||||
|
||||
- ECML-PKDD：`c3ecdb032aeacf6a365aace5457773827d1405872ad9bc6be6fc05851d5e68d6`。
|
||||
- ICAPS：`de1d217660cb819825f58112cdaf774d0f68da6ed87e3e90a59055e52f152081`。
|
||||
|
||||
与固定 DBLP 2026-11-01 快照的完整 proceedings 目录对照后，仅对下列具体卷识别前身主系列：
|
||||
|
||||
| 原始系列 | 事件年 | DBLP 卷 key | 卷出版年 | 原始逐篇书目计数 |
|
||||
|---|---:|---|---:|---:|
|
||||
| EWSL | 1987 | `conf/ecml/1987` | 1987 | 16 |
|
||||
| EWSL | 1988 | `conf/ecml/1988` | 1988 | 21 |
|
||||
| EWSL | 1989 | `conf/ecml/1989` | 1989 | 0 |
|
||||
| EWSL | 1991 | `conf/ecml/1991` | 1991 | 44 |
|
||||
| EPS | 1990 | `conf/aips/1990` | 1990 | 48 |
|
||||
| ECP | 1997 | `conf/ecp/1997` | 1997 | 35 |
|
||||
| ECP | 1999 | `conf/ecp/1999` | 2000 | 28 |
|
||||
| EWSP | 1991 | `conf/ewsp/1991` | 1991 | 10 |
|
||||
|
||||
合计 202 条是原始 `inproceedings` 书目计数，尚未经过前言等材料排除、身份核查与生产验收，不能作为新论文数量。EWSL 1989 在当前 catalogue 中有卷信息，但未匹配到 `inproceedings` 条目；其他记录类型和其他来源仍待核查。代码支持它的范围规则不表示已补上这一年的论文。
|
||||
|
||||
核验同时要求来源 family、booktitle 基名、空轨道后缀、完整 proceedings key、完整标题及卷出版年与固定证据一致。邻近年份、标题附加内容、`w` 卷后缀或工作坊 booktitle 均不能继承此范围。ECP、EWSP 及 EPS 的未知卷保持待审。原有 AIPS、ICML/IROS、AISTATS/COLT 规则保持各自边界。
|
||||
|
||||
EWSL 的主轨道仍属于 `EWSL`，通过 `ECML-PKDD` 别名纳入家族查询；EPS、ECP、EWSP 保留原名，通过 `ICAPS` 别名关联。`main` 在此表示该原始系列的主轨道。原始 booktitle、作者、论文书目年、卷标题、卷出版年、事件年及历史页 URL 分开保存在 metadata 中。例如 ECP 1999 的事件年 1999 与卷出版年 2000 都保留，不用其中一个覆盖另一个。
|
||||
|
||||
## 候选文件与完整快照
|
||||
|
||||
旧的窄范围 `dblp-ai-candidates.jsonl` 未纳入 `ecp`、`ewsp` family，但完整 catalogue 的 proceedings 与 volume_counts 中有它们。新的 catalogue 导出会纳入这两个 family。
|
||||
|
||||
选择 ICAPS 且截止年达到 EWSP 1991 或 ECP 1997 时，若候选 manifest 缺少相应 family，`collect dblp --candidates` 会明确报错，要求使用原始完整 `--archive`。错误发生在写输出之前，不会把遗漏当作成功采集。选择其他会议或截止 EPS 1990 的旧候选采集不受这项新家族检查影响。
|
||||
|
||||
在已确认的远程运行环境中使用完整快照：
|
||||
|
||||
```sh
|
||||
paper-atlas collect dblp \
|
||||
--catalogue /data/paper-index/collected/dblp/dblp-catalogue.sqlite \
|
||||
--archive /path/to/pinned-dblp.xml.gz --dtd /path/to/pinned-dblp.dtd \
|
||||
--venue ECML-PKDD --venue ICAPS --through-year 2001 \
|
||||
--out /data/paper-index/collected/historical-predecessors
|
||||
```
|
||||
|
||||
完整快照与 DTD 均沿用现有固定身份校验。该命令也会输出已覆盖的其他年度；生产导入前须单独筛选目标卷与年度，检查 raw source、前言材料、稳定标识、标题、作者及现有投稿，不能直接导入整个输出目录。不会下载论文 PDF。
|
||||
|
||||
## 验证和当前限制
|
||||
|
||||
回归检查使用独立保存的八卷原始目录 fixture，合成论文只用于隔离测试；不代表真实逐篇来源。检查原始系列/别名、完整标题和年份边界、旧候选文件显式失败、扩展 family 的 catalogue 导出、原生导入后的元数据与关系。
|
||||
|
||||
代码、测试、采集和生产导入是不同阶段。新范围不表示本批数据已采集或已导入，更不表示公开拒稿、摘要或最终去重完整。原始书目仅提供发表条目证据，不是原始决定信；没有记录不能推断为拒稿或未举办。后续各批需保留独立验收报告，保护既有跨会投稿、决定与标签历史。
|
||||
@@ -0,0 +1,44 @@
|
||||
# ICLR 2014–2016 官方历史录用档案
|
||||
|
||||
此适配器是长期维护工具的一部分，按需运行。只识别已经独立核对的三种页面布局；后续页面发生变化必须重新核查，不能放松计数或轨道边界以使采集成功。
|
||||
|
||||
| 官方页面 | 页面链接次数 | 整理后的不同记录 |
|
||||
|---|---:|---:|
|
||||
| [2014 Conference Proceedings](https://iclr.cc/archive/2014/conference-proceedings/) | 35 | main 35 |
|
||||
| [2015 accepted-main](https://iclr.cc/archive/www/doku.php%3Fid=iclr2015:accepted-main.html) | 116 | main 31、workshop 74 |
|
||||
| [2016 accepted-main](https://iclr.cc/archive/www/doku.php%3Fid=iclr2016:accepted-main.html) | 80 | main 80 |
|
||||
|
||||
2015 的 11 篇 oral 全部再次出现在 poster 段落中。相同年份、轨道及 arXiv 标识的重复展示只生成一条来源记录，但每次展示的标题、作者显示字符串、链接、版本和段落均保留在 `source_variants`。两处标题或解析作者冲突时停止采集、等待人工核查。不同年份或不同轨道不合并投稿。
|
||||
|
||||
2014 的作者以分号列出，一条有末尾分号，一条标题和作者斜体块不在普通段落中，部分作者段落拆成多个格式片段；必须读取完整作者段落或独立斜体块，不能只读首个格式片段。2016 的代码链接不属于作者。保留原文，解析结果不替代来源证据。oral、poster 作为 `iclr_archive:presentation` 标签，不改成独立投稿轨道。arXiv 版本留在来源变体中，稳定标识使用去版本号的编号。
|
||||
|
||||
## 来源快照与运行
|
||||
|
||||
2026-11-09 已冻结的官方响应 SHA256：
|
||||
|
||||
- 2014：`a33858176fbffae0066ed3c10310f7dd98a66d12a36334b6339ef1f915ef07af`
|
||||
- 2015：`131491663742cb084662a4804009b8318e0fa7bc184d4a9044d16dd28f05b888`
|
||||
- 2016：`7320b8652431793b6ea0743f871ee446bf08b6361ccd4e421b6b53449e71f4f6`
|
||||
|
||||
在服务器运行，输出新目录，不下载论文 PDF：
|
||||
|
||||
```sh
|
||||
paper-atlas collect iclr-archive --year 2015 \
|
||||
--snapshots /data/paper-index/collected/historical-next-source-evidence-v1 \
|
||||
--out /data/paper-index/collected/iclr-archive-source-v1/2015
|
||||
|
||||
# 或重新下载官方 HTML，并保留 HTTP 缓存与收据
|
||||
paper-atlas collect iclr-archive --year 2016 \
|
||||
--cache /data/paper-index/cache/iclr-archive \
|
||||
--out /data/paper-index/collected/iclr-archive-source-v1/2016
|
||||
```
|
||||
|
||||
冻结输入目录必须包含 `receipts.json`，每项有精确 `url`、HTTP `status: 200`、相对 `file`、`bytes`、`sha256` 和带时区的 `finished_at`。读取时重新核对尺寸、哈希和路径；拒绝越界及指向目录外的软链接。已完成输出不可覆盖，应检查报告后另建明确版本目录。解析或计数失败时不生成完成报告，也不导入数据库。
|
||||
|
||||
输出分轨道 JSONL、manifest、原始 HTML 证据和年度采集报告。仅支持这三年已审查的页面。2014 workshop、其他未列轨道、逐篇摘要及公开拒稿范围尚未通过这些页面核验，不能据此声称完整。缺席不代表 rejected、withdrawn 或从未被录用。
|
||||
|
||||
## 导入前后的边界
|
||||
|
||||
这三页合计 220 条不同来源记录，不等于新增 220 篇论文；同一工作可能已存在于 arXiv 或其他会议。先核验来源字段与原始页面，再执行 PostgreSQL 只读预检，比较稳定标识、标题、作者和原始来源。标题/作者差异、版本关联及冲突必须保留证据并明确评估；不能仅靠标题猜测身份，也不能覆盖原有摘要。
|
||||
|
||||
采集器不连接数据库。正式导入使用既有 `import records`，必须有新的生产保护基线和独立验收；全部导入结束后仍需全库去重和历史覆盖审计。来自官方录用清单的 accepted 是清单观察，不是原始审稿决定签名或决定日期。
|
||||
@@ -0,0 +1,38 @@
|
||||
# RSS 官方论文元数据
|
||||
|
||||
这是长期维护、按需运行的 package collector，不是定时服务或一次性导入脚本。目前仅接受独立核对过的 2025 年版式；不把 2026 年或其他年份套用同一分类规则。
|
||||
|
||||
## 数据范围和证据
|
||||
|
||||
- [RSS XXI 官方论文集](https://www.roboticsproceedings.org/rss21/)列出 163 个不同的逐篇 HTML 链接。每篇有 citation 元数据、BibTeX 和摘要区；PDF 链接作为原始元数据保存，不请求全文。
|
||||
- [官方 Accepted Papers](https://roboticsconference.org/2025/program/papers/)按论文 ID、标题和 session 对账。网页作者列包含隐藏的重复展示，逐篇有序 citation_author 与论文集目录作者完整对比后才采用；不截断作者、不将重复展示当新论文。
|
||||
- [官方 Demos](https://roboticsconference.org/2025/program/demos/)明确列出 15 个论文 ID，包括标题不以 Demonstrating 开头的 ID 93。这些条目标为 `demo`，保留整行节目证据。其他 148 条使用 `proceedings_unspecified`：没有逐篇类型依据时，不因缺席演示表就自动推断 main。
|
||||
- [作者说明](https://roboticsconference.org/2025/information/authorinfo/)描述 Science/Systems 与 Demo 两类投稿，两类都进入论文集。分类属于该来源的投稿/发表语义，不自动生成通用 research_type 标签。
|
||||
|
||||
`rss_proceedings:session` 保留官方 session，`rss_proceedings:program_role` 保留明确的 demonstration。二者与其他平台标签隔离；session 主题不等于投稿 track。单篇来源 ID 是 `2025/pNNN`，DOI 同时保留原始 BibTeX 大小写和标准化身份键，显式的 `publication_doi` 供原生导入器在同一会议年度/track 中关联其他正式发表来源；不能据此合并不同 track 或不同会议的投稿经历。
|
||||
|
||||
只有已发表、被官方录用表列出的条目才产生 accepted 记录。采集观察时间来自响应回执；出版日期单独留在 metadata，不伪造原始决定日期。来源未列出的论文不能推断为 rejected、desk rejected 或 withdrawn；公开负面决定是另一个待核范围。
|
||||
|
||||
## 运行
|
||||
|
||||
构建、安装和项目测试仅使用已确认的 `ces-control:host` 远程 runner。本机不能执行隐式构建的验证。服务器已安装环境运行示例：
|
||||
|
||||
```sh
|
||||
paper-atlas collect rss-proceedings --year 2025 \
|
||||
--snapshots /data/paper-index/collected/rss2025-full-evidence-v1 \
|
||||
--out /data/paper-index/collected/rss2025-source-v1
|
||||
```
|
||||
|
||||
也可使用 `--cache /data/paper-index/raw/rss-http` 请求限定官方 HTML 元数据。两种输入互斥。冻结快照的 `receipts.json` 每项必须包含 `file`、准确 `url/final_url`、`http_status: 200`、`content_type: text/html`、`bytes`、`sha256` 和含时区的观察时间 `at`。路径必须位于快照内，不能通过符号链接逃逸。所有输入逐个核验字节数与哈希，UTF-8 解码；重定向、未知版式和异常 ID 需要先核查。
|
||||
|
||||
采集只输出 JSONL、每个 track 的 manifest、完整 HTML 证据和 `rss-2025.json` 来源报告，不连接生产数据库。两份清单状态保持 `partial`；即使该卷 163 个链接全部解析，也不表示独立作品去重、逐篇类型或公开负面决定全集已完成。明确为空的摘要保存 `null` 和原因；缺失或重复摘要边界属于版式异常，停止输出。
|
||||
|
||||
## 冻结与恢复
|
||||
|
||||
2026-11-09 在服务器冻结 167 个官方 HTML 响应：目录、录用表、演示表、作者说明和 163 个逐篇页面。回执 SHA256：`f427d93a113702030864e789efe1176d69f71d2c567e81536d155e53d9513a93`。六个未改写样例位于 `tests/fixtures/rss2025/`，附其真实响应回执；完整原件保存在服务器来源目录与独立审阅备份，不作为运行时下载依赖。
|
||||
|
||||
整个采集持有输出目录的排他锁。全部解析和交叉检查通过后才发布 JSONL、manifest 和完成报告；完成报告最后发布。已有完成报告或 JSONL 时拒绝覆盖。若发布中断，先核对目录、回执和所有文件哈希，保留现场，再换新输出目录重采；不能看到日志安静就重启或重复导入。新的官方页面内容或计数与冻结样本不同，必须另留快照、解释变化并验证后再更新采集器。
|
||||
|
||||
生产导入前必须另外做来源独立核验和 PostgreSQL 只读身份预检。DOI、同标题候选和既有 arXiv 实体需要逐项核查，保留跨会的所有投稿和决定关系；发现冲突时隔离，不能只按标题强合并。已导入队列不得重放。生产独立验收通过后才重新统计年度缺口；导入成功和 CI 不能替代最终全库去重、摘要/标签及约束和实际查询审计。
|
||||
|
||||
本说明中的 163/15 是冻结官方页面的来源清单，不是当前生产库导入结果。历史缺口在本批生产验收前仍为 47。
|
||||
@@ -13,6 +13,8 @@ COMMANDS = {
|
||||
('download',): ('paper_atlas.io.snapshots', {}, '下载并校验固定版本快照'),
|
||||
('collect','pmlr'): ('paper_atlas.collectors.pmlr', {}, 'PMLR 官方元数据'),
|
||||
('collect','acl'): ('paper_atlas.collectors.acl', {}, 'ACL Anthology XML'),
|
||||
('collect','iclr-archive'): ('paper_atlas.collectors.iclr_archive', {}, '核对ICLR 2014–2016官方历史录用档案'),
|
||||
('collect','rss-proceedings'): ('paper_atlas.collectors.rss', {}, '核对RSS 2025官方论文摘要与演示证据'),
|
||||
('collect','iclr-program'): ('paper_atlas.collectors.iclr', {}, 'ICLR 官方公开程序'),
|
||||
('collect','iclr-snapshot'): ('paper_atlas.collectors.openreview', {}, 'ICLR 公开投稿与决定快照'),
|
||||
('collect','public-openreview'): ('paper_atlas.collectors.public_openreview', {}, '核验其他会议的公开拒稿及撤稿标签'),
|
||||
|
||||
@@ -20,7 +20,7 @@ DTD_MD5='96e4283e71886dec2173718957293648'
|
||||
DTD_NAMES={'dblp.dtd','dblp-2023-06-28.dtd'}
|
||||
RELEASE='https://doi.org/10.4230/dblp.xml.2026-11-01'
|
||||
# Broad discovery candidates, including historical names; not a venue mapping.
|
||||
FAMILIES=set('nips icml iclr colm colt aistats uai clear acl emnlp naacl eacl coling conll cvpr iccv eccv aaai ijcai ecai kdd sigir www wsdm recsys cikm icdm sdm ecml pkdd ecmlpkdd icra iros rss corl aips icaps kr atal aamas icassp interspeech eurospeech icslp ismir miccai fat facct lrec mm mlsys'.split())
|
||||
FAMILIES=set('nips icml iclr colm colt aistats uai clear acl emnlp naacl eacl coling conll cvpr iccv eccv aaai ijcai ecai kdd sigir www wsdm recsys cikm icdm sdm ecml pkdd ecmlpkdd icra iros rss corl aips icaps ecp ewsp kr atal aamas icassp interspeech eurospeech icslp ismir miccai fat facct lrec mm mlsys'.split())
|
||||
|
||||
|
||||
def records(archive,dtd_bytes):
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Exact predecessor volumes, with original names and independent family histories."""
|
||||
ECML_HISTORY = 'https://ecmlpkdd2011.org/pastConferences.php'
|
||||
ICAPS_HISTORY = 'https://www.icaps-conference.org/conference-series/'
|
||||
|
||||
# family/year: booktitle, original series, target family, publication year, full title
|
||||
VOLUMES = {
|
||||
('ecml', 1987): ('EWSL', 'EWSL', 'ECML-PKDD', 1987,
|
||||
'Progress in Machine Learning - Proceedings of EWSL 87: 2nd European Working Session on Learning, Bled, Yugoslavia, May 1987'),
|
||||
('ecml', 1988): ('EWSL', 'EWSL', 'ECML-PKDD', 1988,
|
||||
'Proceedings of the Third European Working Session on Learning, EWSL 1988, Turing Institute, Glasgow, UK, October 3-5, 1988'),
|
||||
('ecml', 1989): ('EWSL', 'EWSL', 'ECML-PKDD', 1989,
|
||||
'Proceedings of the 4th European Working Session on Learning, Montpellier, France, 4-6 December 1989'),
|
||||
('ecml', 1991): ('EWSL', 'EWSL', 'ECML-PKDD', 1991,
|
||||
'Machine Learning - EWSL-91, European Working Session on Learning, Porto, Portugal, March 6-8, 1991, Proceedings'),
|
||||
('aips', 1990): ('Expert Planning Systems', 'EPS', 'ICAPS', 1990,
|
||||
'Proceedings of the First International Conference on Expert Planning Systems, Brighton, UK, June 27-29, 1990'),
|
||||
('ecp', 1997): ('ECP', 'ECP', 'ICAPS', 1997,
|
||||
"Recent Advances in AI Planning, 4th European Conference on Planning, ECP'97, Toulouse, France, September 24-26, 1997, Proceedings"),
|
||||
('ecp', 1999): ('ECP', 'ECP', 'ICAPS', 2000,
|
||||
"Recent Advances in AI Planning, 5th European Conference on Planning, ECP'99, Durham, UK, September 8-10, 1999, Proceedings"),
|
||||
('ewsp', 1991): ('EWSP', 'EWSP', 'ICAPS', 1991,
|
||||
"European Workshop on Planning, EWSP '91, Sankt Augustin, Germany, March 18-19, 1991, Proceedings"),
|
||||
}
|
||||
|
||||
|
||||
def predecessor_scope(family, base, suffix, volume, year, title):
|
||||
definition = VOLUMES.get((family, year))
|
||||
if definition is None:
|
||||
return None
|
||||
book, original, target, publication_year, expected_title = definition
|
||||
if (suffix or base != book or volume['key'] != f'conf/{family}/{year}'
|
||||
or title != expected_title or volume['fields'].get('year') != [str(publication_year)]):
|
||||
return None
|
||||
return {'original_series': original, 'target_family': target,
|
||||
'history_url': ECML_HISTORY if target == 'ECML-PKDD' else ICAPS_HISTORY,
|
||||
'proceedings_key': volume['key'], 'proceedings_title': title,
|
||||
'event_year': year, 'proceedings_year_fields': list(volume['fields']['year'])}
|
||||
|
||||
|
||||
def missing_candidate_families(targets, families, through_year):
|
||||
# These two families were absent from the earlier narrowed candidate JSONL,
|
||||
# although their proceedings and volume_counts exist in the full catalogue.
|
||||
required = {family for (family, year), definition in VOLUMES.items()
|
||||
if family in ('ecp', 'ewsp') and definition[2] in targets and year <= through_year}
|
||||
return sorted(family for family in required if not families.get(family))
|
||||
@@ -15,6 +15,7 @@ from urllib.parse import parse_qs,unquote,urlsplit
|
||||
from paper_atlas.config import DEFAULT_REGISTRY
|
||||
from paper_atlas.collectors.dblp import XML_SHA,DTD_MD5,RELEASE,records
|
||||
from paper_atlas.collectors.dblp_scope import scope
|
||||
from paper_atlas.collectors.dblp_predecessors import missing_candidate_families
|
||||
from paper_atlas.db.submissions import timestamp
|
||||
from paper_atlas.io.snapshots import digest_file
|
||||
from paper_atlas.io.publication_ids import mlsys_id
|
||||
@@ -88,6 +89,8 @@ def normalize_row(row,volumes,targets,include_workshops=False,through_year=2026)
|
||||
'scope_rule':edition['rule'],'author_names_raw':fields.get('author',[]),'publtype':row.get('publtype'),
|
||||
'bibliography':{key:value for key,value in fields.items() if key in ('booktitle','year','ee','url','pages','volume','number','publisher','series','isbn')},
|
||||
'ambiguous_identifiers':ambiguous,'publication_evidence':'inproceedings bibliography; not an original decision note'}
|
||||
if edition.get('historical_predecessor_evidence'):
|
||||
metadata['historical_predecessor_evidence'] = edition['historical_predecessor_evidence']
|
||||
if edition.get('joint_edition_evidence'):
|
||||
metadata['joint_edition_evidence'] = edition['joint_edition_evidence']
|
||||
if edition.get('track_scope_note'):
|
||||
@@ -124,8 +127,9 @@ def collect(catalogue,out,venues=None,candidates=None,archive=None,dtd=None,incl
|
||||
families = manifest.get('candidate_family_counts', {})
|
||||
missing = [venue for venue, family in [('MM', 'mm'), ('MLSys', 'mlsys')]
|
||||
if venue in targets and not families.get(family)]
|
||||
missing += ['ICAPS/'+family for family in missing_candidate_families(targets, families, through_year)]
|
||||
if missing:
|
||||
raise ValueError('Candidate file predates requested families ' + ', '.join(missing) + '; use the original --archive')
|
||||
raise ValueError('Candidate file omits requested families ' + ', '.join(missing) + '; use the original --archive')
|
||||
out=Path(out);out.mkdir(parents=True,exist_ok=True)
|
||||
with (out/'dblp.normalize.lock').open('a') as lock:
|
||||
fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
"""Explicit bibliographic aliases; a directory family alone is not venue evidence."""
|
||||
import re
|
||||
|
||||
from paper_atlas.collectors.dblp_predecessors import predecessor_scope
|
||||
|
||||
|
||||
# Ordered long/joint names precede their shorter forms.
|
||||
BOOKS = {
|
||||
@@ -27,7 +29,8 @@ BOOKS = {
|
||||
'pkdd':[('ECML/PKDD','ECML-PKDD',[]),('ECML PKDD','ECML-PKDD',[]),('PKDD','PKDD',['ECML-PKDD'])],
|
||||
'icra':[('ICRA','ICRA',[])], 'iros':[('IROS','IROS',[])],
|
||||
'rss':[('Robotics: Science and Systems','RSS',[])], 'corl':[('CoRL','CoRL',[])],
|
||||
'aips':[('ICAPS','ICAPS',[]),('AIPS','AIPS',['ICAPS'])], 'icaps':[('ICAPS','ICAPS',[])],
|
||||
'aips':[('ICAPS','ICAPS',[]),('AIPS','AIPS',['ICAPS']),('Expert Planning Systems','EPS',['ICAPS'])], 'icaps':[('ICAPS','ICAPS',[])],
|
||||
'ecp':[('ECP','ECP',['ICAPS'])], 'ewsp':[('EWSP','EWSP',['ICAPS'])],
|
||||
'kr':[('KR','KR',[])], 'atal':[('AAMAS','AAMAS',[])],
|
||||
'icassp':[('ICASSP','ICASSP',[])],
|
||||
'interspeech':[('INTERSPEECH','INTERSPEECH',[]),('EUROSPEECH','EUROSPEECH',['INTERSPEECH']),('ICSLP','ICSLP',['INTERSPEECH']),('ECST','ECST',['INTERSPEECH'])],
|
||||
@@ -135,7 +138,11 @@ def scope(row,volume,through_year=2026):
|
||||
if len(bibliographic)!=1 or not re.fullmatch(r'\d{4}',bibliographic[0]):return None,'ambiguous_bibliographic_year'
|
||||
if int(bibliographic[0])!=year and not re.search(r'\b'+str(year)+r'\b',title):return None,'unverified_year_difference'
|
||||
text=(suffix+' '+title).casefold();track='main'
|
||||
historical = historical_track(family, base, suffix, volume['key'], year, title)
|
||||
predecessor = predecessor_scope(family, base, suffix, volume, year, title)
|
||||
if (family in ('ecp', 'ewsp') or (family == 'aips' and base == 'Expert Planning Systems')) and not predecessor:
|
||||
return None, 'unverified_historical_predecessor'
|
||||
historical = (('main', 'Historical predecessor; exact volume, full title, event and publication years verified; original series retained.', predecessor['history_url'])
|
||||
if predecessor else historical_track(family, base, suffix, volume['key'], year, title))
|
||||
if historical:track=historical[0]
|
||||
elif 'findings' in text:track='findings'
|
||||
elif 'shared task' in text:track='shared_task'
|
||||
@@ -165,6 +172,8 @@ def scope(row,volume,through_year=2026):
|
||||
joint_evidence = 'https://www.eurai.org/ecai/history'
|
||||
result = {'venue':venue,'venue_aliases':aliases,'year':year,'track':track,'bibliographic_year':int(bibliographic[0]),
|
||||
'rule':family+':'+base,'proceedings_title':title,'edition_year_basis':year_basis}
|
||||
if predecessor:
|
||||
result['historical_predecessor_evidence'] = predecessor
|
||||
if joint_evidence:
|
||||
result['joint_edition_evidence'] = joint_evidence
|
||||
if historical:
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
"""Frozen official ICLR 2014–2016 lists; presentation evidence is not a track."""
|
||||
import argparse
|
||||
from collections import Counter, defaultdict
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import tempfile
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from paper_atlas.db.submissions import timestamp
|
||||
from paper_atlas.io.html import Document, Node, clean
|
||||
from paper_atlas.io.http import fetch
|
||||
|
||||
SOURCE = 'iclr_archive'
|
||||
PAGES = {
|
||||
2014: ('https://iclr.cc/archive/2014/conference-proceedings/', {'main': 35}),
|
||||
2015: ('https://iclr.cc/archive/www/doku.php%3Fid=iclr2015:accepted-main.html', {'main': 31, 'workshop': 74}),
|
||||
2016: ('https://iclr.cc/archive/www/doku.php%3Fid=iclr2016:accepted-main.html', {'main': 80}),
|
||||
}
|
||||
SECTIONS = {
|
||||
2015: {'Main Conference - Oral Presentations': ('main', 'oral'),
|
||||
'Main Conference - Poster Presentations': ('main', 'poster'),
|
||||
'Workshop Papers': ('workshop', 'workshop')},
|
||||
2016: {'Oral Presentations': ('main', 'oral'), 'Poster Presentations': ('main', 'poster')},
|
||||
}
|
||||
|
||||
|
||||
def arxiv_link(url):
|
||||
parsed = urlsplit(url)
|
||||
if parsed.hostname != 'arxiv.org':
|
||||
return None
|
||||
match = re.fullmatch(r'/(?:abs|pdf)/(\d{4}\.\d{4,5})(v[1-9]\d*)?(?:\.pdf)?', parsed.path)
|
||||
if (parsed.scheme not in ('http', 'https') or parsed.netloc != 'arxiv.org'
|
||||
or parsed.query or parsed.fragment or not match):
|
||||
raise ValueError('Unexpected official arXiv link')
|
||||
return match[1], match[2] or None
|
||||
|
||||
|
||||
def without_links(node):
|
||||
if node.tag in ('a', 'script', 'style'):
|
||||
return ''
|
||||
value = ''.join(without_links(n) if isinstance(n, Node) else n for n in node.children)
|
||||
return ' '+value+' ' if node.tag in ('div', 'li', 'p', 'br') else value
|
||||
|
||||
|
||||
def authors(display, year):
|
||||
parts = display.rstrip(';').split(';') if year == 2014 else re.split(r',\s*(?:and\s+)?|\s+and\s+', display)
|
||||
names = [clean(p) for p in parts]
|
||||
if not names or any(not name for name in names):
|
||||
raise ValueError('Missing author or unsupported author delimiter')
|
||||
return names
|
||||
|
||||
|
||||
def parse_page(html, year, evidence):
|
||||
if year not in PAGES:
|
||||
raise ValueError('Unreviewed ICLR archive year')
|
||||
document = Document(html)
|
||||
nodes = list(document.root.walk())
|
||||
headings = [clean(n.text()) for n in nodes if n.tag in ('h1', 'h2', 'h3')]
|
||||
if year == 2014:
|
||||
if 'Conference Proceedings' not in headings or 'conference papers accepted' not in clean(document.root.text()):
|
||||
raise ValueError('Unexpected ICLR 2014 accepted list')
|
||||
elif (set(SECTIONS[year]) - set(headings)
|
||||
or (year == 2016 and 'Accepted Papers (Conference Track)' not in headings)):
|
||||
raise ValueError('Unexpected ICLR accepted sections')
|
||||
entries = []
|
||||
section = None
|
||||
for index, node in enumerate(nodes):
|
||||
if node.tag in ('h1', 'h2', 'h3'):
|
||||
section = clean(node.text())
|
||||
if year == 2014:
|
||||
if node.tag != 'a' or not arxiv_link(node.attrs.get('href', '')):
|
||||
continue
|
||||
link = node
|
||||
# This archived Google Sites page has an entry outside a paragraph.
|
||||
# Authors can span several differently formatted italic spans.
|
||||
# Read their whole paragraph, never only the first styled fragment.
|
||||
display = None
|
||||
for following in nodes[index+sum(1 for _ in link.walk()):]:
|
||||
if following.tag == 'a' and arxiv_link(following.attrs.get('href', '')):
|
||||
break
|
||||
if following.tag in ('i', 'em') or (following.tag == 'p' and any(
|
||||
n.tag in ('i', 'em') or 'font-style:italic' in n.attrs.get('style', '').replace(' ', '')
|
||||
for n in following.walk())):
|
||||
text = clean(following.text())
|
||||
if text:
|
||||
display = text
|
||||
break
|
||||
if not display:
|
||||
raise ValueError('Missing explicit ICLR 2014 author span')
|
||||
track, presentation = 'main', 'conference'
|
||||
else:
|
||||
if node.tag != 'li':
|
||||
continue
|
||||
links = [n for n in node.walk() if n.tag == 'a' and arxiv_link(n.attrs.get('href', ''))]
|
||||
if not links:
|
||||
continue
|
||||
if len(links) != 1 or section not in SECTIONS[year]:
|
||||
raise ValueError('Ambiguous paper link or unknown accepted section')
|
||||
link, = links
|
||||
display = clean(without_links(node)).lstrip(',').strip()
|
||||
track, presentation = SECTIONS[year][section]
|
||||
arxiv, version = arxiv_link(link.attrs['href'])
|
||||
title = clean(link.text())
|
||||
if not title:
|
||||
raise ValueError('Empty official title')
|
||||
entries.append({'arxiv': arxiv, 'version': version, 'title': title,
|
||||
'authors': authors(display, year), 'authors_display': display,
|
||||
'paper_url': link.attrs['href'], 'track': track,
|
||||
'section': section, 'presentation': presentation})
|
||||
# Check every arXiv occurrence, including links outside the expected layout.
|
||||
occurrences = sum(n.tag == 'a' and bool(arxiv_link(n.attrs.get('href', ''))) for n in nodes)
|
||||
if len(entries) != occurrences or not entries:
|
||||
raise ValueError('Some official paper links were not parsed')
|
||||
grouped = defaultdict(list)
|
||||
for entry in entries:
|
||||
grouped[(entry['track'], entry['arxiv'])].append(entry)
|
||||
result = []
|
||||
for (track, arxiv), variants in grouped.items():
|
||||
first = variants[0]
|
||||
if any((v['title'], v['authors']) != (first['title'], first['authors']) for v in variants):
|
||||
raise ValueError('Repeated presentation has conflicting title or authors; review required')
|
||||
presentations = sorted({v['presentation'] for v in variants})
|
||||
result.append({'source': SOURCE, 'external_id': f'{year}/{track}/{arxiv}',
|
||||
'url': PAGES[year][0], 'title': first['title'], 'abstract': None,
|
||||
'authors': first['authors'], 'identifiers': {'arxiv': arxiv},
|
||||
'venue': 'ICLR', 'venue_aliases': [], 'year': year, 'track': track,
|
||||
'status': 'accepted', 'observed_at': evidence['observed_at'],
|
||||
'raw_decision': 'listed in official ICLR accepted '+track+' list',
|
||||
'metadata': {'source_evidence': evidence, 'source_variants': variants,
|
||||
'presentations': presentations, 'decision_date_precision': 'not_supplied',
|
||||
'identity_note': 'Official arXiv link, not title matching; validate canonical identity before import.',
|
||||
'decision_evidence': 'Archived accepted list, not an original decision note.'},
|
||||
'tags': [{'namespace': SOURCE+':presentation', 'value': value} for value in presentations]})
|
||||
return result
|
||||
|
||||
|
||||
def read_page(year, *, cache=None, snapshots=None):
|
||||
if bool(cache) == bool(snapshots):
|
||||
raise ValueError('Choose exactly one cache or frozen snapshot directory')
|
||||
url = PAGES[year][0]
|
||||
if snapshots:
|
||||
root = Path(snapshots).resolve()
|
||||
receipts = json.loads((root/'receipts.json').read_bytes())['receipts']
|
||||
matches = [r for r in receipts if r['url'] == url]
|
||||
if len(matches) != 1:
|
||||
raise ValueError('Expected one exact official response receipt')
|
||||
receipt, = matches
|
||||
filename = Path(receipt['file'])
|
||||
path = root/filename
|
||||
if (filename.is_absolute() or '..' in filename.parts or not path.resolve().is_relative_to(root)
|
||||
or receipt['status'] != 200):
|
||||
raise ValueError('Unsafe or unsuccessful archive snapshot')
|
||||
data = path.read_bytes()
|
||||
if (len(data) > 8*1024**2 or len(data) != receipt['bytes']
|
||||
or hashlib.sha256(data).hexdigest() != receipt['sha256']):
|
||||
raise ValueError('Archive snapshot checksum or size differs')
|
||||
observed = timestamp(receipt['finished_at'])
|
||||
else:
|
||||
data = fetch(url, cache, 8*1024**2, delay=1)
|
||||
receipt = json.loads((Path(cache)/hashlib.sha256(url.encode()).hexdigest()).with_suffix('.json').read_bytes())
|
||||
observed = timestamp(receipt['retrieved_at'])
|
||||
evidence = {'url': url, 'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data),
|
||||
'file': 'evidence/'+hashlib.sha256(data).hexdigest()+'.html',
|
||||
'observed_at': observed, 'http_status': 200, 'response_receipt': receipt}
|
||||
return data, evidence
|
||||
|
||||
|
||||
def collect(year, out, *, cache=None, snapshots=None):
|
||||
if year not in PAGES:
|
||||
raise ValueError('Only independently reviewed ICLR archive years are supported')
|
||||
out = Path(out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
with (out/'iclr-archive.collect.lock').open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
report = out/f'iclr-archive-{year}.json'
|
||||
if report.exists():
|
||||
raise FileExistsError('Existing completed archive output must be inspected, not overwritten')
|
||||
data, evidence = read_page(year, cache=cache, snapshots=snapshots)
|
||||
records = parse_page(data.decode('utf-8'), year, evidence)
|
||||
counts = Counter(r['track'] for r in records)
|
||||
if dict(counts) != PAGES[year][1]:
|
||||
raise ValueError('Distinct archive entries differ from reviewed inventory')
|
||||
files = []
|
||||
with tempfile.TemporaryDirectory(prefix='.iclr-archive-', dir=out) as temporary:
|
||||
stage = Path(temporary)
|
||||
(stage/'evidence').mkdir()
|
||||
(stage/evidence['file']).write_bytes(data)
|
||||
for track in sorted(counts):
|
||||
group = [r for r in records if r['track'] == track]
|
||||
path = stage/f'iclr-archive-{year}-{track}.jsonl'
|
||||
path.write_text(''.join(json.dumps(r, ensure_ascii=False)+'\n' for r in group))
|
||||
manifest = {'source': SOURCE, 'parser_version': 1, 'venue': 'ICLR', 'year': year,
|
||||
'track': track, 'count': len(group), 'file': str(out/path.name),
|
||||
'file_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
'list_url': PAGES[year][0], 'status': 'partial', 'missing_abstract_count': len(group)}
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps(manifest, ensure_ascii=False, indent=2)+'\n')
|
||||
files.append(manifest)
|
||||
result = {'source': SOURCE, 'year': year, 'records': len(records), 'files': files,
|
||||
'source_evidence': evidence, 'database_written': False, 'pdfs_downloaded': False,
|
||||
'status': 'collected_pending_identity_review',
|
||||
'note': 'Accepted list only. Oral/poster are presentation evidence; absent papers are not rejected. '
|
||||
'ICLR2014 workshop and other unlisted tracks remain outside these reviewed pages.'}
|
||||
(stage/report.name).write_text(json.dumps(result, ensure_ascii=False, indent=2)+'\n')
|
||||
(out/'evidence').mkdir(exist_ok=True)
|
||||
for path in (stage/'evidence').iterdir():
|
||||
path.replace(out/'evidence'/path.name)
|
||||
for path in stage.iterdir():
|
||||
if path.is_file() and path.name != report.name:
|
||||
path.replace(out/path.name)
|
||||
(stage/report.name).replace(report)
|
||||
return result
|
||||
|
||||
|
||||
def main(argv=None, *, prog=None):
|
||||
parser = argparse.ArgumentParser(prog=prog)
|
||||
parser.add_argument('--year', type=int, choices=sorted(PAGES), required=True)
|
||||
parser.add_argument('--out', required=True)
|
||||
source = parser.add_mutually_exclusive_group(required=True)
|
||||
source.add_argument('--cache')
|
||||
source.add_argument('--snapshots')
|
||||
args = parser.parse_args(argv)
|
||||
print(json.dumps(collect(args.year, args.out, cache=args.cache, snapshots=args.snapshots), ensure_ascii=False))
|
||||
@@ -0,0 +1,279 @@
|
||||
"""Reviewed RSS 2025 proceedings HTML; no PDF or private review access."""
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from datetime import date
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import tempfile
|
||||
import unicodedata
|
||||
|
||||
from paper_atlas.db.submissions import timestamp
|
||||
from paper_atlas.io.html import Document, Node, clean
|
||||
from paper_atlas.io.http import fetch
|
||||
|
||||
SOURCE = 'rss_proceedings'
|
||||
ROOT = 'https://www.roboticsproceedings.org/rss21/'
|
||||
HEADING = 'Robotics: Science and Systems XXI'
|
||||
PAGES = {'index': ROOT,
|
||||
'accepted': 'https://roboticsconference.org/2025/program/papers/',
|
||||
'demos': 'https://roboticsconference.org/2025/program/demos/',
|
||||
'authorinfo': 'https://roboticsconference.org/2025/information/authorinfo/'}
|
||||
EXPECTED = 163
|
||||
DEMO_IDS = {2, 20, 21, 33, 58, 59, 62, 84, 86, 92, 93, 107, 129, 132, 141}
|
||||
|
||||
|
||||
def one(values, description):
|
||||
if len(values) != 1:
|
||||
raise ValueError('Expected one '+description)
|
||||
return values[0]
|
||||
|
||||
|
||||
def comparable(text):
|
||||
return unicodedata.normalize('NFC', clean(text))
|
||||
|
||||
|
||||
def parse_index(html):
|
||||
document = Document(html)
|
||||
nodes = list(document.root.walk())
|
||||
if [clean(n.text()) for n in nodes if n.tag == 'h1'] != [HEADING]:
|
||||
raise ValueError('Unexpected RSS proceedings edition')
|
||||
result = {}
|
||||
for row in (n for n in nodes if n.tag == 'tr'):
|
||||
links = [n for n in row.walk() if n.tag == 'a'
|
||||
and re.fullmatch(r'p[0-9]{3}\.html', n.attrs.get('href', ''))]
|
||||
if not links:
|
||||
continue
|
||||
link = one(links, 'proceedings entry link')
|
||||
pid = int(link.attrs['href'][1:4])
|
||||
title = clean(link.text())
|
||||
authors = clean(one([n for n in row.walk() if n.tag == 'i'], 'index authors').text())
|
||||
if pid in result or pid < 1 or not title or not authors:
|
||||
raise ValueError('Empty or repeated RSS index entry')
|
||||
result[pid] = {'title': title, 'authors_display': authors,
|
||||
'url': ROOT+link.attrs['href']}
|
||||
occurrences = sum(n.tag == 'a' and bool(re.fullmatch(r'p[0-9]{3}\.html', n.attrs.get('href', '')))
|
||||
for n in nodes)
|
||||
if not result or len(result) != occurrences:
|
||||
raise ValueError('Unparsed or duplicate proceedings links')
|
||||
return result
|
||||
|
||||
|
||||
def parse_program(html, heading, columns):
|
||||
nodes = list(Document(html).root.walk())
|
||||
if [clean(n.text()) for n in nodes if n.tag == 'h1'] != [heading]:
|
||||
raise ValueError('Unexpected RSS program page')
|
||||
headers = [[clean(n.text()) for n in row.children if isinstance(n, Node) and n.tag == 'th']
|
||||
for row in nodes if row.tag == 'tr']
|
||||
if list(columns) not in headers:
|
||||
raise ValueError('Unexpected RSS program columns')
|
||||
result = {}
|
||||
for row in (n for n in nodes if n.tag == 'tr'):
|
||||
cells = [n for n in row.children if isinstance(n, Node) and n.tag == 'td']
|
||||
if not cells:
|
||||
continue
|
||||
if len(cells) != len(columns):
|
||||
raise ValueError('Unexpected RSS program row width')
|
||||
raw_id = clean(cells[0].text())
|
||||
if not re.fullmatch('[1-9][0-9]*', raw_id):
|
||||
raise ValueError('Invalid program ID')
|
||||
pid = int(raw_id)
|
||||
if pid in result:
|
||||
raise ValueError('Duplicate program ID')
|
||||
result[pid] = {key: clean(node.text()) for key, node in zip(columns, cells)}
|
||||
if not result[pid]['Title']:
|
||||
raise ValueError('Missing program title')
|
||||
if not result:
|
||||
raise ValueError('No RSS program rows')
|
||||
return result
|
||||
|
||||
|
||||
def validate_program(index, accepted, demos, authorinfo):
|
||||
if set(index) != set(accepted) or len(index) != EXPECTED or set(demos) != DEMO_IDS:
|
||||
raise ValueError('RSS program and reviewed proceedings census differ')
|
||||
text = clean(Document(authorinfo).root.text())
|
||||
if ('There are two types of submissions to RSS' not in text
|
||||
or 'Both types of papers appear in the RSS proceedings.' not in text):
|
||||
raise ValueError('Submission category evidence changed')
|
||||
for pid, item in index.items():
|
||||
variants = [accepted[pid]['Title']] + ([demos[pid]['Title']] if pid in demos else [])
|
||||
if any(comparable(title) != comparable(item['title']) for title in variants):
|
||||
raise ValueError('Conflicting official program title for '+str(pid))
|
||||
|
||||
|
||||
def parse_paper(html, pid, entry, accepted, demo, evidence):
|
||||
nodes = list(Document(html).root.walk())
|
||||
meta = {}
|
||||
for node in nodes:
|
||||
if node.tag == 'meta' and node.attrs.get('name', '').startswith('citation_'):
|
||||
meta.setdefault(node.attrs['name'], []).append(node.attrs.get('content', ''))
|
||||
title = one(meta.get('citation_title', []), 'citation title')
|
||||
authors = meta.get('citation_author', [])
|
||||
publication_date = one(meta.get('citation_publication_date', []), 'publication date')
|
||||
if not re.fullmatch(r'2025/[0-9]{2}/[0-9]{2}', publication_date):
|
||||
raise ValueError('Unexpected publication year/date')
|
||||
date.fromisoformat(publication_date.replace('/', '-'))
|
||||
if (one(meta.get('citation_conference_title', []), 'conference') != HEADING
|
||||
or one(meta.get('citation_volume', []), 'volume') != '21'
|
||||
or [clean(n.text()) for n in nodes if n.tag == 'h1'] != [HEADING]
|
||||
or [comparable(n.text()) for n in nodes if n.tag == 'h3'] != [comparable(title)]
|
||||
or comparable(title) != comparable(entry['title']) or not authors
|
||||
or any(not clean(author) for author in authors)
|
||||
or comparable(', '.join(authors)) != comparable(entry['authors_display'])):
|
||||
raise ValueError('Proceedings/citation title, authors or edition disagree')
|
||||
pdf = one(meta.get('citation_pdf_url', []), 'citation PDF URL')
|
||||
if pdf not in (ROOT+f'p{pid:03d}.pdf', ROOT.replace('https:', 'http:')+f'p{pid:03d}.pdf'):
|
||||
raise ValueError('Citation PDF ID differs; URL is metadata only')
|
||||
bibtex = one([n.text() for n in nodes if n.tag == 'pre'], 'original BibTeX')
|
||||
doi = one(re.findall(r'^\s*DOI\s*=\s*\{([^{}]+)\}', bibtex, re.I | re.M), 'BibTeX DOI')
|
||||
year = one(re.findall(r'^\s*YEAR\s*=\s*\{([^{}]+)\}', bibtex, re.I | re.M), 'BibTeX year')
|
||||
if doi != f'10.15607/RSS.2025.XXI.{pid:03d}' or year != '2025':
|
||||
raise ValueError('BibTeX DOI/year differs from volume and paper ID')
|
||||
abstract_parts = []
|
||||
inside = False
|
||||
markers = []
|
||||
for node in (n for n in nodes if n.tag == 'p'):
|
||||
labels = [clean(n.text()) for n in node.walk() if n.tag == 'b']
|
||||
if 'Abstract:' in labels:
|
||||
markers.append('start')
|
||||
inside = True
|
||||
value = clean(node.text()).removeprefix('Abstract:').strip()
|
||||
if value:
|
||||
abstract_parts.append(value)
|
||||
elif 'Download:' in labels:
|
||||
markers.append('end')
|
||||
inside = False
|
||||
elif inside:
|
||||
abstract_parts.append(clean(node.text()))
|
||||
if markers != ['start', 'end']:
|
||||
raise ValueError('Abstract boundaries missing or repeated')
|
||||
abstract = clean(' '.join(abstract_parts)) or None
|
||||
track = 'demo' if demo else 'proceedings_unspecified'
|
||||
tags = [{'namespace': SOURCE+':session', 'value': accepted['Session']}]
|
||||
if demo:
|
||||
tags.append({'namespace': SOURCE+':program_role', 'value': 'demonstration'})
|
||||
return {'source': SOURCE, 'external_id': f'2025/p{pid:03d}', 'url': entry['url'],
|
||||
'title': title, 'abstract': abstract, 'authors': authors,
|
||||
'identifiers': {'doi': doi.lower()},
|
||||
'submission_identifiers': [{'scheme': 'publication_doi', 'value': doi.lower()}],
|
||||
'venue': 'RSS', 'venue_aliases': [], 'year': 2025, 'track': track,
|
||||
'status': 'accepted', 'observed_at': evidence['paper']['observed_at'],
|
||||
'raw_decision': 'listed in official RSS 2025 proceedings and accepted program',
|
||||
'tags': tags,
|
||||
'metadata': {'publisher_paper_id': pid, 'citation_metadata': meta,
|
||||
'publication_date': publication_date, 'original_bibtex': bibtex,
|
||||
'index_entry': entry, 'accepted_program_entry': accepted,
|
||||
'demo_program_entry': demo, 'source_evidence': evidence,
|
||||
'missing_abstract_reason': None if abstract else 'empty_official_abstract_section',
|
||||
'track_scope_note': ('Explicit official demo program ID; not inferred from title.' if demo else
|
||||
'Proceedings and accepted list do not individually identify submission type; session is not a track.'),
|
||||
'decision_date_precision': 'not_supplied',
|
||||
'publication_evidence': 'Published proceedings and accepted program; not an original decision note.'}}
|
||||
|
||||
|
||||
class Pages:
|
||||
def __init__(self, *, cache=None, snapshots=None):
|
||||
if bool(cache) == bool(snapshots):
|
||||
raise ValueError('Choose exactly one cache or frozen snapshot directory')
|
||||
self.cache = Path(cache) if cache else None
|
||||
self.root = Path(snapshots).resolve() if snapshots else None
|
||||
self.receipts = json.loads((self.root/'receipts.json').read_bytes())['receipts'] if self.root else []
|
||||
self.evidence = {}
|
||||
self.data = {}
|
||||
|
||||
def read(self, url):
|
||||
if self.root:
|
||||
receipt = one([r for r in self.receipts if r['url'] == url], 'exact frozen response receipt')
|
||||
filename = Path(receipt['file'])
|
||||
path = self.root/filename
|
||||
if (filename.is_absolute() or '..' in filename.parts or not path.resolve().is_relative_to(self.root)
|
||||
or receipt['http_status'] != 200 or receipt['content_type'] != 'text/html'
|
||||
or receipt.get('final_url', url) != url):
|
||||
raise ValueError('Unsafe or unsuccessful frozen response')
|
||||
data = path.read_bytes()
|
||||
observed = receipt['at']
|
||||
else:
|
||||
data = fetch(url, self.cache, 2*1024**2, delay=1)
|
||||
receipt = json.loads((self.cache/hashlib.sha256(url.encode()).hexdigest()).with_suffix('.json').read_bytes())
|
||||
observed = receipt['retrieved_at']
|
||||
if (len(data) > 2*1024**2 or len(data) != receipt['bytes']
|
||||
or hashlib.sha256(data).hexdigest() != receipt['sha256'] or receipt['url'] != url):
|
||||
raise ValueError('Response checksum, URL or size differs')
|
||||
evidence = {'url': url, 'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data),
|
||||
'file': 'evidence/'+hashlib.sha256(data).hexdigest()+'.html',
|
||||
'observed_at': timestamp(observed), 'http_status': 200, 'response_receipt': receipt}
|
||||
self.evidence[url] = evidence
|
||||
self.data[url] = data
|
||||
return data.decode('utf-8'), evidence
|
||||
|
||||
|
||||
def collect(year, out, *, cache=None, snapshots=None):
|
||||
if year != 2025:
|
||||
raise ValueError('Only the independently reviewed RSS 2025 edition is supported')
|
||||
out = Path(out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
with (out/'rss.collect.lock').open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
report_path = out/'rss-2025.json'
|
||||
if report_path.exists() or list(out.glob('rss-2025-*.jsonl')):
|
||||
raise FileExistsError('Inspect existing output; never overwrite a completed or partial queue')
|
||||
pages = Pages(cache=cache, snapshots=snapshots)
|
||||
context = {key: pages.read(url) for key, url in PAGES.items()}
|
||||
index = parse_index(context['index'][0])
|
||||
accepted = parse_program(context['accepted'][0], 'Accepted Papers', ('ID', 'Session', 'Title', 'Authors'))
|
||||
demos = parse_program(context['demos'][0], 'Demos', ('ID', 'Location', 'Title', 'Day', 'Time'))
|
||||
validate_program(index, accepted, demos, context['authorinfo'][0])
|
||||
records = []
|
||||
for pid, entry in sorted(index.items()):
|
||||
html, evidence = pages.read(entry['url'])
|
||||
sources = {key: item[1] for key, item in context.items()}
|
||||
sources['paper'] = evidence
|
||||
records.append(parse_paper(html, pid, entry, accepted[pid], demos.get(pid), sources))
|
||||
counts = Counter(r['track'] for r in records)
|
||||
if counts != {'demo': len(DEMO_IDS), 'proceedings_unspecified': EXPECTED-len(DEMO_IDS)}:
|
||||
raise ValueError('Track census differs; review official source changes')
|
||||
files = []
|
||||
with tempfile.TemporaryDirectory(prefix='.rss-', dir=out) as temporary:
|
||||
stage = Path(temporary)
|
||||
(stage/'evidence').mkdir()
|
||||
for url, data in pages.data.items():
|
||||
(stage/pages.evidence[url]['file']).write_bytes(data)
|
||||
for track in sorted(counts):
|
||||
group = [r for r in records if r['track'] == track]
|
||||
path = stage/f'rss-2025-{track}.jsonl'
|
||||
path.write_text(''.join(json.dumps(r, ensure_ascii=False)+'\n' for r in group))
|
||||
manifest = {'source': SOURCE, 'parser_version': 1, 'venue': 'RSS', 'year': 2025,
|
||||
'track': track, 'count': len(group), 'file': str(out/path.name),
|
||||
'file_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
'list_url': ROOT, 'status': 'partial',
|
||||
'missing_abstract_count': sum(r['abstract'] is None for r in group)}
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps(manifest, ensure_ascii=False, indent=2)+'\n')
|
||||
files.append(manifest)
|
||||
result = {'source': SOURCE, 'year': 2025, 'records': len(records), 'files': files,
|
||||
'source_evidence': list(pages.evidence.values()), 'database_written': False,
|
||||
'pdfs_downloaded': False, 'status': 'collected_pending_identity_review',
|
||||
'note': 'Only the published proceedings. Demo IDs have explicit program evidence; '
|
||||
'other individual submission types remain unknown. No absent-based negative decisions. '
|
||||
'Separate source identity review and independent production acceptance are required.'}
|
||||
(stage/report_path.name).write_text(json.dumps(result, ensure_ascii=False, indent=2)+'\n')
|
||||
(out/'evidence').mkdir(exist_ok=True)
|
||||
for path in (stage/'evidence').iterdir():
|
||||
path.replace(out/'evidence'/path.name)
|
||||
for path in stage.iterdir():
|
||||
if path.is_file() and path.name != report_path.name:
|
||||
path.replace(out/path.name)
|
||||
(stage/report_path.name).replace(report_path)
|
||||
return result
|
||||
|
||||
|
||||
def main(argv=None, *, prog=None):
|
||||
parser = argparse.ArgumentParser(prog=prog)
|
||||
parser.add_argument('--year', type=int, choices=[2025], required=True)
|
||||
parser.add_argument('--out', required=True)
|
||||
group = parser.add_mutually_exclusive_group(required=True)
|
||||
group.add_argument('--cache')
|
||||
group.add_argument('--snapshots')
|
||||
args = parser.parse_args(argv)
|
||||
print(json.dumps(collect(args.year, args.out, cache=args.cache, snapshots=args.snapshots), ensure_ascii=False))
|
||||
@@ -4,7 +4,7 @@ import re
|
||||
from paper_atlas.io.publication_ids import mlsys_id
|
||||
|
||||
|
||||
PUBLICATIONS={'pmlr','acl_anthology','colm','neurips','mlsys','cvf','ecva','dblp'}
|
||||
PUBLICATIONS={'pmlr','acl_anthology','colm','neurips','mlsys','cvf','ecva','dblp','aamas_proceedings'}
|
||||
|
||||
|
||||
def timestamp(value=None):
|
||||
|
||||
+314
@@ -0,0 +1,314 @@
|
||||
{
|
||||
"source_snapshot_sha256": "0376ad019b216553faff0a14e1757b2d4c7a3b84c1beacf0511ef83695211b18",
|
||||
"scope": "Pinned proceedings metadata, not authentic example paper records or evidence that all volumes contain indexed papers. Test papers are synthetic.",
|
||||
"volumes": [
|
||||
{
|
||||
"key": "conf/ecml/1987",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Ivan Bratko",
|
||||
"Nada Lavrac"
|
||||
],
|
||||
"title": [
|
||||
"Progress in Machine Learning - Proceedings of EWSL 87: 2nd European Working Session on Learning, Bled, Yugoslavia, May 1987"
|
||||
],
|
||||
"booktitle": [
|
||||
"EWSL"
|
||||
],
|
||||
"publisher": [
|
||||
"Sigma Press, Wilmslow"
|
||||
],
|
||||
"year": [
|
||||
"1987"
|
||||
],
|
||||
"isbn": [
|
||||
"1-85058-088-X"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ecml/ewsl87.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1987",
|
||||
"booktitle": "EWSL",
|
||||
"record_type": "inproceedings",
|
||||
"count": 16
|
||||
}
|
||||
],
|
||||
"expected_original_series": "EWSL",
|
||||
"expected_target_family": "ECML-PKDD"
|
||||
},
|
||||
{
|
||||
"key": "conf/ecml/1988",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Derek H. Sleeman"
|
||||
],
|
||||
"title": [
|
||||
"Proceedings of the Third European Working Session on Learning, EWSL 1988, Turing Institute, Glasgow, UK, October 3-5, 1988"
|
||||
],
|
||||
"booktitle": [
|
||||
"EWSL"
|
||||
],
|
||||
"publisher": [
|
||||
"Pitman Publishing"
|
||||
],
|
||||
"year": [
|
||||
"1988"
|
||||
],
|
||||
"isbn": [
|
||||
"0-273-08800-9"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ecml/ewsl88.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1988",
|
||||
"booktitle": "EWSL",
|
||||
"record_type": "inproceedings",
|
||||
"count": 21
|
||||
}
|
||||
],
|
||||
"expected_original_series": "EWSL",
|
||||
"expected_target_family": "ECML-PKDD"
|
||||
},
|
||||
{
|
||||
"key": "conf/ecml/1989",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Katharina Morik"
|
||||
],
|
||||
"title": [
|
||||
"Proceedings of the 4th European Working Session on Learning, Montpellier, France, 4-6 December 1989"
|
||||
],
|
||||
"booktitle": [
|
||||
"EWSL"
|
||||
],
|
||||
"publisher": [
|
||||
"Pitman and Morgan Kaufman Publishers"
|
||||
],
|
||||
"year": [
|
||||
"1989"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [],
|
||||
"expected_original_series": "EWSL",
|
||||
"expected_target_family": "ECML-PKDD"
|
||||
},
|
||||
{
|
||||
"key": "conf/ecml/1991",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Yves Kodratoff"
|
||||
],
|
||||
"title": [
|
||||
"Machine Learning - EWSL-91, European Working Session on Learning, Porto, Portugal, March 6-8, 1991, Proceedings"
|
||||
],
|
||||
"booktitle": [
|
||||
"EWSL"
|
||||
],
|
||||
"series": [
|
||||
"Lecture Notes in Computer Science"
|
||||
],
|
||||
"volume": [
|
||||
"482"
|
||||
],
|
||||
"publisher": [
|
||||
"Springer"
|
||||
],
|
||||
"year": [
|
||||
"1991"
|
||||
],
|
||||
"isbn": [
|
||||
"3-540-53816-X"
|
||||
],
|
||||
"ee": [
|
||||
"https://doi.org/10.1007/BFb0016999"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ecml/ewsl91.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1991",
|
||||
"booktitle": "EWSL",
|
||||
"record_type": "inproceedings",
|
||||
"count": 44
|
||||
}
|
||||
],
|
||||
"expected_original_series": "EWSL",
|
||||
"expected_target_family": "ECML-PKDD"
|
||||
},
|
||||
{
|
||||
"key": "conf/ecp/1997",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Sam Steel",
|
||||
"Rachid Alami 0001"
|
||||
],
|
||||
"title": [
|
||||
"Recent Advances in AI Planning, 4th European Conference on Planning, ECP'97, Toulouse, France, September 24-26, 1997, Proceedings"
|
||||
],
|
||||
"booktitle": [
|
||||
"ECP"
|
||||
],
|
||||
"series": [
|
||||
"Lecture Notes in Computer Science"
|
||||
],
|
||||
"volume": [
|
||||
"1348"
|
||||
],
|
||||
"publisher": [
|
||||
"Springer"
|
||||
],
|
||||
"year": [
|
||||
"1997"
|
||||
],
|
||||
"isbn": [
|
||||
"3-540-63912-8"
|
||||
],
|
||||
"ee": [
|
||||
"https://doi.org/10.1007/3-540-63912-8"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ecp/ecp1997.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1997",
|
||||
"booktitle": "ECP",
|
||||
"record_type": "inproceedings",
|
||||
"count": 35
|
||||
}
|
||||
],
|
||||
"expected_original_series": "ECP",
|
||||
"expected_target_family": "ICAPS"
|
||||
},
|
||||
{
|
||||
"key": "conf/ecp/1999",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Susanne Biundo",
|
||||
"Maria Fox 0001"
|
||||
],
|
||||
"title": [
|
||||
"Recent Advances in AI Planning, 5th European Conference on Planning, ECP'99, Durham, UK, September 8-10, 1999, Proceedings"
|
||||
],
|
||||
"booktitle": [
|
||||
"ECP"
|
||||
],
|
||||
"series": [
|
||||
"Lecture Notes in Computer Science"
|
||||
],
|
||||
"volume": [
|
||||
"1809"
|
||||
],
|
||||
"publisher": [
|
||||
"Springer"
|
||||
],
|
||||
"year": [
|
||||
"2000"
|
||||
],
|
||||
"isbn": [
|
||||
"3-540-67866-2"
|
||||
],
|
||||
"ee": [
|
||||
"https://doi.org/10.1007/10720246"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ecp/ecp1999.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1999",
|
||||
"booktitle": "ECP",
|
||||
"record_type": "inproceedings",
|
||||
"count": 28
|
||||
}
|
||||
],
|
||||
"expected_original_series": "ECP",
|
||||
"expected_target_family": "ICAPS"
|
||||
},
|
||||
{
|
||||
"key": "conf/ewsp/1991",
|
||||
"fields": {
|
||||
"editor": [
|
||||
"Joachim Hertzberg"
|
||||
],
|
||||
"title": [
|
||||
"European Workshop on Planning, EWSP '91, Sankt Augustin, Germany, March 18-19, 1991, Proceedings"
|
||||
],
|
||||
"booktitle": [
|
||||
"EWSP"
|
||||
],
|
||||
"series": [
|
||||
"Lecture Notes in Computer Science"
|
||||
],
|
||||
"volume": [
|
||||
"522"
|
||||
],
|
||||
"publisher": [
|
||||
"Springer"
|
||||
],
|
||||
"year": [
|
||||
"1991"
|
||||
],
|
||||
"isbn": [
|
||||
"3-540-54364-3"
|
||||
],
|
||||
"ee": [
|
||||
"https://doi.org/10.1007/BFb0052944"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/ewsp/ewsp1991.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1991",
|
||||
"booktitle": "EWSP",
|
||||
"record_type": "inproceedings",
|
||||
"count": 10
|
||||
}
|
||||
],
|
||||
"expected_original_series": "EWSP",
|
||||
"expected_target_family": "ICAPS"
|
||||
},
|
||||
{
|
||||
"key": "conf/aips/1990",
|
||||
"fields": {
|
||||
"title": [
|
||||
"Proceedings of the First International Conference on Expert Planning Systems, Brighton, UK, June 27-29, 1990"
|
||||
],
|
||||
"booktitle": [
|
||||
"Expert Planning Systems"
|
||||
],
|
||||
"publisher": [
|
||||
"AAAI"
|
||||
],
|
||||
"year": [
|
||||
"1990"
|
||||
],
|
||||
"url": [
|
||||
"db/conf/aips/aips1990.html"
|
||||
]
|
||||
},
|
||||
"bibliographic_counts": [
|
||||
{
|
||||
"year": "1990",
|
||||
"booktitle": "Expert Planning Systems",
|
||||
"record_type": "inproceedings",
|
||||
"count": 48
|
||||
}
|
||||
],
|
||||
"expected_original_series": "EPS",
|
||||
"expected_target_family": "ICAPS"
|
||||
}
|
||||
]
|
||||
}
|
||||
Vendored
+4356
File diff suppressed because it is too large
Load Diff
+1501
File diff suppressed because it is too large
Load Diff
Vendored
+1646
File diff suppressed because it is too large
Load Diff
Vendored
+1585
File diff suppressed because it is too large
Load Diff
Vendored
+148
@@ -0,0 +1,148 @@
|
||||
<html>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
||||
<head>
|
||||
<meta name="citation_title" content="Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation" />
|
||||
<meta name="citation_author" content="Pascal Roth" />
|
||||
<meta name="citation_author" content="Jonas Frey" />
|
||||
<meta name="citation_author" content="Cesar Cadena" />
|
||||
<meta name="citation_author" content="Marco Hutter" />
|
||||
<meta name="citation_publication_date" content="2025/06/21" />
|
||||
<meta name="citation_conference_title" content="Robotics: Science and Systems XXI" />
|
||||
<meta name="citation_isbn" content="979-8-9902848-1-4" />
|
||||
<meta name="citation_volume" content="21" />
|
||||
<meta name="citation_pdf_url" content="http://www.roboticsproceedings.org/rss21/p001.pdf" />
|
||||
<title>Robotics: Science and Systems XXI - Online Proceedings</title>
|
||||
<link type="text/css" rel="stylesheet" href="../style.css">
|
||||
<!--[if IE 5]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie5.css">
|
||||
<![endif]-->
|
||||
<!--[if IE 6]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie6.css">
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div class="menu">
|
||||
<ul>
|
||||
<li class="here"><a class="menu" href="../index.html">Home</a></li>
|
||||
<li class="label">RSS XXII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XXI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/authors.html">Authors</a></li>
|
||||
<li class="label">RSS X</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/authors.html">Authors</a></li>
|
||||
<li class="label">RSS V</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/authors.html">Authors</a></li>
|
||||
<li class="label">RSS III</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/authors.html">Authors</a></li>
|
||||
<li class="label">RSS II</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/authors.html">Authors</a></li>
|
||||
<li class="label">RSS I</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/authors.html">Authors</a></li>
|
||||
<li class="label">Ethics and Malpractice</span></li>
|
||||
<li class="menu"><a class="menu" href="../statement.html">Statement</a></li>
|
||||
<li class="label">Contact</span></li>
|
||||
<li class="last"><a class="menu" href="../contact.html">Webmaster</a></li>
|
||||
<li class="no-style"><center><img border="0" src="../robot-small.jpg"></center>
|
||||
<br><br><br>
|
||||
<center>We are using google analytics for statistics!</center>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="content">
|
||||
|
||||
<h1>Robotics: Science and Systems XXI</h1>
|
||||
<h3>Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation</h3>
|
||||
<i>Pascal Roth, Jonas Frey, Cesar Cadena, Marco Hutter</i><br>
|
||||
<p>
|
||||
<b>Abstract:</b>
|
||||
</p><p style="text-align: justify;">
|
||||
Ensuring safe navigation in complex environments requires accurate real-time traversability assessment and understanding of environmental interactions relative to the robot's capabilities. Traditional methods, which assume simplified dynamics, often require designing and tuning cost functions to safely guide paths or actions toward the goal. This process is tedious, environment-dependent, and not generalizable. To overcome these issues, we propose a novel learned perceptive Forward Dynamics Model (FDM) that predicts the robot's future state conditioned on the surrounding geometry and history of proprioceptive measurements, proposing a more scalable, safer, and heuristic-free solution. The FDM is trained on multiple years of simulated navigation experience, including high-risk maneuvers, and real-world interactions to incorporate the full system dynamics beyond rigid body simulation. We integrate our perceptive FDM into a zero-shot Model Predictive Path Integral (MPPI) planning framework, leveraging the learned mapping between actions, future states, and failure probability. This allows for optimizing a simplified cost function, eliminating the need for extensive cost-tuning to ensure safety. On the legged robot ANYmal, the proposed perceptive FDM improves the position estimation by on average 41% over competitive baselines, which translates into a 27% higher navigation success rate in rough simulation environments. Moreover, we demonstrate effective sim-to-real transfer and showcase the benefit of training on synthetic and real data. Code and models are made publicly available under https://github.com/leggedrobotics/fdm.
|
||||
</p>
|
||||
<p>
|
||||
<b>Download:</b> <a href="p001.pdf" target="_blank" onclick="_gaq.push(['_trackEvent','rss10/p001.pdf','PDF',this.href]);"><img src="../icon-pdf.png" border=0></a>
|
||||
</p>
|
||||
<p>
|
||||
<b>Bibtex:</b>
|
||||
<pre>
|
||||
@INPROCEEDINGS{RothP-RSS-25,
|
||||
AUTHOR = {Pascal Roth AND Jonas Frey AND Cesar Cadena AND Marco Hutter},
|
||||
TITLE = {{Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation}},
|
||||
BOOKTITLE = {Proceedings of Robotics: Science and Systems},
|
||||
YEAR = {2025},
|
||||
ADDRESS = {LosAngeles, CA, USA},
|
||||
MONTH = {June},
|
||||
DOI = {10.15607/RSS.2025.XXI.001}
|
||||
}
|
||||
|
||||
</pre>
|
||||
</p>
|
||||
|
||||
</div>
|
||||
<div class="analyt">
|
||||
<script type="text/javascript">
|
||||
|
||||
var _gaq = _gaq || [];
|
||||
_gaq.push(['_setAccount', 'UA-24631810-1']);
|
||||
_gaq.push(['_trackPageview']);
|
||||
|
||||
(function() {
|
||||
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
|
||||
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
||||
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
|
||||
})();
|
||||
|
||||
</script>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Vendored
+155
@@ -0,0 +1,155 @@
|
||||
<html>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
||||
<head>
|
||||
<meta name="citation_title" content="Demonstrating ViSafe: Vision-enabled Safety for High-speed Detect and Avoid" />
|
||||
<meta name="citation_author" content="Parv Kapoor" />
|
||||
<meta name="citation_author" content="Ian Higgins" />
|
||||
<meta name="citation_author" content="Nikhil Varma Keetha" />
|
||||
<meta name="citation_author" content="Jay Patrikar" />
|
||||
<meta name="citation_author" content="Brady Moon" />
|
||||
<meta name="citation_author" content="Zelin Ye" />
|
||||
<meta name="citation_author" content="Yao He" />
|
||||
<meta name="citation_author" content="Ivan Cisneros" />
|
||||
<meta name="citation_author" content="Changliu Liu" />
|
||||
<meta name="citation_author" content="Eunsuk Kang" />
|
||||
<meta name="citation_author" content="Sebastian Scherer" />
|
||||
<meta name="citation_publication_date" content="2025/06/21" />
|
||||
<meta name="citation_conference_title" content="Robotics: Science and Systems XXI" />
|
||||
<meta name="citation_isbn" content="979-8-9902848-1-4" />
|
||||
<meta name="citation_volume" content="21" />
|
||||
<meta name="citation_pdf_url" content="http://www.roboticsproceedings.org/rss21/p002.pdf" />
|
||||
<title>Robotics: Science and Systems XXI - Online Proceedings</title>
|
||||
<link type="text/css" rel="stylesheet" href="../style.css">
|
||||
<!--[if IE 5]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie5.css">
|
||||
<![endif]-->
|
||||
<!--[if IE 6]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie6.css">
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div class="menu">
|
||||
<ul>
|
||||
<li class="here"><a class="menu" href="../index.html">Home</a></li>
|
||||
<li class="label">RSS XXII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XXI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/authors.html">Authors</a></li>
|
||||
<li class="label">RSS X</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/authors.html">Authors</a></li>
|
||||
<li class="label">RSS V</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/authors.html">Authors</a></li>
|
||||
<li class="label">RSS III</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/authors.html">Authors</a></li>
|
||||
<li class="label">RSS II</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/authors.html">Authors</a></li>
|
||||
<li class="label">RSS I</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/authors.html">Authors</a></li>
|
||||
<li class="label">Ethics and Malpractice</span></li>
|
||||
<li class="menu"><a class="menu" href="../statement.html">Statement</a></li>
|
||||
<li class="label">Contact</span></li>
|
||||
<li class="last"><a class="menu" href="../contact.html">Webmaster</a></li>
|
||||
<li class="no-style"><center><img border="0" src="../robot-small.jpg"></center>
|
||||
<br><br><br>
|
||||
<center>We are using google analytics for statistics!</center>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="content">
|
||||
|
||||
<h1>Robotics: Science and Systems XXI</h1>
|
||||
<h3>Demonstrating ViSafe: Vision-enabled Safety for High-speed Detect and Avoid</h3>
|
||||
<i>Parv Kapoor, Ian Higgins, Nikhil Varma Keetha, Jay Patrikar, Brady Moon, Zelin Ye, Yao He, Ivan Cisneros, Changliu Liu, Eunsuk Kang, Sebastian Scherer</i><br>
|
||||
<p>
|
||||
<b>Abstract:</b>
|
||||
</p><p style="text-align: justify;">
|
||||
Assured safe-separation is essential for achieving seamless high-density operation of airborne vehicles in a shared airspace. To equip resource-constrained aerial systems with this safety-critical capability, we present ViSafe, a high-speed vision-only airborne collision avoidance system. ViSafe offers a full-stack solution to the Detect and Avoid (DAA) problem by tightly integrating a learning-based edge-AI framework with a custom multi-camera hardware prototype designed under SWaP-C constraints. By leveraging perceptual input-focused control barrier functions (CBF) to design, encode, and enforce safety thresholds, ViSafe can provide provably safe runtime guarantees for self-separation in high-speed aerial operations. We evaluate ViSafe’s performance through an extensive test campaign involving both simulated digital twins and real-world flight scenarios. By independently varying agent types, closure rates, interaction geometries, and environmental conditions (e.g., weather and lighting), we demonstrate that ViSafe consistently ensures self-separation across diverse scenarios. In first-of-its-kind real-world high-speed collision avoidance tests with closure rates reaching 144 km/h, ViSafe sets a new benchmark for vision-only autonomous collision avoidance, establishing a new standard for safety in high-speed aerial navigation.
|
||||
</p>
|
||||
<p>
|
||||
<b>Download:</b> <a href="p002.pdf" target="_blank" onclick="_gaq.push(['_trackEvent','rss10/p002.pdf','PDF',this.href]);"><img src="../icon-pdf.png" border=0></a>
|
||||
</p>
|
||||
<p>
|
||||
<b>Bibtex:</b>
|
||||
<pre>
|
||||
@INPROCEEDINGS{KapoorP-RSS-25,
|
||||
AUTHOR = {Parv Kapoor AND Ian Higgins AND Nikhil Varma Keetha AND Jay Patrikar AND Brady Moon AND Zelin Ye AND Yao He AND Ivan Cisneros AND Changliu Liu AND Eunsuk Kang AND Sebastian Scherer},
|
||||
TITLE = {{Demonstrating ViSafe: Vision-enabled Safety for High-speed Detect and Avoid}},
|
||||
BOOKTITLE = {Proceedings of Robotics: Science and Systems},
|
||||
YEAR = {2025},
|
||||
ADDRESS = {LosAngeles, CA, USA},
|
||||
MONTH = {June},
|
||||
DOI = {10.15607/RSS.2025.XXI.002}
|
||||
}
|
||||
|
||||
</pre>
|
||||
</p>
|
||||
|
||||
</div>
|
||||
<div class="analyt">
|
||||
<script type="text/javascript">
|
||||
|
||||
var _gaq = _gaq || [];
|
||||
_gaq.push(['_setAccount', 'UA-24631810-1']);
|
||||
_gaq.push(['_trackPageview']);
|
||||
|
||||
(function() {
|
||||
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
|
||||
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
||||
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
|
||||
})();
|
||||
|
||||
</script>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Vendored
+65
@@ -0,0 +1,65 @@
|
||||
{
|
||||
"scope": "Six unmodified official HTML fixtures; exact observations from full167 server freeze. No research PDFs.",
|
||||
"receipts": [
|
||||
{
|
||||
"file": "index.html",
|
||||
"url": "https://www.roboticsproceedings.org/rss21/",
|
||||
"final_url": "https://www.roboticsproceedings.org/rss21/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 77753,
|
||||
"sha256": "5d8e530b5ce7184795c14a7dca615f9c1a8b0b189661e0e6e795167b68b8516d",
|
||||
"at": "2026-11-09T09:53:09.121733+00:00"
|
||||
},
|
||||
{
|
||||
"file": "accepted.html",
|
||||
"url": "https://roboticsconference.org/2025/program/papers/",
|
||||
"final_url": "https://roboticsconference.org/2025/program/papers/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 197708,
|
||||
"sha256": "fa1a5bce496a85eca8cec79aed8dbc148cda791fe6859375ba0c294983a18683",
|
||||
"at": "2026-11-09T09:53:10.171450+00:00"
|
||||
},
|
||||
{
|
||||
"file": "demos.html",
|
||||
"url": "https://roboticsconference.org/2025/program/demos/",
|
||||
"final_url": "https://roboticsconference.org/2025/program/demos/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 43542,
|
||||
"sha256": "0b154ed110f6d49ebe2ba87f2c9c498df5fc1ea00bfe508b548522dfe4779698",
|
||||
"at": "2026-11-09T09:53:11.145288+00:00"
|
||||
},
|
||||
{
|
||||
"file": "authorinfo.html",
|
||||
"url": "https://roboticsconference.org/2025/information/authorinfo/",
|
||||
"final_url": "https://roboticsconference.org/2025/information/authorinfo/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 54508,
|
||||
"sha256": "24d023daf9a074434c6e004001b863dfd260330e523671d614fa120b052927ab",
|
||||
"at": "2026-11-09T09:53:12.170246+00:00"
|
||||
},
|
||||
{
|
||||
"file": "p001.html",
|
||||
"url": "https://www.roboticsproceedings.org/rss21/p001.html",
|
||||
"final_url": "https://www.roboticsproceedings.org/rss21/p001.html",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 9048,
|
||||
"sha256": "973172348096a5cbf3e311bdc1ce33b27447761fa1058a050e611aa17547d821",
|
||||
"at": "2026-11-09T09:53:13.437943+00:00"
|
||||
},
|
||||
{
|
||||
"file": "p002.html",
|
||||
"url": "https://www.roboticsproceedings.org/rss21/p002.html",
|
||||
"final_url": "https://www.roboticsproceedings.org/rss21/p002.html",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 9335,
|
||||
"sha256": "30b5b68e7ed6a809ca51282c5efd1416513417ba91eba2845f2e73ed18b90ba4",
|
||||
"at": "2026-11-09T09:53:14.697857+00:00"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Cross-source publisher identities must preserve attempts and decision history."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from paper_atlas.db.submissions import identities_for_row
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
|
||||
DOI = '10.1234/aamas.identity.example'
|
||||
|
||||
|
||||
def row(source, external_id, *, venue='AAMAS', year=2012, track='demo', status='accepted', identifiers=None):
|
||||
return {'source': source, 'external_id': external_id,
|
||||
'url': 'https://example.org/'+source+'/'+external_id,
|
||||
'title': 'A shared publication with independently recorded evidence',
|
||||
'abstract': 'An independently recorded source abstract.',
|
||||
'authors': ['Jane Doe', 'Bob Smith'],
|
||||
'identifiers': identifiers if identifiers is not None else {'doi': DOI},
|
||||
'venue': venue, 'year': year, 'track': track, 'status': status,
|
||||
'observed_at': '2026-11-09T00:00:00Z',
|
||||
'raw_decision': source+' observed '+status,
|
||||
'metadata': {'original_source': source}, 'tags': []}
|
||||
|
||||
|
||||
def ingest(db, root, record):
|
||||
path = root/(record['source']+'-'+record['external_id'].replace('/', '-')+'.jsonl')
|
||||
path.write_text(json.dumps(record)+'\n')
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps({
|
||||
'source': record['source'], 'parser_version': 1, 'venue': record['venue'],
|
||||
'year': record['year'], 'track': record['track'], 'count': 1,
|
||||
'list_url': 'https://example.org/'+record['source'], 'status': 'partial',
|
||||
'file_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
'missing_abstract_count': 0}))
|
||||
ingest_file(db, path)
|
||||
|
||||
|
||||
class AamasPublicationIdentityTests(unittest.TestCase):
|
||||
def test_only_accepted_known_publisher_rows_supply_publication_doi_identity(self):
|
||||
record = row('aamas_proceedings', '2012/D1_1', identifiers={'doi': 'https://doi.org/'+DOI.upper()})
|
||||
self.assertEqual(identities_for_row(record), [('publication_doi', DOI)])
|
||||
for status in ['rejected', 'desk_rejected', 'withdrawn', 'under_review', 'unknown']:
|
||||
with self.subTest(status=status):
|
||||
self.assertEqual(identities_for_row({**record, 'status': status}), [])
|
||||
self.assertEqual(identities_for_row({**record, 'source': 'unverified_source'}), [])
|
||||
|
||||
def test_same_edition_doi_joins_sources_in_both_orders_without_duplicate_attempts(self):
|
||||
publisher = row('aamas_proceedings', '2012/D1_1')
|
||||
bibliography = row('dblp', 'conf/atal/Example12')
|
||||
for order in [(publisher, bibliography), (bibliography, publisher)]:
|
||||
with self.subTest(first=order[0]['source']), tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
db = root/'sample.sqlite'
|
||||
for record in order:
|
||||
ingest(db, root, record)
|
||||
ingest(db, root, publisher)
|
||||
with sqlite3.connect(db) as connection:
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM papers').fetchone()[0], 1)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM source_records').fetchone()[0], 2)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM submissions').fetchone()[0], 1)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM submission_sources').fetchone()[0], 2)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM decision_sources').fetchone()[0], 2)
|
||||
self.assertEqual(connection.execute('SELECT scheme,value FROM submission_identifiers').fetchall(), [('publication_doi', DOI)])
|
||||
metadata = [json.loads(r[0]) for r in connection.execute('SELECT metadata_json FROM source_records')]
|
||||
self.assertEqual({r['original_source'] for r in metadata}, {'aamas_proceedings', 'dblp'})
|
||||
self.assertEqual(connection.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
|
||||
def test_cross_venue_rejection_and_distinct_tracks_remain_separate_attempts(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
db = root/'sample.sqlite'
|
||||
rejected = row('openreview_snapshot', 'original-forum', venue='AAAI', year=2011,
|
||||
track='main', status='rejected', identifiers={'arxiv': '1101.00001', 'openreview': 'original-forum'})
|
||||
accepted = row('aamas_proceedings', '2012/D1_1', identifiers={'arxiv': '1101.00001', 'doi': DOI})
|
||||
another_track = row('aamas_proceedings', '2012/F1_1', track='proceedings_unspecified', identifiers={'doi': DOI})
|
||||
for record in [rejected, accepted, another_track]:
|
||||
ingest(db, root, record)
|
||||
with sqlite3.connect(db) as connection:
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM papers').fetchone()[0], 1)
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM submissions').fetchone()[0], 3)
|
||||
history = connection.execute('''SELECT v.acronym,e.year,e.track,d.status FROM decisions d
|
||||
JOIN submissions s ON s.id=d.submission_id JOIN editions e ON e.id=s.edition_id
|
||||
JOIN venues v ON v.id=e.venue_id ORDER BY v.acronym,e.track''').fetchall()
|
||||
self.assertEqual(set(history), {('AAAI',2011,'main','rejected'),('AAMAS',2012,'demo','accepted'),('AAMAS',2012,'proceedings_unspecified','accepted')})
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM decision_sources').fetchone()[0], 3)
|
||||
self.assertEqual(connection.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Original series, exact edition boundaries, and complete candidate discovery."""
|
||||
from contextlib import closing
|
||||
from copy import deepcopy
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
|
||||
from paper_atlas.collectors.dblp import catalogue, XML_SHA
|
||||
from paper_atlas.collectors.dblp_records import collect, normalize_row
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
FIXTURE = json.loads((Path(__file__).parent/'fixtures/historical_predecessors.json').read_text())
|
||||
EXAMPLES = {row['key']: row for row in FIXTURE['volumes']}
|
||||
|
||||
|
||||
def example(key, *, bibliographic_year=None):
|
||||
fixed = EXAMPLES[key]
|
||||
family = key.split('/')[1]
|
||||
year = int(key.split('/')[-1])
|
||||
volume = {'record_type': 'proceedings', 'key': key, 'fields': deepcopy(fixed['fields'])}
|
||||
row = {'record_type': 'inproceedings', 'key': f'conf/{family}/SyntheticExample{year}', 'publtype': None,
|
||||
'fields': {'title': [f'Synthetic historical research example {family} {year}'],
|
||||
'author': ['Jane Doe 0001', 'John Smith'], 'booktitle': fixed['fields']['booktitle'],
|
||||
'year': [str(bibliographic_year or year)], 'crossref': [key],
|
||||
'ee': [f'https://doi.org/10.1234/synthetic.{family}.{year}']}}
|
||||
return row, volume
|
||||
|
||||
|
||||
class HistoricalPredecessorTests(unittest.TestCase):
|
||||
def normalize(self, key, **kwargs):
|
||||
row, volume = example(key, **kwargs)
|
||||
fixed = EXAMPLES[key]
|
||||
result, reason = normalize_row(row, {key: volume}, {fixed['expected_target_family']})
|
||||
self.assertIsNone(reason)
|
||||
self.assertIsNotNone(result)
|
||||
return result
|
||||
|
||||
def test_ewsl_primary_series_retains_original_name_and_family_alias(self):
|
||||
for year in [1987, 1988, 1989, 1991]:
|
||||
with self.subTest(year=year):
|
||||
record = self.normalize(f'conf/ecml/{year}')
|
||||
self.assertEqual((record['venue'],record['venue_aliases'],record['year'],record['track']), ('EWSL',['ECML-PKDD'],year,'main'))
|
||||
proof = record['metadata']['historical_predecessor_evidence']
|
||||
self.assertEqual(proof['original_series'], 'EWSL')
|
||||
self.assertEqual(proof['history_url'], 'https://ecmlpkdd2011.org/pastConferences.php')
|
||||
self.assertEqual(record['metadata']['bibliography']['booktitle'], ['EWSL'])
|
||||
self.assertEqual(record['metadata']['author_names_raw'], ['Jane Doe 0001','John Smith'])
|
||||
self.assertEqual(record['authors'], ['Jane Doe','John Smith'])
|
||||
self.assertEqual(EXAMPLES['conf/ecml/1989']['bibliographic_counts'], [])
|
||||
|
||||
def test_eps_ecp_ewsp_keep_original_series_instead_of_renaming_to_icaps(self):
|
||||
for key in ['conf/aips/1990','conf/ecp/1997','conf/ecp/1999','conf/ewsp/1991']:
|
||||
with self.subTest(key=key):
|
||||
record = self.normalize(key)
|
||||
fixed = EXAMPLES[key]
|
||||
self.assertEqual(record['venue'], fixed['expected_original_series'])
|
||||
self.assertEqual(record['venue_aliases'], ['ICAPS'])
|
||||
self.assertEqual(record['track'], 'main')
|
||||
proof = record['metadata']['historical_predecessor_evidence']
|
||||
self.assertEqual(proof['history_url'], 'https://www.icaps-conference.org/conference-series/')
|
||||
self.assertEqual(proof['proceedings_title'], fixed['fields']['title'][0])
|
||||
self.assertEqual(proof['proceedings_year_fields'], fixed['fields']['year'])
|
||||
|
||||
def test_altered_titles_keys_suffixes_publication_years_and_unreviewed_editions_stay_held(self):
|
||||
for key in ['conf/ecml/1988','conf/aips/1990','conf/ecp/1997','conf/ewsp/1991']:
|
||||
row, volume = example(key)
|
||||
for changed in ['title','key','suffix','publication_year']:
|
||||
with self.subTest(key=key, changed=changed):
|
||||
r, v = deepcopy(row), deepcopy(volume)
|
||||
if changed == 'title': v['fields']['title'][0] += ', Affiliated Workshop'
|
||||
elif changed == 'key': v['key'] += 'w'; r['fields']['crossref'] = [v['key']]
|
||||
elif changed == 'suffix': r['fields']['booktitle'][0] += ' (Workshop)'
|
||||
else: v['fields']['year'] = ['2001']
|
||||
result, _ = normalize_row(r, {v['key']: v}, {EXAMPLES[key]['expected_target_family']})
|
||||
self.assertIsNone(result)
|
||||
row, volume = example('conf/ecp/1997')
|
||||
row['fields']['year'] = ['2001']; row['fields']['crossref'] = ['conf/ecp/2001']
|
||||
volume['key'] = 'conf/ecp/2001'; volume['fields']['year'] = ['2001']
|
||||
self.assertIsNone(normalize_row(row, {volume['key']:volume}, {'ICAPS'})[0])
|
||||
|
||||
def test_ecp1999_event_and_bibliographic_years_are_independently_retained(self):
|
||||
record = self.normalize('conf/ecp/1999', bibliographic_year=2000)
|
||||
self.assertEqual(record['year'], 1999)
|
||||
self.assertEqual(record['metadata']['bibliographic_year'], 2000)
|
||||
self.assertEqual(record['metadata']['bibliography']['year'], ['2000'])
|
||||
self.assertEqual(record['metadata']['historical_predecessor_evidence']['proceedings_year_fields'], ['2000'])
|
||||
self.assertEqual(record['metadata']['historical_predecessor_evidence']['event_year'], 1999)
|
||||
|
||||
def test_old_candidates_require_new_families_only_when_requested_years_need_them(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary); row, volume = example('conf/aips/1990')
|
||||
db = root/'dblp-catalogue.sqlite'
|
||||
with closing(sqlite3.connect(db)) as connection, connection:
|
||||
connection.execute('CREATE TABLE proceedings(key TEXT PRIMARY KEY,metadata_json TEXT)')
|
||||
connection.execute('INSERT INTO proceedings VALUES(?,?)', (volume['key'],json.dumps(volume)))
|
||||
candidate = root/'candidate.jsonl'; candidate.write_text(json.dumps(row)+'\n')
|
||||
db.with_name('dblp-ai-candidates.manifest.json').write_text(json.dumps({
|
||||
'snapshot_sha256': XML_SHA, 'candidate_sha256': hashlib.sha256(candidate.read_bytes()).hexdigest(),
|
||||
'candidate_records': 1, 'candidate_family_counts': {'aips':1},
|
||||
'record_types': {'proceedings':1,'inproceedings':1}}))
|
||||
# Tiny CI fixtures may live on tmpfs; production reserve is tested separately.
|
||||
with patch('paper_atlas.collectors.dblp_records.shutil.disk_usage', return_value=SimpleNamespace(free=16*1024**3)):
|
||||
result = collect(db, root/'early', ['ICAPS'], candidates=candidate, through_year=1990)
|
||||
self.assertEqual((result['normalized_records'],result['files'][0]['venue']), (1,'EPS'))
|
||||
for year, missing in [(1991,'ICAPS/ewsp'),(1997,'ICAPS/ecp')]:
|
||||
out = root/str(year); out.mkdir(); sentinel=out/'previous.txt';sentinel.write_text('accepted previous output')
|
||||
with self.subTest(year=year), self.assertRaisesRegex(ValueError, missing):
|
||||
collect(db, out, ['ICAPS'], candidates=candidate, through_year=year)
|
||||
self.assertEqual(sentinel.read_text(), 'accepted previous output')
|
||||
self.assertFalse((out/'dblp-normalization.json').exists())
|
||||
|
||||
def test_new_catalogue_includes_previously_missing_families_without_decision_inference(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary);dtd=root/'dblp.dtd';dtd.write_bytes(b'<!ENTITY eacute "é">')
|
||||
archive=root/'dblp.xml.gz'
|
||||
body=''.join(f'<inproceedings key="conf/{family}/Synthetic"><title>Synthetic example</title><year>{year}</year><booktitle>{book}</booktitle><crossref>conf/{family}/{year}</crossref></inproceedings>' for family,year,book in [('ecp',1997,'ECP'),('ewsp',1991,'EWSP')])
|
||||
with gzip.open(archive,'wb') as stream:stream.write(('<?xml version="1.0"?><!DOCTYPE dblp SYSTEM "dblp.dtd"><dblp>'+body+'</dblp>').encode())
|
||||
result=catalogue(archive,dtd,root/'out',hashlib.sha256(archive.read_bytes()).hexdigest(),hashlib.md5(dtd.read_bytes()).hexdigest())
|
||||
rows=[json.loads(line)for line in Path(result['file']).read_text().splitlines()]
|
||||
self.assertEqual({r['key'] for r in rows}, {'conf/ecp/Synthetic','conf/ewsp/Synthetic'})
|
||||
self.assertEqual(result['candidate_records'], 2)
|
||||
self.assertTrue(all('status' not in r and 'venue' not in r for r in rows))
|
||||
|
||||
def test_native_import_keeps_original_series_aliases_and_full_source_evidence(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root=Path(temporary);db=root/'sample.sqlite'
|
||||
for key in ['conf/ecml/1988','conf/ecp/1999']:
|
||||
record=self.normalize(key);path=root/(key.replace('/','-')+'.jsonl')
|
||||
path.write_text(json.dumps(record,ensure_ascii=False)+'\n')
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps({'source':'dblp','venue':record['venue'],'year':record['year'],'track':'main','count':1,'list_url':'https://doi.org/10.4230/dblp.xml.2026-11-01','file_sha256':hashlib.sha256(path.read_bytes()).hexdigest(),'status':'partial','missing_abstract_count':1}))
|
||||
ingest_file(db,path)
|
||||
with closing(sqlite3.connect(db)) as connection:
|
||||
rows=connection.execute('SELECT v.acronym,a.alias,e.year FROM editions e JOIN venues v ON v.id=e.venue_id JOIN venue_aliases a ON a.venue_id=v.id ORDER BY v.acronym').fetchall()
|
||||
self.assertEqual(rows,[('ECP','ICAPS',1999),('EWSL','ECML-PKDD',1988)])
|
||||
metadata=[json.loads(r[0])for r in connection.execute('SELECT metadata_json FROM source_records')]
|
||||
self.assertEqual({r['historical_predecessor_evidence']['original_series'] for r in metadata}, {'ECP','EWSL'})
|
||||
self.assertEqual(connection.execute('SELECT count(*) FROM decisions WHERE status=?',('accepted',)).fetchone()[0],2)
|
||||
self.assertEqual(connection.execute('PRAGMA foreign_key_check').fetchall(),[])
|
||||
|
||||
def test_existing_aips_normalization_remains_original_aips(self):
|
||||
row, volume=example('conf/aips/1990');row['fields']['booktitle']=['AIPS'];row['fields']['year']=['1994'];row['fields']['crossref']=['conf/aips/1994']
|
||||
volume['key']='conf/aips/1994';volume['fields']['title']=['AIPS 1994 Proceedings'];volume['fields']['year']=['1994']
|
||||
record,reason=normalize_row(row,{volume['key']:volume},{'ICAPS'})
|
||||
self.assertIsNone(reason);self.assertEqual((record['venue'],record['venue_aliases']),('AIPS',['ICAPS']))
|
||||
self.assertNotIn('historical_predecessor_evidence',record['metadata'])
|
||||
@@ -0,0 +1,148 @@
|
||||
"""Historical accepted lists, repeated presentations and preserved work histories."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from paper_atlas.collectors import iclr_archive as archive
|
||||
from paper_atlas.db.store import Store
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
OBSERVED = '2026-11-09T00:00:00Z'
|
||||
|
||||
|
||||
def entry(identifier, title='A shared research paper', display='Jane Doe and Bob Smith'):
|
||||
return f'<li><div><a href="http://arxiv.org/abs/{identifier}">{title}</a>, {display}</div></li>'
|
||||
|
||||
|
||||
def page2015():
|
||||
return ('<h3>Main Conference - Oral Presentations</h3><ol>'+entry('1412.0001v2')+'</ol>'
|
||||
'<h3>Main Conference - Poster Presentations</h3><ol>'+entry('1412.0001v2')+'</ol>'
|
||||
'<h3>Workshop Papers</h3><ol>'+entry('1412.0002', 'A workshop paper')+'</ol>')
|
||||
|
||||
|
||||
class IclrArchiveTests(unittest.TestCase):
|
||||
def frozen(self, root, year, html):
|
||||
root.mkdir()
|
||||
data = html.encode()
|
||||
(root/'page.html').write_bytes(data)
|
||||
receipt = {'url': archive.PAGES[year][0], 'file': 'page.html', 'status': 200,
|
||||
'bytes': len(data), 'sha256': hashlib.sha256(data).hexdigest(), 'finished_at': OBSERVED}
|
||||
(root/'receipts.json').write_text(json.dumps({'receipts': [receipt]}))
|
||||
return root
|
||||
|
||||
def test_oral_poster_repeat_is_one_attempt_with_all_source_variants(self):
|
||||
rows = archive.parse_page(page2015(), 2015, {'observed_at': OBSERVED})
|
||||
main, workshop = rows
|
||||
self.assertEqual((main['track'], workshop['track']), ('main', 'workshop'))
|
||||
self.assertEqual(main['identifiers'], {'arxiv': '1412.0001'})
|
||||
self.assertEqual(main['external_id'], '2015/main/1412.0001')
|
||||
self.assertEqual(main['metadata']['presentations'], ['oral', 'poster'])
|
||||
self.assertEqual(len(main['metadata']['source_variants']), 2)
|
||||
self.assertEqual(main['metadata']['source_variants'][0]['version'], 'v2')
|
||||
self.assertEqual(main['authors'], ['Jane Doe', 'Bob Smith'])
|
||||
self.assertTrue(all(r['status'] == 'accepted' and r['abstract'] is None for r in rows))
|
||||
|
||||
def test_2014_author_span_outside_paragraph_and_trailing_separator(self):
|
||||
html = ('<h3>Conference Proceedings</h3><p>conference papers accepted</p>'
|
||||
'<div><b><a href="http://arxiv.org/abs/1312.0001">A <i>research</i> title</a></b></div>'
|
||||
'<p><span style="font-style: italic">Jane Doe; </span>'
|
||||
'<span style="font-style:italic">Bob Smith; Carol Wu;</span></p>'
|
||||
'<div><a href="http://arxiv.org/abs/1312.0002">Another title</a>'
|
||||
'<div><i><span>First Author; </span><span>Second Author</span></i></div></div>')
|
||||
row, unwrapped = archive.parse_page(html, 2014, {'observed_at': OBSERVED})
|
||||
self.assertEqual(row['title'], 'A research title')
|
||||
self.assertEqual(row['authors'], ['Jane Doe', 'Bob Smith', 'Carol Wu'])
|
||||
self.assertEqual(row['metadata']['source_variants'][0]['authors_display'], 'Jane Doe; Bob Smith; Carol Wu;')
|
||||
self.assertEqual(row['track'], 'main')
|
||||
self.assertEqual(unwrapped['authors'], ['First Author', 'Second Author'])
|
||||
|
||||
def test_same_arxiv_in_different_tracks_keeps_separate_source_attempts(self):
|
||||
html = page2015().replace('1412.0002', '1412.0001v3')
|
||||
rows = archive.parse_page(html, 2015, {'observed_at': OBSERVED})
|
||||
self.assertEqual(len(rows), 2)
|
||||
self.assertEqual({r['external_id'] for r in rows}, {'2015/main/1412.0001', '2015/workshop/1412.0001'})
|
||||
self.assertEqual(rows[1]['metadata']['source_variants'][0]['version'], 'v3')
|
||||
|
||||
def test_2016_code_links_and_breaks_do_not_become_authors(self):
|
||||
html = ('<h1>Accepted Papers (Conference Track)</h1><h3>Oral Presentations</h3><ol>'
|
||||
'<li><a href="http://arxiv.org/abs/1511.00001">A title</a>'
|
||||
'<a href="https://example.org/code">[code]</a><br>Jane Doe, Bob Smith</li></ol>'
|
||||
'<h3>Poster Presentations</h3>')
|
||||
row, = archive.parse_page(html, 2016, {'observed_at': OBSERVED})
|
||||
self.assertEqual(row['authors'], ['Jane Doe', 'Bob Smith'])
|
||||
self.assertEqual(row['metadata']['presentations'], ['oral'])
|
||||
|
||||
def test_conflicting_repeat_unknown_sections_and_unparsed_links_fail(self):
|
||||
variants = [page2015().replace('Poster Presentations', 'Other Presentations'),
|
||||
page2015().replace('</ol>', '</ol><a href="http://arxiv.org/abs/1412.0003">stray</a>', 1),
|
||||
page2015().replace('Jane Doe and Bob Smith', 'Different Author', 1)]
|
||||
for html in variants:
|
||||
with self.subTest(html=html), self.assertRaises(ValueError):
|
||||
archive.parse_page(html, 2015, {'observed_at': OBSERVED})
|
||||
|
||||
def test_arxiv_versions_are_preserved_and_unsafe_links_rejected(self):
|
||||
self.assertEqual(archive.arxiv_link('https://arxiv.org/pdf/1511.00001v3.pdf'), ('1511.00001', 'v3'))
|
||||
self.assertIsNone(archive.arxiv_link('https://arxiv.org.example.org/abs/1511.00001'))
|
||||
for url in ['https://arxiv.org/abs/1511.00001?other=1', 'ftp://arxiv.org/abs/1511.00001',
|
||||
'https://user@arxiv.org/abs/1511.00001', 'https://arxiv.org/abs/not-a-paper']:
|
||||
with self.subTest(url=url), self.assertRaises(ValueError):
|
||||
archive.arxiv_link(url)
|
||||
|
||||
def test_receipt_hash_status_path_and_symlink_cannot_escape_snapshot(self):
|
||||
for failure in ['hash', 'status', 'path', 'symlink']:
|
||||
with self.subTest(failure=failure), tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
source = self.frozen(root/'source', 2015, page2015())
|
||||
file = source/'receipts.json'
|
||||
data = json.loads(file.read_text())
|
||||
receipt = data['receipts'][0]
|
||||
if failure == 'hash': receipt['sha256'] = '0'*64
|
||||
elif failure == 'status': receipt['status'] = 404
|
||||
elif failure == 'path': receipt['file'] = '../outside.html'
|
||||
else:
|
||||
(root/'outside.html').write_text(page2015())
|
||||
(source/'page.html').unlink()
|
||||
(source/'page.html').symlink_to(root/'outside.html')
|
||||
file.write_text(json.dumps(data))
|
||||
with self.assertRaises(ValueError):
|
||||
archive.read_page(2015, snapshots=source)
|
||||
|
||||
def test_changed_inventory_leaves_no_completed_output(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
source = self.frozen(root/'source', 2015, page2015())
|
||||
with self.assertRaises(ValueError):
|
||||
archive.collect(2015, root/'out', snapshots=source)
|
||||
self.assertFalse((root/'out/iclr-archive-2015.json').exists())
|
||||
|
||||
def test_native_import_keeps_rejected_to_accepted_history_and_abstract(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
source = self.frozen(root/'source', 2015, page2015())
|
||||
with patch.dict(archive.PAGES, {2015: (archive.PAGES[2015][0], {'main': 1, 'workshop': 1})}):
|
||||
result = archive.collect(2015, root/'out', snapshots=source)
|
||||
with self.assertRaises(FileExistsError):
|
||||
archive.collect(2015, root/'out', snapshots=source)
|
||||
db = root/'sample.sqlite'
|
||||
store = Store(db)
|
||||
pid, rid = store.record('arxiv', '1412.0001', 'https://arxiv.org/abs/1412.0001',
|
||||
'A shared research paper', 'Keep this canonical abstract.',
|
||||
['Jane Doe', 'Bob Smith'], {'arxiv': '1412.0001'}, {})
|
||||
store.decision(pid, rid, 'Earlier Conference', 2014, 'main', 'original-attempt',
|
||||
'https://example.org/rejected', 'rejected', 'explicit earlier decision', observed_at=OBSERVED)
|
||||
store.conn.commit()
|
||||
store.conn.close()
|
||||
for manifest in result['files']:
|
||||
ingest_file(db, manifest['file'])
|
||||
ingest_file(db, manifest['file'])
|
||||
with sqlite3.connect(db) as c:
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0], 2)
|
||||
self.assertEqual(c.execute('SELECT abstract FROM papers WHERE id=?', (pid,)).fetchone()[0], 'Keep this canonical abstract.')
|
||||
history = c.execute('SELECT v.acronym,e.year,d.status FROM decisions d JOIN submissions s ON s.id=d.submission_id JOIN editions e ON e.id=s.edition_id JOIN venues v ON v.id=e.venue_id WHERE s.paper_id=? ORDER BY e.year', (pid,)).fetchall()
|
||||
self.assertEqual(history, [('Earlier Conference', 2014, 'rejected'), ('ICLR', 2015, 'accepted')])
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM submissions').fetchone()[0], 3)
|
||||
self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
@@ -0,0 +1,186 @@
|
||||
"""Frozen publisher examples, conflicting evidence and preserved attempt identity."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from paper_atlas.collectors import rss
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
FIXTURES = Path(__file__).parent/'fixtures/rss2025'
|
||||
OBSERVED = '2026-11-09T10:00:00Z'
|
||||
|
||||
|
||||
def fixture(name):
|
||||
return (FIXTURES/name).read_text()
|
||||
|
||||
|
||||
class RssProceedingsTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.index = rss.parse_index(fixture('index.html'))
|
||||
self.accepted = rss.parse_program(fixture('accepted.html'), 'Accepted Papers', ('ID', 'Session', 'Title', 'Authors'))
|
||||
self.demos = rss.parse_program(fixture('demos.html'), 'Demos', ('ID', 'Location', 'Title', 'Day', 'Time'))
|
||||
|
||||
def paper(self, pid=1, html=None, entry=None):
|
||||
return rss.parse_paper(html or fixture(f'p{pid:03d}.html'), pid, entry or self.index[pid],
|
||||
self.accepted[pid], self.demos.get(pid), {'paper': {'observed_at': OBSERVED}})
|
||||
|
||||
def test_frozen_official_census_and_all_fixture_hashes(self):
|
||||
receipts = json.loads((FIXTURES/'receipts.json').read_bytes())['receipts']
|
||||
self.assertEqual(len(receipts), 6)
|
||||
for r in receipts:
|
||||
data = (FIXTURES/r['file']).read_bytes()
|
||||
self.assertEqual((len(data), hashlib.sha256(data).hexdigest()), (r['bytes'], r['sha256']))
|
||||
self.assertEqual(len(self.index), 163)
|
||||
self.assertEqual(set(self.index), set(self.accepted))
|
||||
self.assertEqual(len(self.demos), 15)
|
||||
rss.validate_program(self.index, self.accepted, self.demos, fixture('authorinfo.html'))
|
||||
self.assertFalse(self.demos[93]['Title'].startswith('Demonstrating'))
|
||||
|
||||
def test_complete_authors_abstract_and_source_metadata_are_preserved(self):
|
||||
row = self.paper()
|
||||
self.assertEqual(row['authors'], ['Pascal Roth', 'Jonas Frey', 'Cesar Cadena', 'Marco Hutter'])
|
||||
self.assertTrue(row['abstract'].startswith('Ensuring safe navigation'))
|
||||
self.assertTrue(row['abstract'].endswith('https://github.com/leggedrobotics/fdm.'))
|
||||
self.assertNotIn('Download:', row['abstract'])
|
||||
self.assertEqual(row['identifiers'], {'doi': '10.15607/rss.2025.xxi.001'})
|
||||
self.assertEqual(row['metadata']['publication_date'], '2025/06/21')
|
||||
self.assertIn('AUTHOR', row['metadata']['original_bibtex'])
|
||||
self.assertEqual(row['status'], 'accepted')
|
||||
self.assertNotIn('decision_date', row)
|
||||
self.assertEqual(row['metadata']['decision_date_precision'], 'not_supplied')
|
||||
|
||||
def test_demo_evidence_is_independent_of_title_and_session_topic(self):
|
||||
row = self.paper(2)
|
||||
self.assertEqual(row['track'], 'demo')
|
||||
self.assertEqual(row['metadata']['demo_program_entry']['ID'], '2')
|
||||
self.assertEqual(self.paper()['track'], 'proceedings_unspecified')
|
||||
self.assertEqual({t['namespace'] for t in row['tags']}, {'rss_proceedings:session', 'rss_proceedings:program_role'})
|
||||
self.assertFalse(any('research_type' in t['namespace'] for t in row['tags']))
|
||||
html = fixture('p001.html').replace('Learned Perceptive Forward Dynamics Model', 'Demonstrating a Learned Model')
|
||||
entry = {**self.index[1], 'title': self.index[1]['title'].replace('Learned Perceptive Forward Dynamics Model', 'Demonstrating a Learned Model')}
|
||||
self.assertEqual(self.paper(html=html, entry=entry)['track'], 'proceedings_unspecified')
|
||||
|
||||
def test_missing_abstract_is_explicit_and_multiple_paragraphs_are_complete(self):
|
||||
html = fixture('p001.html')
|
||||
beginning = html.index('<p style="text-align: justify;">')
|
||||
end = html.index('</p>', beginning)
|
||||
empty = html[:beginning]+'<p style="text-align: justify;">'+html[end:]
|
||||
row = self.paper(html=empty)
|
||||
self.assertIsNone(row['abstract'])
|
||||
self.assertEqual(row['metadata']['missing_abstract_reason'], 'empty_official_abstract_section')
|
||||
multi = html.replace('</p> \n<p> \n<b>Download:', '</p><p>Second abstract paragraph.</p><p><b>Download:', 1)
|
||||
self.assertTrue(self.paper(html=multi)['abstract'].endswith('Second abstract paragraph.'))
|
||||
|
||||
def test_metadata_drift_author_truncation_and_wrong_paper_doi_fail(self):
|
||||
original = fixture('p001.html')
|
||||
for altered in [original.replace('XXI.001}', 'XXI.002}'),
|
||||
original.replace('content="2025/06/21"', 'content="2026/06/21"'),
|
||||
original.replace('<meta name="citation_author" content="Marco Hutter" />', ''),
|
||||
original.replace('<b>Abstract:</b>', '<b>Summary:</b>'),
|
||||
original.replace('rss21/p001.pdf', 'rss21/p002.pdf'),
|
||||
original.replace('citation_volume" content="21', 'citation_volume" content="22')]:
|
||||
with self.subTest(altered=altered[:60]), self.assertRaises(ValueError):
|
||||
self.paper(html=altered)
|
||||
|
||||
def test_duplicate_and_unparsed_index_entries_fail(self):
|
||||
for html in [fixture('index.html').replace('p002.html', 'p001.html'),
|
||||
fixture('index.html')+'<a href="p999.html">Stray paper</a>',
|
||||
fixture('index.html').replace( rss.HEADING, 'A different volume')]:
|
||||
with self.subTest(html=html[:40]), self.assertRaises(ValueError):
|
||||
rss.parse_index(html)
|
||||
|
||||
def test_program_title_id_category_and_header_changes_require_review(self):
|
||||
for change in ['title', 'id', 'category']:
|
||||
index = {k: dict(v) for k, v in self.index.items()}
|
||||
demos = {k: dict(v) for k, v in self.demos.items()}
|
||||
if change == 'title':
|
||||
demos[2]['Title'] = 'Another paper'
|
||||
elif change == 'id':
|
||||
del index[1]
|
||||
else:
|
||||
del demos[93]
|
||||
with self.subTest(change=change), self.assertRaises(ValueError):
|
||||
rss.validate_program(index, self.accepted, demos, fixture('authorinfo.html'))
|
||||
with self.assertRaises(ValueError):
|
||||
rss.parse_program(fixture('demos.html').replace('>Location<', '>Other<'), 'Demos', ('ID', 'Location', 'Title', 'Day', 'Time'))
|
||||
|
||||
def snapshot(self, root):
|
||||
root.mkdir()
|
||||
data = (FIXTURES/'p001.html').read_bytes()
|
||||
(root/'paper.html').write_bytes(data)
|
||||
receipt = {'file': 'paper.html', 'url': self.index[1]['url'], 'final_url': self.index[1]['url'],
|
||||
'http_status': 200, 'content_type': 'text/html', 'bytes': len(data),
|
||||
'sha256': hashlib.sha256(data).hexdigest(), 'at': OBSERVED}
|
||||
(root/'receipts.json').write_text(json.dumps({'receipts': [receipt]}))
|
||||
return receipt
|
||||
|
||||
def test_frozen_receipt_hash_status_redirect_and_path_guards(self):
|
||||
for failure in ['hash', 'status', 'redirect', 'path', 'duplicate', 'symlink']:
|
||||
with self.subTest(failure=failure), tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)/'source'
|
||||
r = self.snapshot(root)
|
||||
receipts = [r]
|
||||
if failure == 'hash': r['sha256'] = '0'*64
|
||||
elif failure == 'status': r['http_status'] = 404
|
||||
elif failure == 'redirect': r['final_url'] = 'https://example.org/p001.html'
|
||||
elif failure == 'path': r['file'] = '../outside.html'
|
||||
elif failure == 'duplicate': receipts.append(dict(r))
|
||||
else:
|
||||
(root.parent/'outside.html').write_bytes((root/'paper.html').read_bytes())
|
||||
(root/'paper.html').unlink()
|
||||
(root/'paper.html').symlink_to(root.parent/'outside.html')
|
||||
(root/'receipts.json').write_text(json.dumps({'receipts': receipts}))
|
||||
with self.assertRaises(ValueError): rss.Pages(snapshots=root).read(self.index[1]['url'])
|
||||
|
||||
def test_unreviewed_edition_and_existing_output_refuse_work(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with self.assertRaises(ValueError): rss.collect(2026, tmp, snapshots=FIXTURES)
|
||||
(Path(tmp)/'rss-2025.json').write_text('{}')
|
||||
with self.assertRaises(FileExistsError): rss.collect(2025, tmp, snapshots=FIXTURES)
|
||||
|
||||
def test_native_roundtrip_is_idempotent_with_same_doi_and_distinct_tracks(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
original = self.paper()
|
||||
dblp = {**original, 'source': 'dblp', 'external_id': 'conf/rss/example', 'metadata': {}, 'tags': []}
|
||||
variant = {**original, 'external_id': '2025/other-track', 'track': 'workshop'}
|
||||
for n, record in enumerate([dblp, original, variant, original]):
|
||||
path = root/f'{n}.jsonl'
|
||||
path.write_text(json.dumps(record)+'\n')
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps({'source': record['source'], 'list_url': rss.ROOT, 'count': 1}))
|
||||
ingest_file(root/'sample.sqlite', path)
|
||||
with sqlite3.connect(root/'sample.sqlite') as c:
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0], 1)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM source_records').fetchone()[0], 3)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM submissions').fetchone()[0], 2)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM decisions').fetchone()[0], 2)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM submission_sources').fetchone()[0], 3)
|
||||
self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
|
||||
def test_two_real_papers_collection_commits_evidence_only_after_success(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
index = {pid: self.index[pid] for pid in [1, 2]}
|
||||
accepted = {pid: self.accepted[pid] for pid in [1, 2]}
|
||||
demos = {2: self.demos[2]}
|
||||
with (patch.object(rss, 'EXPECTED', 2), patch.object(rss, 'DEMO_IDS', {2}),
|
||||
patch.object(rss, 'parse_index', return_value=index),
|
||||
patch.object(rss, 'parse_program', side_effect=[accepted, demos])):
|
||||
result = rss.collect(2025, root/'out', snapshots=FIXTURES)
|
||||
self.assertEqual((result['records'], len(result['files'])), (2, 2))
|
||||
self.assertFalse(result['database_written'])
|
||||
self.assertFalse(result['pdfs_downloaded'])
|
||||
self.assertEqual(len(result['source_evidence']), 6)
|
||||
for item in result['source_evidence']:
|
||||
self.assertEqual(hashlib.sha256((root/'out'/item['file']).read_bytes()).hexdigest(), item['sha256'])
|
||||
for item in result['files']:
|
||||
self.assertEqual(item['missing_abstract_count'], 0)
|
||||
self.assertEqual(hashlib.sha256(Path(item['file']).read_bytes()).hexdigest(), item['file_sha256'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user