Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 codex-bot
|
443a79dce2 |
@@ -51,7 +51,7 @@ jobs:
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','www-archive','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','aamas-proceedings','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','iclr-archive','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','rss-proceedings','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','collect','acl','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','postgres','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','edition-tracks','--help'],cwd=outside,check=True)
|
||||
subprocess.run([python,'-I','-B','-m','paper_atlas','maintain','joint-editions','--help'],cwd=outside,check=True)
|
||||
|
||||
@@ -8,10 +8,10 @@ Paper Atlas 将 arXiv、AI 会议论文集及公开投稿决定整理到可追
|
||||
|
||||
历史 ICLR 2014–2016 官方录用页由 `collect iclr-archive` 适配，保留主会/workshop及重复展示证据；运行和身份核验边界见 [历史档案说明](docs/coverage/iclr-archive-2014-2016.md)。
|
||||
|
||||
RSS 2025 的 `collect rss-proceedings` 读取官方逐篇标题、完整作者、摘要和 DOI，并核对录用与演示节目单；不下载 PDF。15 个演示 ID 有明确来源，其余逐篇投稿类型保持未明确，不用标题或会话主题代替 track。运行、证据恢复和限制见 [RSS 元数据说明](docs/rss-proceedings.md)。
|
||||
|
||||
## 项目结构
|
||||
|
||||
ACL 1978 的年会与 TINLAP-2 来源关系按精确卷适配，保留原系列和不明逐篇角色；1965 的替代关系不转成全局别名。来源、排除条目和恢复边界见 [ACL 早期来源核查](docs/coverage/acl-early-sources-2026-11-09.md)。
|
||||
|
||||
```text
|
||||
paper_atlas/
|
||||
cli.py 统一命令入口
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# ACL 早期年份：会议存在、出版目录与逐篇资料分开核验
|
||||
|
||||
固定普查中 ACL 1963–1978 仍有年份缺口。不能因为会议年表连续，就假设每年都有独立的正式论文集；也不能因为当前目录始于 1979，就断言此前没有摘要、通知或其他公开资料。本次不修改固定的 1,266 个目标年份基准。
|
||||
|
||||
## 1978 年：限定 T78-1 的来源关系
|
||||
|
||||
[同期年会通知](https://aclanthology.org/J77-4001.pdf)的 PDF 第 9–10 页（印刷页码 9–10）明确说明，1978 年 ACL 年会采用第二届 Theoretical Issues in Natural Language Processing 的形式，地点为 Urbana/Champaign，日期为 7 月 25–27 日。通知中的 ACM 联合赞助确认仍处于待确认状态，不能将该预告改写为已完成的行政决定。
|
||||
|
||||
[1982 年 Donald E. Walker 的历史回顾](https://aclanthology.org/www.mt-archive.info/70/ACL-1982-Walker-1.pdf)第 3 页（印刷页码 91）的年会表及缩写说明，独立对应了 1978 年的时间、地点与 TNLP/TINLAP-2 关系。两份扫描件相关页已渲染并逐页目视核对，不仅依靠 OCR。
|
||||
|
||||
[官方卷目录](https://aclanthology.org/volumes/T78-1/)与固定 XML `T78.xml` 保留原系列名 TINLAP、完整卷名、1978 年及编辑 David L. Waltz。适配仅接受精确的 collection `T78`、volume `1` 和上述字段组合；关键元数据变化会拒绝处理，等待重新核查。
|
||||
|
||||
- 规范事件名使用 `ACL+TINLAP`，原系列名 `TINLAP`、原始 XML 的 `venue_codes: [tinlap]` 及证据均保留。
|
||||
- ACL/TINLAP 别名只挂在这一规范事件系列上，不把所有 TINLAP 年份都算成 ACL。T75、T87 及 T78 其他卷没有此例外。
|
||||
- 通知描述了邀请与提交论文两种参与方式，XML 没有逐篇角色。统一暂存为 `proceedings_unspecified`，不猜测每篇属于 invited、main 或 workshop。
|
||||
- XML 共 36 条条目，其中 T78-1036 是无作者的场次讨论问题列表。精确识别并在 manifest 的 `excluded_materials` 保留其标识、标题与原因；其余 35 条作为具名贡献整理。该排除不推广到所有无作者条目，已识别条目发生变化也须重新核查。
|
||||
- 全部 35 条 XML 均无摘要。出版目录中的 accepted 是来源观察；通知提及录用／不录用通知时间，不提供任何可识别的拒稿决定，不能制造 rejected 或 withdrawn。
|
||||
|
||||
采集器仅生成元数据文件，不写业务数据库。真实来源逐条验收、PostgreSQL 身份预检、串行导入与独立验收仍是后续门槛。35 条来源不等于 35 个新增论文实体，更不等于完成全库去重。
|
||||
|
||||
## 1965 年：替代关系不改写为别名
|
||||
|
||||
Walker 回顾第 1 页（印刷页码 89）说明，1965 年 ICCL（后来称 COLING）替代了当年的年会。[官方 C65-1 目录](https://aclanthology.org/volumes/C65-1/)及 XML 仍以 COLING 标记该卷。这是限定年份的历史关系，不证明 COLING 与 ACL 是两个可全局互换的名字。
|
||||
|
||||
本次保留 C65 原有 `COLING` 归属，不添加 ACL 全局别名、不重放已导入记录、不将某一年的日历关系推广到整个系列，也不据此悄悄减少固定普查缺口。其他 1963–1977 年资料仍须逐年核查；本次不能证明这些年份整体属于非归档例外。
|
||||
|
||||
## 固定来源与恢复
|
||||
|
||||
服务器证据目录：`/data/paper-index/collected/acl-early-source-evidence-v1`。五份官方响应均 HTTP 200，另有两份来自既有官方 XML checkout 的文件，已与 Git 对象逐字节核对。未下载论文全文集合；两份 PDF 是本次历史／行政取证资料，不进入论文全文库。
|
||||
|
||||
| 文件 | SHA256 |
|
||||
|---|---|
|
||||
| `acl-venue.html` | `c60f64cb4d72e677aeb84a5844cde28a8d50579e1995f08e8c7801eed1e3fac5` |
|
||||
| `acl-history-1982.pdf` | `35c21bd4983975bcaccbb94bd8ab91c9019fc190feec1791255407cf18a7e49a` |
|
||||
| `acl-1978-notice.pdf` | `14b5deb471c55afb07d789529b95d0462a2a1a77df3dfba7fa1f5d3ab75eb411` |
|
||||
| `tinlap1978-volume.html` | `7e20d709f72f415819e4d17e3674b19dc7d871eabcf0cf8b4935f0392d775729` |
|
||||
| `coling1965-volume.html` | `0d2f2f49f2a311f9c0b80a8832f6476eb32f94f35abf36f6d2f2e10063ac880f` |
|
||||
| `T78.xml` | `f712f94cff0cba05f786b7ac31dad2ea9408fde2a17a04531130df623f57dd84` |
|
||||
| `C65.xml` | `572791610e20b7c6045d984d968e73a18a144bcb5c3da64112001c781826808e` |
|
||||
|
||||
XML commit：`a7be623d01c48c61405dec31c871121cdbed5a01`。测试 fixture `tests/fixtures/acl-t78.xml` 是该原始文件，不是手工模拟论文；不包含 PDF。PDF 哈希建立完整性，不单独证明语义，语义依据是上述具体页面。
|
||||
|
||||
在服务器使用既有命令，固定 checkout 并输出到新目录：
|
||||
|
||||
```sh
|
||||
paper-atlas collect acl \
|
||||
--checkout /data/paper-index/raw/metadata-checkouts/acl-anthology \
|
||||
--collection T78 \
|
||||
--out /data/paper-index/collected/acl-t78-source-v1
|
||||
```
|
||||
|
||||
该命令会校验／更新官方 metadata checkout，实际 commit 必须以生成清单为准；严格重现本次取证时，使用已冻结的 XML 和上述 commit，不把更新后的内容冒充旧快照。不要将无 collection 限制的全部历史输出直接作为增量队列。采集完成不代表导入完成，恢复前先检查当前单元和报告，不因日志安静重启已执行的写入。
|
||||
@@ -1,38 +0,0 @@
|
||||
# RSS 官方论文元数据
|
||||
|
||||
这是长期维护、按需运行的 package collector，不是定时服务或一次性导入脚本。目前仅接受独立核对过的 2025 年版式；不把 2026 年或其他年份套用同一分类规则。
|
||||
|
||||
## 数据范围和证据
|
||||
|
||||
- [RSS XXI 官方论文集](https://www.roboticsproceedings.org/rss21/)列出 163 个不同的逐篇 HTML 链接。每篇有 citation 元数据、BibTeX 和摘要区；PDF 链接作为原始元数据保存，不请求全文。
|
||||
- [官方 Accepted Papers](https://roboticsconference.org/2025/program/papers/)按论文 ID、标题和 session 对账。网页作者列包含隐藏的重复展示，逐篇有序 citation_author 与论文集目录作者完整对比后才采用；不截断作者、不将重复展示当新论文。
|
||||
- [官方 Demos](https://roboticsconference.org/2025/program/demos/)明确列出 15 个论文 ID，包括标题不以 Demonstrating 开头的 ID 93。这些条目标为 `demo`，保留整行节目证据。其他 148 条使用 `proceedings_unspecified`：没有逐篇类型依据时，不因缺席演示表就自动推断 main。
|
||||
- [作者说明](https://roboticsconference.org/2025/information/authorinfo/)描述 Science/Systems 与 Demo 两类投稿，两类都进入论文集。分类属于该来源的投稿/发表语义，不自动生成通用 research_type 标签。
|
||||
|
||||
`rss_proceedings:session` 保留官方 session，`rss_proceedings:program_role` 保留明确的 demonstration。二者与其他平台标签隔离；session 主题不等于投稿 track。单篇来源 ID 是 `2025/pNNN`，DOI 同时保留原始 BibTeX 大小写和标准化身份键，显式的 `publication_doi` 供原生导入器在同一会议年度/track 中关联其他正式发表来源；不能据此合并不同 track 或不同会议的投稿经历。
|
||||
|
||||
只有已发表、被官方录用表列出的条目才产生 accepted 记录。采集观察时间来自响应回执；出版日期单独留在 metadata，不伪造原始决定日期。来源未列出的论文不能推断为 rejected、desk rejected 或 withdrawn；公开负面决定是另一个待核范围。
|
||||
|
||||
## 运行
|
||||
|
||||
构建、安装和项目测试仅使用已确认的 `ces-control:host` 远程 runner。本机不能执行隐式构建的验证。服务器已安装环境运行示例：
|
||||
|
||||
```sh
|
||||
paper-atlas collect rss-proceedings --year 2025 \
|
||||
--snapshots /data/paper-index/collected/rss2025-full-evidence-v1 \
|
||||
--out /data/paper-index/collected/rss2025-source-v1
|
||||
```
|
||||
|
||||
也可使用 `--cache /data/paper-index/raw/rss-http` 请求限定官方 HTML 元数据。两种输入互斥。冻结快照的 `receipts.json` 每项必须包含 `file`、准确 `url/final_url`、`http_status: 200`、`content_type: text/html`、`bytes`、`sha256` 和含时区的观察时间 `at`。路径必须位于快照内，不能通过符号链接逃逸。所有输入逐个核验字节数与哈希，UTF-8 解码；重定向、未知版式和异常 ID 需要先核查。
|
||||
|
||||
采集只输出 JSONL、每个 track 的 manifest、完整 HTML 证据和 `rss-2025.json` 来源报告，不连接生产数据库。两份清单状态保持 `partial`；即使该卷 163 个链接全部解析，也不表示独立作品去重、逐篇类型或公开负面决定全集已完成。明确为空的摘要保存 `null` 和原因；缺失或重复摘要边界属于版式异常，停止输出。
|
||||
|
||||
## 冻结与恢复
|
||||
|
||||
2026-11-09 在服务器冻结 167 个官方 HTML 响应：目录、录用表、演示表、作者说明和 163 个逐篇页面。回执 SHA256：`f427d93a113702030864e789efe1176d69f71d2c567e81536d155e53d9513a93`。六个未改写样例位于 `tests/fixtures/rss2025/`，附其真实响应回执；完整原件保存在服务器来源目录与独立审阅备份，不作为运行时下载依赖。
|
||||
|
||||
整个采集持有输出目录的排他锁。全部解析和交叉检查通过后才发布 JSONL、manifest 和完成报告；完成报告最后发布。已有完成报告或 JSONL 时拒绝覆盖。若发布中断，先核对目录、回执和所有文件哈希，保留现场，再换新输出目录重采；不能看到日志安静就重启或重复导入。新的官方页面内容或计数与冻结样本不同，必须另留快照、解释变化并验证后再更新采集器。
|
||||
|
||||
生产导入前必须另外做来源独立核验和 PostgreSQL 只读身份预检。DOI、同标题候选和既有 arXiv 实体需要逐项核查，保留跨会的所有投稿和决定关系；发现冲突时隔离，不能只按标题强合并。已导入队列不得重放。生产独立验收通过后才重新统计年度缺口；导入成功和 CI 不能替代最终全库去重、摘要/标签及约束和实际查询审计。
|
||||
|
||||
本说明中的 163/15 是冻结官方页面的来源清单，不是当前生产库导入结果。历史缺口在本批生产验收前仍为 47。
|
||||
@@ -14,7 +14,6 @@ COMMANDS = {
|
||||
('collect','pmlr'): ('paper_atlas.collectors.pmlr', {}, 'PMLR 官方元数据'),
|
||||
('collect','acl'): ('paper_atlas.collectors.acl', {}, 'ACL Anthology XML'),
|
||||
('collect','iclr-archive'): ('paper_atlas.collectors.iclr_archive', {}, '核对ICLR 2014–2016官方历史录用档案'),
|
||||
('collect','rss-proceedings'): ('paper_atlas.collectors.rss', {}, '核对RSS 2025官方论文摘要与演示证据'),
|
||||
('collect','iclr-program'): ('paper_atlas.collectors.iclr', {}, 'ICLR 官方公开程序'),
|
||||
('collect','iclr-snapshot'): ('paper_atlas.collectors.openreview', {}, 'ICLR 公开投稿与决定快照'),
|
||||
('collect','public-openreview'): ('paper_atlas.collectors.public_openreview', {}, '核验其他会议的公开拒稿及撤稿标签'),
|
||||
|
||||
@@ -6,6 +6,7 @@ import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
|
||||
from paper_atlas.io.git_metadata import checkout_metadata
|
||||
from paper_atlas.collectors.acl_historical import event_scope, material_reason
|
||||
|
||||
VENUES={'acl':'ACL','emnlp':'EMNLP','naacl':'NAACL','eacl':'EACL','coling':'COLING','conll':'CoNLL'}
|
||||
|
||||
@@ -29,7 +30,10 @@ def paper_id(collection,volume,number):
|
||||
return f'{collection}-{int(volume):0{width}d}{int(number):0{digits}d}'
|
||||
|
||||
|
||||
def volume_scope(meta,volume):
|
||||
def volume_scope(meta,volume,collection=None):
|
||||
historical=event_scope(meta,volume,collection)
|
||||
if historical:
|
||||
return historical['canonical_venue'],historical['venue_aliases'],historical['track']
|
||||
codes=[text(n).lower() for n in meta.findall('venue')]
|
||||
book=text(meta.find('booktitle'))
|
||||
lower=book.lower()
|
||||
@@ -59,7 +63,7 @@ def collect_xml(path,out,commit):
|
||||
vid=volume.attrib['id']
|
||||
meta=volume.find('meta')
|
||||
if meta is None:raise ValueError('Volume metadata missing')
|
||||
scope=volume_scope(meta,vid)
|
||||
scope=volume_scope(meta,vid,collection)
|
||||
key=f'{collection}-{vid}'
|
||||
if scope is None:
|
||||
results.append({'volume':key,'status':'excluded_by_venue','venue_codes':[text(n) for n in meta.findall('venue')],'booktitle':text(meta.find('booktitle')),'year':text(meta.find('year'))})
|
||||
@@ -70,12 +74,18 @@ def collect_xml(path,out,commit):
|
||||
year=int(year_text)
|
||||
records=[]
|
||||
frontmatter=0
|
||||
excluded_materials=[]
|
||||
historical=event_scope(meta,vid,collection)
|
||||
for paper in volume.findall('paper'):
|
||||
pid=paper.attrib['id']
|
||||
if int(pid)==0:
|
||||
frontmatter+=1
|
||||
continue
|
||||
identifier=paper_id(collection,vid,pid)
|
||||
reason=material_reason(collection,vid,paper)
|
||||
if reason:
|
||||
excluded_materials.append({'external_id':identifier,'title':text(paper.find('title')),'reason':reason})
|
||||
continue
|
||||
doi=text(paper.find('doi'))
|
||||
if doi.startswith('10.18653/v1/') and doi.split('10.18653/v1/',1)[1].casefold()!=identifier.casefold():
|
||||
raise ValueError('Official DOI disagrees with constructed Anthology ID: '+identifier)
|
||||
@@ -83,6 +93,8 @@ def collect_xml(path,out,commit):
|
||||
if not title:raise ValueError('Missing paper title: '+identifier)
|
||||
authors=[{'given':text(a.find('first')),'family':text(a.find('last'))} for a in paper.findall('author')]
|
||||
records.append({'source':'acl_anthology','external_id':identifier,'url':f'https://aclanthology.org/{identifier}/','title':title,'abstract':text(paper.find('abstract')) or None,'authors':authors,'identifiers':{'acl':identifier,'doi':doi},'venue':venue,'venue_aliases':aliases if len(aliases)>1 else [],'year':year,'track':track,'status':'accepted','raw_decision':'listed in authoritative ACL Anthology proceedings XML','metadata':{'collection':collection,'volume':vid,'booktitle':text(meta.find('booktitle')),'venue_codes':[text(n) for n in meta.findall('venue')],'pages':text(paper.find('pages')),'publisher':text(meta.find('publisher')),'metadata_commit':commit,'source_xml':f'https://github.com/acl-org/acl-anthology/blob/{commit}/data/xml/{collection}.xml'}})
|
||||
if historical:
|
||||
records[-1]['metadata']['historical_event_scope']=historical
|
||||
if not records:
|
||||
results.append({'volume':key,'status':'unavailable','reason':'no research paper elements after excluding front matter'})
|
||||
continue
|
||||
@@ -93,7 +105,7 @@ def collect_xml(path,out,commit):
|
||||
for record in records:output.write(json.dumps(record,ensure_ascii=False)+'\n')
|
||||
part.replace(target)
|
||||
missing=sum(not row['abstract'] for row in records)
|
||||
manifest={'source':'acl_anthology','venue':venue,'year':year,'track':track,'count':len(records),'xml_paper_elements':len(volume.findall('paper')),'excluded_frontmatter':frontmatter,'missing_abstract_count':missing,'list_url':f'https://aclanthology.org/{key}/','metadata_commit':commit,'file':str(target),'status':'partial' if missing else 'collected'}
|
||||
manifest={'source':'acl_anthology','venue':venue,'year':year,'track':track,'count':len(records),'xml_paper_elements':len(volume.findall('paper')),'excluded_frontmatter':frontmatter,'excluded_materials':excluded_materials,'missing_abstract_count':missing,'list_url':f'https://aclanthology.org/{key}/','metadata_commit':commit,'file':str(target),'status':'partial' if missing else 'collected'}
|
||||
target.with_suffix('.manifest.json').write_text(json.dumps(manifest,indent=2))
|
||||
results.append(manifest)
|
||||
print(json.dumps(manifest),flush=True)
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Edition-specific historical scope; never alias every TINLAP year to ACL."""
|
||||
|
||||
HISTORY_URL = 'https://aclanthology.org/www.mt-archive.info/70/ACL-1982-Walker-1.pdf'
|
||||
NOTICE_URL = 'https://aclanthology.org/J77-4001.pdf'
|
||||
BOOK = 'Theoretical Issues in Natural Language Processing-2'
|
||||
MATERIAL_TITLE = 'List of questions suggested for consideration in each session'
|
||||
|
||||
|
||||
def text(node):
|
||||
return ' '.join(''.join(node.itertext()).split()) if node is not None else ''
|
||||
|
||||
|
||||
def event_scope(meta, volume, collection):
|
||||
if (collection, volume) != ('T78', '1'):
|
||||
return None
|
||||
if ([text(n) for n in meta.findall('venue')] != ['tinlap']
|
||||
or [text(n) for n in meta.findall('year')] != ['1978']
|
||||
or [text(n) for n in meta.findall('booktitle')] != [BOOK]
|
||||
or [(text(n.find('first')), text(n.find('last'))) for n in meta.findall('editor')]
|
||||
!= [('David L.', 'Waltz')]):
|
||||
raise ValueError('Reviewed T78-1 scope metadata changed; independent review required')
|
||||
return {
|
||||
'collection': 'T78', 'volume': '1', 'original_venue': 'TINLAP',
|
||||
'event_year': 1978, 'event_dates': ['1978-07-25', '1978-07-27'],
|
||||
'relationship': 'ACL annual meeting took the form of TINLAP-2',
|
||||
'canonical_venue': 'ACL+TINLAP', 'venue_aliases': ['ACL', 'TINLAP'],
|
||||
'track': 'proceedings_unspecified',
|
||||
'track_scope_note': 'Invited and submitted contributions are described at volume level; individual roles are not identified in XML.',
|
||||
'evidence': [
|
||||
{'url': NOTICE_URL, 'sha256': '14b5deb471c55afb07d789529b95d0462a2a1a77df3dfba7fa1f5d3ab75eb411',
|
||||
'pdf_pages': [9, 10], 'printed_pages': ['9', '10'],
|
||||
'scope': 'Contemporary annual meeting notice, not individual decision notes; ACM confirmation was pending in this notice.'},
|
||||
{'url': HISTORY_URL, 'sha256': '35c21bd4983975bcaccbb94bd8ab91c9019fc190feec1791255407cf18a7e49a',
|
||||
'pdf_pages': [3], 'printed_pages': ['91'],
|
||||
'scope': 'Retrospective annual meeting table and TNLP abbreviation confirm the 1978 event association.'},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def material_reason(collection, volume, paper):
|
||||
if (collection, volume, paper.attrib.get('id')) != ('T78', '1', '36'):
|
||||
return None
|
||||
if text(paper.find('title')) != MATERIAL_TITLE or paper.findall('author'):
|
||||
raise ValueError('Reviewed T78-1036 material changed; independent review required')
|
||||
return 'Session question list, not an authored research contribution; no decision inferred.'
|
||||
@@ -1,279 +0,0 @@
|
||||
"""Reviewed RSS 2025 proceedings HTML; no PDF or private review access."""
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from datetime import date
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import tempfile
|
||||
import unicodedata
|
||||
|
||||
from paper_atlas.db.submissions import timestamp
|
||||
from paper_atlas.io.html import Document, Node, clean
|
||||
from paper_atlas.io.http import fetch
|
||||
|
||||
SOURCE = 'rss_proceedings'
|
||||
ROOT = 'https://www.roboticsproceedings.org/rss21/'
|
||||
HEADING = 'Robotics: Science and Systems XXI'
|
||||
PAGES = {'index': ROOT,
|
||||
'accepted': 'https://roboticsconference.org/2025/program/papers/',
|
||||
'demos': 'https://roboticsconference.org/2025/program/demos/',
|
||||
'authorinfo': 'https://roboticsconference.org/2025/information/authorinfo/'}
|
||||
EXPECTED = 163
|
||||
DEMO_IDS = {2, 20, 21, 33, 58, 59, 62, 84, 86, 92, 93, 107, 129, 132, 141}
|
||||
|
||||
|
||||
def one(values, description):
|
||||
if len(values) != 1:
|
||||
raise ValueError('Expected one '+description)
|
||||
return values[0]
|
||||
|
||||
|
||||
def comparable(text):
|
||||
return unicodedata.normalize('NFC', clean(text))
|
||||
|
||||
|
||||
def parse_index(html):
|
||||
document = Document(html)
|
||||
nodes = list(document.root.walk())
|
||||
if [clean(n.text()) for n in nodes if n.tag == 'h1'] != [HEADING]:
|
||||
raise ValueError('Unexpected RSS proceedings edition')
|
||||
result = {}
|
||||
for row in (n for n in nodes if n.tag == 'tr'):
|
||||
links = [n for n in row.walk() if n.tag == 'a'
|
||||
and re.fullmatch(r'p[0-9]{3}\.html', n.attrs.get('href', ''))]
|
||||
if not links:
|
||||
continue
|
||||
link = one(links, 'proceedings entry link')
|
||||
pid = int(link.attrs['href'][1:4])
|
||||
title = clean(link.text())
|
||||
authors = clean(one([n for n in row.walk() if n.tag == 'i'], 'index authors').text())
|
||||
if pid in result or pid < 1 or not title or not authors:
|
||||
raise ValueError('Empty or repeated RSS index entry')
|
||||
result[pid] = {'title': title, 'authors_display': authors,
|
||||
'url': ROOT+link.attrs['href']}
|
||||
occurrences = sum(n.tag == 'a' and bool(re.fullmatch(r'p[0-9]{3}\.html', n.attrs.get('href', '')))
|
||||
for n in nodes)
|
||||
if not result or len(result) != occurrences:
|
||||
raise ValueError('Unparsed or duplicate proceedings links')
|
||||
return result
|
||||
|
||||
|
||||
def parse_program(html, heading, columns):
|
||||
nodes = list(Document(html).root.walk())
|
||||
if [clean(n.text()) for n in nodes if n.tag == 'h1'] != [heading]:
|
||||
raise ValueError('Unexpected RSS program page')
|
||||
headers = [[clean(n.text()) for n in row.children if isinstance(n, Node) and n.tag == 'th']
|
||||
for row in nodes if row.tag == 'tr']
|
||||
if list(columns) not in headers:
|
||||
raise ValueError('Unexpected RSS program columns')
|
||||
result = {}
|
||||
for row in (n for n in nodes if n.tag == 'tr'):
|
||||
cells = [n for n in row.children if isinstance(n, Node) and n.tag == 'td']
|
||||
if not cells:
|
||||
continue
|
||||
if len(cells) != len(columns):
|
||||
raise ValueError('Unexpected RSS program row width')
|
||||
raw_id = clean(cells[0].text())
|
||||
if not re.fullmatch('[1-9][0-9]*', raw_id):
|
||||
raise ValueError('Invalid program ID')
|
||||
pid = int(raw_id)
|
||||
if pid in result:
|
||||
raise ValueError('Duplicate program ID')
|
||||
result[pid] = {key: clean(node.text()) for key, node in zip(columns, cells)}
|
||||
if not result[pid]['Title']:
|
||||
raise ValueError('Missing program title')
|
||||
if not result:
|
||||
raise ValueError('No RSS program rows')
|
||||
return result
|
||||
|
||||
|
||||
def validate_program(index, accepted, demos, authorinfo):
|
||||
if set(index) != set(accepted) or len(index) != EXPECTED or set(demos) != DEMO_IDS:
|
||||
raise ValueError('RSS program and reviewed proceedings census differ')
|
||||
text = clean(Document(authorinfo).root.text())
|
||||
if ('There are two types of submissions to RSS' not in text
|
||||
or 'Both types of papers appear in the RSS proceedings.' not in text):
|
||||
raise ValueError('Submission category evidence changed')
|
||||
for pid, item in index.items():
|
||||
variants = [accepted[pid]['Title']] + ([demos[pid]['Title']] if pid in demos else [])
|
||||
if any(comparable(title) != comparable(item['title']) for title in variants):
|
||||
raise ValueError('Conflicting official program title for '+str(pid))
|
||||
|
||||
|
||||
def parse_paper(html, pid, entry, accepted, demo, evidence):
|
||||
nodes = list(Document(html).root.walk())
|
||||
meta = {}
|
||||
for node in nodes:
|
||||
if node.tag == 'meta' and node.attrs.get('name', '').startswith('citation_'):
|
||||
meta.setdefault(node.attrs['name'], []).append(node.attrs.get('content', ''))
|
||||
title = one(meta.get('citation_title', []), 'citation title')
|
||||
authors = meta.get('citation_author', [])
|
||||
publication_date = one(meta.get('citation_publication_date', []), 'publication date')
|
||||
if not re.fullmatch(r'2025/[0-9]{2}/[0-9]{2}', publication_date):
|
||||
raise ValueError('Unexpected publication year/date')
|
||||
date.fromisoformat(publication_date.replace('/', '-'))
|
||||
if (one(meta.get('citation_conference_title', []), 'conference') != HEADING
|
||||
or one(meta.get('citation_volume', []), 'volume') != '21'
|
||||
or [clean(n.text()) for n in nodes if n.tag == 'h1'] != [HEADING]
|
||||
or [comparable(n.text()) for n in nodes if n.tag == 'h3'] != [comparable(title)]
|
||||
or comparable(title) != comparable(entry['title']) or not authors
|
||||
or any(not clean(author) for author in authors)
|
||||
or comparable(', '.join(authors)) != comparable(entry['authors_display'])):
|
||||
raise ValueError('Proceedings/citation title, authors or edition disagree')
|
||||
pdf = one(meta.get('citation_pdf_url', []), 'citation PDF URL')
|
||||
if pdf not in (ROOT+f'p{pid:03d}.pdf', ROOT.replace('https:', 'http:')+f'p{pid:03d}.pdf'):
|
||||
raise ValueError('Citation PDF ID differs; URL is metadata only')
|
||||
bibtex = one([n.text() for n in nodes if n.tag == 'pre'], 'original BibTeX')
|
||||
doi = one(re.findall(r'^\s*DOI\s*=\s*\{([^{}]+)\}', bibtex, re.I | re.M), 'BibTeX DOI')
|
||||
year = one(re.findall(r'^\s*YEAR\s*=\s*\{([^{}]+)\}', bibtex, re.I | re.M), 'BibTeX year')
|
||||
if doi != f'10.15607/RSS.2025.XXI.{pid:03d}' or year != '2025':
|
||||
raise ValueError('BibTeX DOI/year differs from volume and paper ID')
|
||||
abstract_parts = []
|
||||
inside = False
|
||||
markers = []
|
||||
for node in (n for n in nodes if n.tag == 'p'):
|
||||
labels = [clean(n.text()) for n in node.walk() if n.tag == 'b']
|
||||
if 'Abstract:' in labels:
|
||||
markers.append('start')
|
||||
inside = True
|
||||
value = clean(node.text()).removeprefix('Abstract:').strip()
|
||||
if value:
|
||||
abstract_parts.append(value)
|
||||
elif 'Download:' in labels:
|
||||
markers.append('end')
|
||||
inside = False
|
||||
elif inside:
|
||||
abstract_parts.append(clean(node.text()))
|
||||
if markers != ['start', 'end']:
|
||||
raise ValueError('Abstract boundaries missing or repeated')
|
||||
abstract = clean(' '.join(abstract_parts)) or None
|
||||
track = 'demo' if demo else 'proceedings_unspecified'
|
||||
tags = [{'namespace': SOURCE+':session', 'value': accepted['Session']}]
|
||||
if demo:
|
||||
tags.append({'namespace': SOURCE+':program_role', 'value': 'demonstration'})
|
||||
return {'source': SOURCE, 'external_id': f'2025/p{pid:03d}', 'url': entry['url'],
|
||||
'title': title, 'abstract': abstract, 'authors': authors,
|
||||
'identifiers': {'doi': doi.lower()},
|
||||
'submission_identifiers': [{'scheme': 'publication_doi', 'value': doi.lower()}],
|
||||
'venue': 'RSS', 'venue_aliases': [], 'year': 2025, 'track': track,
|
||||
'status': 'accepted', 'observed_at': evidence['paper']['observed_at'],
|
||||
'raw_decision': 'listed in official RSS 2025 proceedings and accepted program',
|
||||
'tags': tags,
|
||||
'metadata': {'publisher_paper_id': pid, 'citation_metadata': meta,
|
||||
'publication_date': publication_date, 'original_bibtex': bibtex,
|
||||
'index_entry': entry, 'accepted_program_entry': accepted,
|
||||
'demo_program_entry': demo, 'source_evidence': evidence,
|
||||
'missing_abstract_reason': None if abstract else 'empty_official_abstract_section',
|
||||
'track_scope_note': ('Explicit official demo program ID; not inferred from title.' if demo else
|
||||
'Proceedings and accepted list do not individually identify submission type; session is not a track.'),
|
||||
'decision_date_precision': 'not_supplied',
|
||||
'publication_evidence': 'Published proceedings and accepted program; not an original decision note.'}}
|
||||
|
||||
|
||||
class Pages:
|
||||
def __init__(self, *, cache=None, snapshots=None):
|
||||
if bool(cache) == bool(snapshots):
|
||||
raise ValueError('Choose exactly one cache or frozen snapshot directory')
|
||||
self.cache = Path(cache) if cache else None
|
||||
self.root = Path(snapshots).resolve() if snapshots else None
|
||||
self.receipts = json.loads((self.root/'receipts.json').read_bytes())['receipts'] if self.root else []
|
||||
self.evidence = {}
|
||||
self.data = {}
|
||||
|
||||
def read(self, url):
|
||||
if self.root:
|
||||
receipt = one([r for r in self.receipts if r['url'] == url], 'exact frozen response receipt')
|
||||
filename = Path(receipt['file'])
|
||||
path = self.root/filename
|
||||
if (filename.is_absolute() or '..' in filename.parts or not path.resolve().is_relative_to(self.root)
|
||||
or receipt['http_status'] != 200 or receipt['content_type'] != 'text/html'
|
||||
or receipt.get('final_url', url) != url):
|
||||
raise ValueError('Unsafe or unsuccessful frozen response')
|
||||
data = path.read_bytes()
|
||||
observed = receipt['at']
|
||||
else:
|
||||
data = fetch(url, self.cache, 2*1024**2, delay=1)
|
||||
receipt = json.loads((self.cache/hashlib.sha256(url.encode()).hexdigest()).with_suffix('.json').read_bytes())
|
||||
observed = receipt['retrieved_at']
|
||||
if (len(data) > 2*1024**2 or len(data) != receipt['bytes']
|
||||
or hashlib.sha256(data).hexdigest() != receipt['sha256'] or receipt['url'] != url):
|
||||
raise ValueError('Response checksum, URL or size differs')
|
||||
evidence = {'url': url, 'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data),
|
||||
'file': 'evidence/'+hashlib.sha256(data).hexdigest()+'.html',
|
||||
'observed_at': timestamp(observed), 'http_status': 200, 'response_receipt': receipt}
|
||||
self.evidence[url] = evidence
|
||||
self.data[url] = data
|
||||
return data.decode('utf-8'), evidence
|
||||
|
||||
|
||||
def collect(year, out, *, cache=None, snapshots=None):
|
||||
if year != 2025:
|
||||
raise ValueError('Only the independently reviewed RSS 2025 edition is supported')
|
||||
out = Path(out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
with (out/'rss.collect.lock').open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
report_path = out/'rss-2025.json'
|
||||
if report_path.exists() or list(out.glob('rss-2025-*.jsonl')):
|
||||
raise FileExistsError('Inspect existing output; never overwrite a completed or partial queue')
|
||||
pages = Pages(cache=cache, snapshots=snapshots)
|
||||
context = {key: pages.read(url) for key, url in PAGES.items()}
|
||||
index = parse_index(context['index'][0])
|
||||
accepted = parse_program(context['accepted'][0], 'Accepted Papers', ('ID', 'Session', 'Title', 'Authors'))
|
||||
demos = parse_program(context['demos'][0], 'Demos', ('ID', 'Location', 'Title', 'Day', 'Time'))
|
||||
validate_program(index, accepted, demos, context['authorinfo'][0])
|
||||
records = []
|
||||
for pid, entry in sorted(index.items()):
|
||||
html, evidence = pages.read(entry['url'])
|
||||
sources = {key: item[1] for key, item in context.items()}
|
||||
sources['paper'] = evidence
|
||||
records.append(parse_paper(html, pid, entry, accepted[pid], demos.get(pid), sources))
|
||||
counts = Counter(r['track'] for r in records)
|
||||
if counts != {'demo': len(DEMO_IDS), 'proceedings_unspecified': EXPECTED-len(DEMO_IDS)}:
|
||||
raise ValueError('Track census differs; review official source changes')
|
||||
files = []
|
||||
with tempfile.TemporaryDirectory(prefix='.rss-', dir=out) as temporary:
|
||||
stage = Path(temporary)
|
||||
(stage/'evidence').mkdir()
|
||||
for url, data in pages.data.items():
|
||||
(stage/pages.evidence[url]['file']).write_bytes(data)
|
||||
for track in sorted(counts):
|
||||
group = [r for r in records if r['track'] == track]
|
||||
path = stage/f'rss-2025-{track}.jsonl'
|
||||
path.write_text(''.join(json.dumps(r, ensure_ascii=False)+'\n' for r in group))
|
||||
manifest = {'source': SOURCE, 'parser_version': 1, 'venue': 'RSS', 'year': 2025,
|
||||
'track': track, 'count': len(group), 'file': str(out/path.name),
|
||||
'file_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
'list_url': ROOT, 'status': 'partial',
|
||||
'missing_abstract_count': sum(r['abstract'] is None for r in group)}
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps(manifest, ensure_ascii=False, indent=2)+'\n')
|
||||
files.append(manifest)
|
||||
result = {'source': SOURCE, 'year': 2025, 'records': len(records), 'files': files,
|
||||
'source_evidence': list(pages.evidence.values()), 'database_written': False,
|
||||
'pdfs_downloaded': False, 'status': 'collected_pending_identity_review',
|
||||
'note': 'Only the published proceedings. Demo IDs have explicit program evidence; '
|
||||
'other individual submission types remain unknown. No absent-based negative decisions. '
|
||||
'Separate source identity review and independent production acceptance are required.'}
|
||||
(stage/report_path.name).write_text(json.dumps(result, ensure_ascii=False, indent=2)+'\n')
|
||||
(out/'evidence').mkdir(exist_ok=True)
|
||||
for path in (stage/'evidence').iterdir():
|
||||
path.replace(out/'evidence'/path.name)
|
||||
for path in stage.iterdir():
|
||||
if path.is_file() and path.name != report_path.name:
|
||||
path.replace(out/path.name)
|
||||
(stage/report_path.name).replace(report_path)
|
||||
return result
|
||||
|
||||
|
||||
def main(argv=None, *, prog=None):
|
||||
parser = argparse.ArgumentParser(prog=prog)
|
||||
parser.add_argument('--year', type=int, choices=[2025], required=True)
|
||||
parser.add_argument('--out', required=True)
|
||||
group = parser.add_mutually_exclusive_group(required=True)
|
||||
group.add_argument('--cache')
|
||||
group.add_argument('--snapshots')
|
||||
args = parser.parse_args(argv)
|
||||
print(json.dumps(collect(args.year, args.out, cache=args.cache, snapshots=args.snapshots), ensure_ascii=False))
|
||||
Vendored
+273
@@ -0,0 +1,273 @@
|
||||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<collection id="T78">
|
||||
<volume id="1" type="proceedings">
|
||||
<!-- https://aclanthology.org/T78-1/ -->
|
||||
<meta>
|
||||
<booktitle>Theoretical Issues in Natural Language <fixed-case>P</fixed-case>rocessing-2</booktitle>
|
||||
<editor id="david-l-waltz"><first>David L.</first><last>Waltz</last></editor>
|
||||
<venue>tinlap</venue>
|
||||
<year>1978</year>
|
||||
</meta>
|
||||
<frontmatter>
|
||||
<!-- https://aclanthology.org/T78-1000/ -->
|
||||
<pdf hash="fb6164ad"/>
|
||||
<bibkey>tinlap-1978-theoretical</bibkey>
|
||||
</frontmatter>
|
||||
<paper id="1">
|
||||
<!-- https://aclanthology.org/T78-1001/ -->
|
||||
<title>Testing The Psychological Reality of a Representational Model</title>
|
||||
<author><first>Dedre</first><last>Gentner</last></author>
|
||||
<pdf hash="f723a956"/>
|
||||
<bibkey>gentner-1978-testing-psychological</bibkey>
|
||||
</paper>
|
||||
<paper id="2">
|
||||
<!-- https://aclanthology.org/T78-1002/ -->
|
||||
<title>What Makes Something “Ad Hoc”</title>
|
||||
<author id="roger-c-schank"><first>Roger C.</first><last>Schank</last></author>
|
||||
<pdf hash="636853f9"/>
|
||||
<bibkey>schank-1978-makes-something</bibkey>
|
||||
</paper>
|
||||
<paper id="3">
|
||||
<!-- https://aclanthology.org/T78-1003/ -->
|
||||
<title>The Relation of Grammar to Cognition–a Synopsis</title>
|
||||
<author><first>Leonard</first><last>Talmy</last></author>
|
||||
<pdf hash="101c67dc"/>
|
||||
<bibkey>talmy-1978-relation-grammar</bibkey>
|
||||
</paper>
|
||||
<paper id="4">
|
||||
<!-- https://aclanthology.org/T78-1004/ -->
|
||||
<title>On primitives, prototypes, and other semantic anomalies</title>
|
||||
<author><first>Terry</first><last>Winograd</last></author>
|
||||
<pdf hash="f8e47023"/>
|
||||
<bibkey>winograd-1978-primitives-prototypes</bibkey>
|
||||
</paper>
|
||||
<paper id="5">
|
||||
<!-- https://aclanthology.org/T78-1005/ -->
|
||||
<title>Taxonomic Lattice Structures for Situation Recognition</title>
|
||||
<author id="william-a-woods"><first>William A.</first><last>Woods</last></author>
|
||||
<pdf hash="92957ddf"/>
|
||||
<bibkey>woods-1978-taxonomic-lattice</bibkey>
|
||||
</paper>
|
||||
<paper id="6">
|
||||
<!-- https://aclanthology.org/T78-1006/ -->
|
||||
<title>Description Formation and Discourse Model Synthesis</title>
|
||||
<author id="bonnie-webber"><first>Bonnie Lynn</first><last>Webber</last></author>
|
||||
<pdf hash="ffded1d2"/>
|
||||
<bibkey>webber-1978-description-formation</bibkey>
|
||||
</paper>
|
||||
<paper id="7">
|
||||
<!-- https://aclanthology.org/T78-1007/ -->
|
||||
<title>The Processing of Referring Expressions within a Semantic Network</title>
|
||||
<author><first>John R.</first><last>Anderson</last></author>
|
||||
<pdf hash="8bcc083a"/>
|
||||
<bibkey>anderson-1978-processing-referring</bibkey>
|
||||
</paper>
|
||||
<paper id="8">
|
||||
<!-- https://aclanthology.org/T78-1008/ -->
|
||||
<title>Reference Diaries</title>
|
||||
<author><first>Herbert H.</first><last>Clark</last></author>
|
||||
<author><first>Catherine</first><last>Marshall</last></author>
|
||||
<pdf hash="d06b7ea2"/>
|
||||
<bibkey>clark-marshall-1978-reference-diaries</bibkey>
|
||||
</paper>
|
||||
<paper id="9">
|
||||
<!-- https://aclanthology.org/T78-1009/ -->
|
||||
<title>Subsequent Reference: Syntactic and Rhetorical Constraints</title>
|
||||
<author id="david-d-mcdonald"><first>David D.</first><last>McDonald</last></author>
|
||||
<pdf hash="2d424559"/>
|
||||
<bibkey>mcdonald-1978-subsequent-reference</bibkey>
|
||||
</paper>
|
||||
<paper id="10">
|
||||
<!-- https://aclanthology.org/T78-1010/ -->
|
||||
<title>Some Psycholinguistic Constraints on the Construction and Interpretation of Definite Descriptions</title>
|
||||
<author><first>Andrew</first><last>Ortony</last></author>
|
||||
<pdf hash="69807727"/>
|
||||
<bibkey>ortony-1978-psycholinguistic-constraints</bibkey>
|
||||
</paper>
|
||||
<paper id="11">
|
||||
<!-- https://aclanthology.org/T78-1011/ -->
|
||||
<title>Bound Variables and Other Anaphors</title>
|
||||
<author id="barbara-h-partee"><first>Barbara H.</first><last>Partee</last></author>
|
||||
<pdf hash="e8e6a510"/>
|
||||
<bibkey>partee-1978-bound-variables</bibkey>
|
||||
</paper>
|
||||
<paper id="12">
|
||||
<!-- https://aclanthology.org/T78-1012/ -->
|
||||
<title>The Use of Focus as a Tool for Disambiguation of Definite Noun Phrases</title>
|
||||
<author id="candace-l-sidner"><first>Candace L.</first><last>Sidner</last></author>
|
||||
<pdf hash="7b96d2b7"/>
|
||||
<bibkey>sidner-1978-use-focus</bibkey>
|
||||
</paper>
|
||||
<paper id="13">
|
||||
<!-- https://aclanthology.org/T78-1013/ -->
|
||||
<title>Focusing in Dialog</title>
|
||||
<author id="barbara-j-grosz"><first>Barbara J.</first><last>Grosz</last></author>
|
||||
<pdf hash="8c8d4c2e"/>
|
||||
<bibkey>grosz-1978-focusing-dialog</bibkey>
|
||||
</paper>
|
||||
<paper id="14">
|
||||
<!-- https://aclanthology.org/T78-1014/ -->
|
||||
<title>Topic Levels</title>
|
||||
<author><first>Joseph E.</first><last>Grimes</last></author>
|
||||
<pdf hash="f485b2ba"/>
|
||||
<bibkey>grimes-1978-topic-levels</bibkey>
|
||||
</paper>
|
||||
<paper id="15">
|
||||
<!-- https://aclanthology.org/T78-1015/ -->
|
||||
<title>Toward a Rational Model of Discourse Comprehension</title>
|
||||
<author><first>Jerry L.</first><last>Morgan</last></author>
|
||||
<pdf hash="5e12a597"/>
|
||||
<bibkey>morgan-1978-toward-rational</bibkey>
|
||||
</paper>
|
||||
<paper id="16">
|
||||
<!-- https://aclanthology.org/T78-1016/ -->
|
||||
<title>Assent and Compliance in Children’s Language Comprehension</title>
|
||||
<author><first>David R.</first><last>Olson</last></author>
|
||||
<pdf hash="708bc4a0"/>
|
||||
<bibkey>olson-1978-assent-compliance</bibkey>
|
||||
</paper>
|
||||
<paper id="17">
|
||||
<!-- https://aclanthology.org/T78-1017/ -->
|
||||
<title>Speech Acts as a Basis for Understanding Dialogue Coherence</title>
|
||||
<author id="c-raymond-perrault"><first>C. Raymond</first><last>Perrault</last></author>
|
||||
<author id="james-allen"><first>James F.</first><last>Allen</last></author>
|
||||
<pdf hash="d111402b"/>
|
||||
<bibkey>perrault-allen-1978-speech</bibkey>
|
||||
</paper>
|
||||
<paper id="18">
|
||||
<!-- https://aclanthology.org/T78-1018/ -->
|
||||
<title>A Framework for Comparing Language Experiences (with particular emphasis on: The Effect of Audience on Discourse Models)</title>
|
||||
<author><first>Andee</first><last>Rubin</last></author>
|
||||
<pdf hash="5d9a6150"/>
|
||||
<bibkey>rubin-1978-framework-comparing</bibkey>
|
||||
</paper>
|
||||
<paper id="19">
|
||||
<!-- https://aclanthology.org/T78-1019/ -->
|
||||
<title>Intentlonallty and Human Conversations</title>
|
||||
<author id="jaime-g-carbonell"><first>Jaime G.</first><last>Carbonell Jr</last></author>
|
||||
<pdf hash="c50fc619"/>
|
||||
<bibkey>carbonell-jr-1978-intentlonallty-human</bibkey>
|
||||
</paper>
|
||||
<paper id="20">
|
||||
<!-- https://aclanthology.org/T78-1020/ -->
|
||||
<title>On the Interdependence of Language and Perception</title>
|
||||
<author id="david-l-waltz"><first>David L.</first><last>Waltz</last></author>
|
||||
<pdf hash="3c708191"/>
|
||||
<bibkey>waltz-1978-interdependence-language</bibkey>
|
||||
</paper>
|
||||
<paper id="21">
|
||||
<!-- https://aclanthology.org/T78-1021/ -->
|
||||
<title>The Problem of Naming Shapes: Vision-Language Interface</title>
|
||||
<author id="ruzena-bajcsy"><first>R.</first><last>Bajcsy</last></author>
|
||||
<author id="aravind-joshi"><first>A. K.</first><last>Joshi</last></author>
|
||||
<pdf hash="50331b4e"/>
|
||||
<bibkey>bajcsy-joshi-1978-problem-naming</bibkey>
|
||||
</paper>
|
||||
<paper id="22">
|
||||
<!-- https://aclanthology.org/T78-1022/ -->
|
||||
<title>An Argument about the Composition of Conceptual Structure</title>
|
||||
<author><first>Ray</first><last>Jackendoff</last></author>
|
||||
<pdf hash="57d001f8"/>
|
||||
<bibkey>jackendoff-1978-argument-composition</bibkey>
|
||||
</paper>
|
||||
<paper id="23">
|
||||
<!-- https://aclanthology.org/T78-1023/ -->
|
||||
<title>On the Ontological Status of Visual Mental Images</title>
|
||||
<author><first>Stephen Michael</first><last>Kosslyn</last></author>
|
||||
<pdf hash="fd06e265"/>
|
||||
<bibkey>kosslyn-1978-ontological-status</bibkey>
|
||||
</paper>
|
||||
<paper id="24">
|
||||
<!-- https://aclanthology.org/T78-1024/ -->
|
||||
<title>What has language to do with perception? Some speculations on the <i>Lingua Mentis</i></title>
|
||||
<author><first>Zenon W.</first><last>Pylyshyn</last></author>
|
||||
<pdf hash="a2f3a044"/>
|
||||
<bibkey>pylyshyn-1978-language-perception</bibkey>
|
||||
</paper>
|
||||
<paper id="25">
|
||||
<!-- https://aclanthology.org/T78-1025/ -->
|
||||
<title>Semantic Primitives in Language and Vision</title>
|
||||
<author id="yorick-wilks"><first>Yorick</first><last>Wilks</last></author>
|
||||
<pdf hash="b4e13896"/>
|
||||
<bibkey>wilks-1978-semantic-primitives</bibkey>
|
||||
</paper>
|
||||
<paper id="26">
|
||||
<!-- https://aclanthology.org/T78-1026/ -->
|
||||
<title>A Note on Partial Match of Descriptions. Can One Simultaneously Question (Retrieve) and Inform (Update)?</title>
|
||||
<author id="aravind-joshi"><first>Aravind K.</first><last>Joshi</last></author>
|
||||
<pdf hash="74969243"/>
|
||||
<bibkey>joshi-1978-note-partial</bibkey>
|
||||
</paper>
|
||||
<paper id="27">
|
||||
<!-- https://aclanthology.org/T78-1027/ -->
|
||||
<title>With a Spoon in Hand This Must Be the Eating Frame</title>
|
||||
<author><first>Eugene</first><last>Charniak</last></author>
|
||||
<pdf hash="f1be68bb"/>
|
||||
<bibkey>charniak-1978-spoon-hand</bibkey>
|
||||
</paper>
|
||||
<paper id="28">
|
||||
<!-- https://aclanthology.org/T78-1028/ -->
|
||||
<title>Fragments of a Theory of Human Plausible Reasoning</title>
|
||||
<author><first>Allan</first><last>Collins</last></author>
|
||||
<pdf hash="67b4bca2"/>
|
||||
<bibkey>collins-1978-fragments-theory</bibkey>
|
||||
</paper>
|
||||
<paper id="29">
|
||||
<!-- https://aclanthology.org/T78-1029/ -->
|
||||
<title>Indirect Responses to Loaded Questions</title>
|
||||
<author><first>S. Jerrold</first><last>Kaplan</last></author>
|
||||
<pdf hash="135037f9"/>
|
||||
<bibkey>kaplan-1978-indirect-responses</bibkey>
|
||||
</paper>
|
||||
<paper id="30">
|
||||
<!-- https://aclanthology.org/T78-1030/ -->
|
||||
<title>On Reasoning by Default</title>
|
||||
<author><first>Raymond</first><last>Reiter</last></author>
|
||||
<pdf hash="0de53d33"/>
|
||||
<bibkey>reiter-1978-reasoning-default</bibkey>
|
||||
</paper>
|
||||
<paper id="31">
|
||||
<!-- https://aclanthology.org/T78-1031/ -->
|
||||
<title>Path-Based and Node-Based Inference in Semantic Networks</title>
|
||||
<author id="stuart-c-shapiro"><first>Stuart C.</first><last>Shapiro</last></author>
|
||||
<pdf hash="2ddc5535"/>
|
||||
<bibkey>shapiro-1978-path-based</bibkey>
|
||||
</paper>
|
||||
<paper id="32">
|
||||
<!-- https://aclanthology.org/T78-1032/ -->
|
||||
<title>The Representation of Derivable Information in Memory: When What Might Have Been Left Unsaid Is Said</title>
|
||||
<author><first>Rand J.</first><last>Spiro</last></author>
|
||||
<author><first>Joseph</first><last>Esposito</last></author>
|
||||
<author><first>Richard J.</first><last>Vondruska</last></author>
|
||||
<pdf hash="7758fb45"/>
|
||||
<bibkey>spiro-etal-1978-representation-derivable</bibkey>
|
||||
</paper>
|
||||
<paper id="33">
|
||||
<!-- https://aclanthology.org/T78-1033/ -->
|
||||
<title>A Heuristic for Paradigms</title>
|
||||
<author><first>Joseph E.</first><last>Grimes</last></author>
|
||||
<pdf hash="53b42c16"/>
|
||||
<bibkey>grimes-1978-heuristic-paradigms</bibkey>
|
||||
</paper>
|
||||
<paper id="34">
|
||||
<!-- https://aclanthology.org/T78-1034/ -->
|
||||
<title>A Computational Account of Some Constraints on Language</title>
|
||||
<author id="mitch-marcus"><first>Mitchell</first><last>Marcus</last></author>
|
||||
<pdf hash="c9724c43"/>
|
||||
<bibkey>marcus-1978-computational-account</bibkey>
|
||||
</paper>
|
||||
<paper id="35">
|
||||
<!-- https://aclanthology.org/T78-1035/ -->
|
||||
<title>Remarks on Processing, Constraints, and the Lexicon</title>
|
||||
<author id="thomas-wasow"><first>Thomas</first><last>Wasow</last></author>
|
||||
<pdf hash="8beed275"/>
|
||||
<bibkey>wasow-1978-remarks-processing</bibkey>
|
||||
</paper>
|
||||
<paper id="36">
|
||||
<!-- https://aclanthology.org/T78-1036/ -->
|
||||
<title>List of questions suggested for consideration in each session</title>
|
||||
<pdf hash="1950b908"/>
|
||||
<bibkey>nn-1978-list-questions</bibkey>
|
||||
</paper>
|
||||
</volume>
|
||||
</collection>
|
||||
Vendored
-4356
File diff suppressed because it is too large
Load Diff
-1501
File diff suppressed because it is too large
Load Diff
Vendored
-1646
File diff suppressed because it is too large
Load Diff
Vendored
-1585
File diff suppressed because it is too large
Load Diff
Vendored
-148
@@ -1,148 +0,0 @@
|
||||
<html>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
||||
<head>
|
||||
<meta name="citation_title" content="Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation" />
|
||||
<meta name="citation_author" content="Pascal Roth" />
|
||||
<meta name="citation_author" content="Jonas Frey" />
|
||||
<meta name="citation_author" content="Cesar Cadena" />
|
||||
<meta name="citation_author" content="Marco Hutter" />
|
||||
<meta name="citation_publication_date" content="2025/06/21" />
|
||||
<meta name="citation_conference_title" content="Robotics: Science and Systems XXI" />
|
||||
<meta name="citation_isbn" content="979-8-9902848-1-4" />
|
||||
<meta name="citation_volume" content="21" />
|
||||
<meta name="citation_pdf_url" content="http://www.roboticsproceedings.org/rss21/p001.pdf" />
|
||||
<title>Robotics: Science and Systems XXI - Online Proceedings</title>
|
||||
<link type="text/css" rel="stylesheet" href="../style.css">
|
||||
<!--[if IE 5]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie5.css">
|
||||
<![endif]-->
|
||||
<!--[if IE 6]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie6.css">
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div class="menu">
|
||||
<ul>
|
||||
<li class="here"><a class="menu" href="../index.html">Home</a></li>
|
||||
<li class="label">RSS XXII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XXI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/authors.html">Authors</a></li>
|
||||
<li class="label">RSS X</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/authors.html">Authors</a></li>
|
||||
<li class="label">RSS V</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/authors.html">Authors</a></li>
|
||||
<li class="label">RSS III</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/authors.html">Authors</a></li>
|
||||
<li class="label">RSS II</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/authors.html">Authors</a></li>
|
||||
<li class="label">RSS I</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/authors.html">Authors</a></li>
|
||||
<li class="label">Ethics and Malpractice</span></li>
|
||||
<li class="menu"><a class="menu" href="../statement.html">Statement</a></li>
|
||||
<li class="label">Contact</span></li>
|
||||
<li class="last"><a class="menu" href="../contact.html">Webmaster</a></li>
|
||||
<li class="no-style"><center><img border="0" src="../robot-small.jpg"></center>
|
||||
<br><br><br>
|
||||
<center>We are using google analytics for statistics!</center>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="content">
|
||||
|
||||
<h1>Robotics: Science and Systems XXI</h1>
|
||||
<h3>Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation</h3>
|
||||
<i>Pascal Roth, Jonas Frey, Cesar Cadena, Marco Hutter</i><br>
|
||||
<p>
|
||||
<b>Abstract:</b>
|
||||
</p><p style="text-align: justify;">
|
||||
Ensuring safe navigation in complex environments requires accurate real-time traversability assessment and understanding of environmental interactions relative to the robot's capabilities. Traditional methods, which assume simplified dynamics, often require designing and tuning cost functions to safely guide paths or actions toward the goal. This process is tedious, environment-dependent, and not generalizable. To overcome these issues, we propose a novel learned perceptive Forward Dynamics Model (FDM) that predicts the robot's future state conditioned on the surrounding geometry and history of proprioceptive measurements, proposing a more scalable, safer, and heuristic-free solution. The FDM is trained on multiple years of simulated navigation experience, including high-risk maneuvers, and real-world interactions to incorporate the full system dynamics beyond rigid body simulation. We integrate our perceptive FDM into a zero-shot Model Predictive Path Integral (MPPI) planning framework, leveraging the learned mapping between actions, future states, and failure probability. This allows for optimizing a simplified cost function, eliminating the need for extensive cost-tuning to ensure safety. On the legged robot ANYmal, the proposed perceptive FDM improves the position estimation by on average 41% over competitive baselines, which translates into a 27% higher navigation success rate in rough simulation environments. Moreover, we demonstrate effective sim-to-real transfer and showcase the benefit of training on synthetic and real data. Code and models are made publicly available under https://github.com/leggedrobotics/fdm.
|
||||
</p>
|
||||
<p>
|
||||
<b>Download:</b> <a href="p001.pdf" target="_blank" onclick="_gaq.push(['_trackEvent','rss10/p001.pdf','PDF',this.href]);"><img src="../icon-pdf.png" border=0></a>
|
||||
</p>
|
||||
<p>
|
||||
<b>Bibtex:</b>
|
||||
<pre>
|
||||
@INPROCEEDINGS{RothP-RSS-25,
|
||||
AUTHOR = {Pascal Roth AND Jonas Frey AND Cesar Cadena AND Marco Hutter},
|
||||
TITLE = {{Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation}},
|
||||
BOOKTITLE = {Proceedings of Robotics: Science and Systems},
|
||||
YEAR = {2025},
|
||||
ADDRESS = {LosAngeles, CA, USA},
|
||||
MONTH = {June},
|
||||
DOI = {10.15607/RSS.2025.XXI.001}
|
||||
}
|
||||
|
||||
</pre>
|
||||
</p>
|
||||
|
||||
</div>
|
||||
<div class="analyt">
|
||||
<script type="text/javascript">
|
||||
|
||||
var _gaq = _gaq || [];
|
||||
_gaq.push(['_setAccount', 'UA-24631810-1']);
|
||||
_gaq.push(['_trackPageview']);
|
||||
|
||||
(function() {
|
||||
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
|
||||
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
||||
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
|
||||
})();
|
||||
|
||||
</script>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Vendored
-155
@@ -1,155 +0,0 @@
|
||||
<html>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
||||
<head>
|
||||
<meta name="citation_title" content="Demonstrating ViSafe: Vision-enabled Safety for High-speed Detect and Avoid" />
|
||||
<meta name="citation_author" content="Parv Kapoor" />
|
||||
<meta name="citation_author" content="Ian Higgins" />
|
||||
<meta name="citation_author" content="Nikhil Varma Keetha" />
|
||||
<meta name="citation_author" content="Jay Patrikar" />
|
||||
<meta name="citation_author" content="Brady Moon" />
|
||||
<meta name="citation_author" content="Zelin Ye" />
|
||||
<meta name="citation_author" content="Yao He" />
|
||||
<meta name="citation_author" content="Ivan Cisneros" />
|
||||
<meta name="citation_author" content="Changliu Liu" />
|
||||
<meta name="citation_author" content="Eunsuk Kang" />
|
||||
<meta name="citation_author" content="Sebastian Scherer" />
|
||||
<meta name="citation_publication_date" content="2025/06/21" />
|
||||
<meta name="citation_conference_title" content="Robotics: Science and Systems XXI" />
|
||||
<meta name="citation_isbn" content="979-8-9902848-1-4" />
|
||||
<meta name="citation_volume" content="21" />
|
||||
<meta name="citation_pdf_url" content="http://www.roboticsproceedings.org/rss21/p002.pdf" />
|
||||
<title>Robotics: Science and Systems XXI - Online Proceedings</title>
|
||||
<link type="text/css" rel="stylesheet" href="../style.css">
|
||||
<!--[if IE 5]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie5.css">
|
||||
<![endif]-->
|
||||
<!--[if IE 6]>
|
||||
<link type="text/css" rel="stylesheet" href="../ie6.css">
|
||||
<![endif]-->
|
||||
</head>
|
||||
<body>
|
||||
<div class="menu">
|
||||
<ul>
|
||||
<li class="here"><a class="menu" href="../index.html">Home</a></li>
|
||||
<li class="label">RSS XXII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss22/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XXI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss21/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss20/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss19/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss18/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss17/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XVI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss16/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss15/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss14/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss13/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss12/authors.html">Authors</a></li>
|
||||
<li class="label">RSS XI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss11/authors.html">Authors</a></li>
|
||||
<li class="label">RSS X</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss10/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IX</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss09/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VIII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss08/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VII</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss07/authors.html">Authors</a></li>
|
||||
<li class="label">RSS VI</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss06/authors.html">Authors</a></li>
|
||||
<li class="label">RSS V</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss05/authors.html">Authors</a></li>
|
||||
<li class="label">RSS IV</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss04/authors.html">Authors</a></li>
|
||||
<li class="label">RSS III</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss03/authors.html">Authors</a></li>
|
||||
<li class="label">RSS II</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss02/authors.html">Authors</a></li>
|
||||
<li class="label">RSS I</span></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/index.html">Content</a></li>
|
||||
<li class="menu"><a class="menu" href="../rss01/authors.html">Authors</a></li>
|
||||
<li class="label">Ethics and Malpractice</span></li>
|
||||
<li class="menu"><a class="menu" href="../statement.html">Statement</a></li>
|
||||
<li class="label">Contact</span></li>
|
||||
<li class="last"><a class="menu" href="../contact.html">Webmaster</a></li>
|
||||
<li class="no-style"><center><img border="0" src="../robot-small.jpg"></center>
|
||||
<br><br><br>
|
||||
<center>We are using google analytics for statistics!</center>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="content">
|
||||
|
||||
<h1>Robotics: Science and Systems XXI</h1>
|
||||
<h3>Demonstrating ViSafe: Vision-enabled Safety for High-speed Detect and Avoid</h3>
|
||||
<i>Parv Kapoor, Ian Higgins, Nikhil Varma Keetha, Jay Patrikar, Brady Moon, Zelin Ye, Yao He, Ivan Cisneros, Changliu Liu, Eunsuk Kang, Sebastian Scherer</i><br>
|
||||
<p>
|
||||
<b>Abstract:</b>
|
||||
</p><p style="text-align: justify;">
|
||||
Assured safe-separation is essential for achieving seamless high-density operation of airborne vehicles in a shared airspace. To equip resource-constrained aerial systems with this safety-critical capability, we present ViSafe, a high-speed vision-only airborne collision avoidance system. ViSafe offers a full-stack solution to the Detect and Avoid (DAA) problem by tightly integrating a learning-based edge-AI framework with a custom multi-camera hardware prototype designed under SWaP-C constraints. By leveraging perceptual input-focused control barrier functions (CBF) to design, encode, and enforce safety thresholds, ViSafe can provide provably safe runtime guarantees for self-separation in high-speed aerial operations. We evaluate ViSafe’s performance through an extensive test campaign involving both simulated digital twins and real-world flight scenarios. By independently varying agent types, closure rates, interaction geometries, and environmental conditions (e.g., weather and lighting), we demonstrate that ViSafe consistently ensures self-separation across diverse scenarios. In first-of-its-kind real-world high-speed collision avoidance tests with closure rates reaching 144 km/h, ViSafe sets a new benchmark for vision-only autonomous collision avoidance, establishing a new standard for safety in high-speed aerial navigation.
|
||||
</p>
|
||||
<p>
|
||||
<b>Download:</b> <a href="p002.pdf" target="_blank" onclick="_gaq.push(['_trackEvent','rss10/p002.pdf','PDF',this.href]);"><img src="../icon-pdf.png" border=0></a>
|
||||
</p>
|
||||
<p>
|
||||
<b>Bibtex:</b>
|
||||
<pre>
|
||||
@INPROCEEDINGS{KapoorP-RSS-25,
|
||||
AUTHOR = {Parv Kapoor AND Ian Higgins AND Nikhil Varma Keetha AND Jay Patrikar AND Brady Moon AND Zelin Ye AND Yao He AND Ivan Cisneros AND Changliu Liu AND Eunsuk Kang AND Sebastian Scherer},
|
||||
TITLE = {{Demonstrating ViSafe: Vision-enabled Safety for High-speed Detect and Avoid}},
|
||||
BOOKTITLE = {Proceedings of Robotics: Science and Systems},
|
||||
YEAR = {2025},
|
||||
ADDRESS = {LosAngeles, CA, USA},
|
||||
MONTH = {June},
|
||||
DOI = {10.15607/RSS.2025.XXI.002}
|
||||
}
|
||||
|
||||
</pre>
|
||||
</p>
|
||||
|
||||
</div>
|
||||
<div class="analyt">
|
||||
<script type="text/javascript">
|
||||
|
||||
var _gaq = _gaq || [];
|
||||
_gaq.push(['_setAccount', 'UA-24631810-1']);
|
||||
_gaq.push(['_trackPageview']);
|
||||
|
||||
(function() {
|
||||
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
|
||||
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
||||
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
|
||||
})();
|
||||
|
||||
</script>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Vendored
-65
@@ -1,65 +0,0 @@
|
||||
{
|
||||
"scope": "Six unmodified official HTML fixtures; exact observations from full167 server freeze. No research PDFs.",
|
||||
"receipts": [
|
||||
{
|
||||
"file": "index.html",
|
||||
"url": "https://www.roboticsproceedings.org/rss21/",
|
||||
"final_url": "https://www.roboticsproceedings.org/rss21/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 77753,
|
||||
"sha256": "5d8e530b5ce7184795c14a7dca615f9c1a8b0b189661e0e6e795167b68b8516d",
|
||||
"at": "2026-11-09T09:53:09.121733+00:00"
|
||||
},
|
||||
{
|
||||
"file": "accepted.html",
|
||||
"url": "https://roboticsconference.org/2025/program/papers/",
|
||||
"final_url": "https://roboticsconference.org/2025/program/papers/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 197708,
|
||||
"sha256": "fa1a5bce496a85eca8cec79aed8dbc148cda791fe6859375ba0c294983a18683",
|
||||
"at": "2026-11-09T09:53:10.171450+00:00"
|
||||
},
|
||||
{
|
||||
"file": "demos.html",
|
||||
"url": "https://roboticsconference.org/2025/program/demos/",
|
||||
"final_url": "https://roboticsconference.org/2025/program/demos/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 43542,
|
||||
"sha256": "0b154ed110f6d49ebe2ba87f2c9c498df5fc1ea00bfe508b548522dfe4779698",
|
||||
"at": "2026-11-09T09:53:11.145288+00:00"
|
||||
},
|
||||
{
|
||||
"file": "authorinfo.html",
|
||||
"url": "https://roboticsconference.org/2025/information/authorinfo/",
|
||||
"final_url": "https://roboticsconference.org/2025/information/authorinfo/",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 54508,
|
||||
"sha256": "24d023daf9a074434c6e004001b863dfd260330e523671d614fa120b052927ab",
|
||||
"at": "2026-11-09T09:53:12.170246+00:00"
|
||||
},
|
||||
{
|
||||
"file": "p001.html",
|
||||
"url": "https://www.roboticsproceedings.org/rss21/p001.html",
|
||||
"final_url": "https://www.roboticsproceedings.org/rss21/p001.html",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 9048,
|
||||
"sha256": "973172348096a5cbf3e311bdc1ce33b27447761fa1058a050e611aa17547d821",
|
||||
"at": "2026-11-09T09:53:13.437943+00:00"
|
||||
},
|
||||
{
|
||||
"file": "p002.html",
|
||||
"url": "https://www.roboticsproceedings.org/rss21/p002.html",
|
||||
"final_url": "https://www.roboticsproceedings.org/rss21/p002.html",
|
||||
"http_status": 200,
|
||||
"content_type": "text/html",
|
||||
"bytes": 9335,
|
||||
"sha256": "30b5b68e7ed6a809ca51282c5efd1416513417ba91eba2845f2e73ed18b90ba4",
|
||||
"at": "2026-11-09T09:53:14.697857+00:00"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
"""Pinned T78 evidence, edition-local aliases and an explicitly identified material."""
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
from paper_atlas.collectors.acl import collect_xml, volume_scope
|
||||
from paper_atlas.collectors.acl_historical import event_scope, material_reason
|
||||
from paper_atlas.db.store import Store
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
FIXTURE = Path(__file__).parent/'fixtures/acl-t78.xml'
|
||||
COMMIT = 'a7be623d01c48c61405dec31c871121cdbed5a01'
|
||||
|
||||
|
||||
class AclHistoricalTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.root = ET.parse(FIXTURE).getroot()
|
||||
self.volume = self.root.find('volume')
|
||||
self.meta = self.volume.find('meta')
|
||||
|
||||
def test_frozen_volume_matches_official_snapshot(self):
|
||||
self.assertEqual(hashlib.sha256(FIXTURE.read_bytes()).hexdigest(), 'f712f94cff0cba05f786b7ac31dad2ea9408fde2a17a04531130df623f57dd84')
|
||||
self.assertEqual(len(self.volume.findall('paper')), 36)
|
||||
self.assertEqual(volume_scope(self.meta, '1', 'T78'), ('ACL+TINLAP', ['ACL', 'TINLAP'], 'proceedings_unspecified'))
|
||||
|
||||
def test_unknown_year_collection_or_volume_is_not_globally_aliased(self):
|
||||
for collection, volume in [('T75', '1'), ('T87', '1'), ('T78', '2'), ('C65', '1')]:
|
||||
with self.subTest(collection=collection, volume=volume):
|
||||
self.assertIsNone(event_scope(self.meta, volume, collection))
|
||||
self.assertIsNone(volume_scope(self.meta, volume, collection))
|
||||
self.assertIsNone(volume_scope(self.meta, '1'))
|
||||
|
||||
def test_known_volume_scope_rejects_critical_metadata_drift(self):
|
||||
for field, replacement in [('venue', 'acl'), ('year', '1979'), ('booktitle', 'Another TINLAP'), ('editor/first', 'Another')]:
|
||||
changed = copy.deepcopy(self.meta)
|
||||
changed.find(field).text = replacement
|
||||
with self.subTest(field=field), self.assertRaises(ValueError):
|
||||
event_scope(changed, '1', 'T78')
|
||||
changed = copy.deepcopy(self.meta)
|
||||
ET.SubElement(changed, 'year').text = '1978'
|
||||
with self.assertRaises(ValueError):
|
||||
event_scope(changed, '1', 'T78')
|
||||
|
||||
def test_evidence_retains_original_series_dates_and_role_boundary(self):
|
||||
scope = event_scope(self.meta, '1', 'T78')
|
||||
self.assertEqual(scope['original_venue'], 'TINLAP')
|
||||
self.assertEqual(scope['event_dates'], ['1978-07-25', '1978-07-27'])
|
||||
self.assertEqual([e['pdf_pages'] for e in scope['evidence']], [[9, 10], [3]])
|
||||
self.assertIn('not identified', scope['track_scope_note'])
|
||||
self.assertIn('pending', scope['evidence'][0]['scope'])
|
||||
|
||||
def test_only_exact_session_questions_material_is_excluded(self):
|
||||
paper = copy.deepcopy(self.volume.findall('paper')[-1])
|
||||
self.assertIn('Session question list', material_reason('T78', '1', paper))
|
||||
self.assertIsNone(material_reason('C65', '1', paper))
|
||||
self.assertIsNone(material_reason('T78', '2', paper))
|
||||
paper.set('id', '37')
|
||||
self.assertIsNone(material_reason('T78', '1', paper))
|
||||
paper.set('id', '36')
|
||||
ET.SubElement(paper, 'author')
|
||||
with self.assertRaises(ValueError):
|
||||
material_reason('T78', '1', paper)
|
||||
|
||||
def test_all_authored_source_fields_and_material_receipt_survive_collection(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
manifest, = collect_xml(FIXTURE, temporary, COMMIT)
|
||||
rows = [json.loads(raw) for raw in Path(manifest['file']).read_text().splitlines()]
|
||||
self.assertEqual(manifest['count'], 35)
|
||||
self.assertEqual(manifest['xml_paper_elements'], 36)
|
||||
self.assertEqual(manifest['missing_abstract_count'], 35)
|
||||
self.assertEqual(manifest['excluded_materials'][0]['external_id'], 'T78-1036')
|
||||
for number, row in enumerate(rows, 1):
|
||||
paper = self.volume.findall('paper')[number-1]
|
||||
self.assertEqual(row['external_id'], f'T78-1{number:03d}')
|
||||
self.assertEqual(row['title'], ' '.join(''.join(paper.find('title').itertext()).split()))
|
||||
self.assertEqual(row['metadata']['venue_codes'], ['tinlap'])
|
||||
self.assertEqual(row['metadata']['historical_event_scope']['original_venue'], 'TINLAP')
|
||||
self.assertEqual(row['metadata']['metadata_commit'], COMMIT)
|
||||
self.assertEqual((row['status'], row['track'], row['abstract']), ('accepted', 'proceedings_unspecified', None))
|
||||
|
||||
def test_old_coling_scope_is_unchanged_by_calendar_replacement_evidence(self):
|
||||
meta = ET.fromstring('<meta><venue>coling</venue><year>1965</year><booktitle>COLING 1965</booktitle></meta>')
|
||||
self.assertEqual(volume_scope(meta, '1', 'C65'), ('COLING', ['COLING'], 'main'))
|
||||
self.assertIsNone(event_scope(meta, '1', 'C65'))
|
||||
|
||||
def test_native_ingestion_does_not_alias_other_tinlap_years_or_infer_negative_status(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
manifest, = collect_xml(FIXTURE, root/'source', COMMIT)
|
||||
db = root/'sample.sqlite'
|
||||
store = Store(db)
|
||||
pid, rid = store.record('test_control', 'tinlap1975', 'https://example.org/tinlap1975',
|
||||
'An earlier TINLAP contribution', None, ['Jane Doe'])
|
||||
store.decision(pid, rid, 'TINLAP', 1975, 'main', 'tinlap1975', 'https://example.org/tinlap1975', 'accepted', 'test control')
|
||||
store.conn.commit()
|
||||
store.conn.close()
|
||||
ingest_file(db, manifest['file'])
|
||||
ingest_file(db, manifest['file'])
|
||||
with sqlite3.connect(db) as c:
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0], 36)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM submissions').fetchone()[0], 36)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM decisions').fetchone()[0], 36)
|
||||
self.assertEqual(c.execute("SELECT count(*) FROM decisions WHERE status!='accepted'").fetchone()[0], 0)
|
||||
self.assertIsNone(c.execute("SELECT 1 FROM source_records WHERE external_id='T78-1036'").fetchone())
|
||||
aliases = c.execute('SELECT v.acronym,a.alias FROM venue_aliases a JOIN venues v ON v.id=a.venue_id ORDER BY a.alias').fetchall()
|
||||
self.assertEqual(aliases, [('ACL+TINLAP', 'ACL'), ('ACL+TINLAP', 'TINLAP')])
|
||||
metadata = json.loads(c.execute("SELECT metadata_json FROM source_records WHERE external_id='T78-1001'").fetchone()[0])
|
||||
self.assertEqual(metadata['historical_event_scope']['event_year'], 1978)
|
||||
self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
@@ -1,186 +0,0 @@
|
||||
"""Frozen publisher examples, conflicting evidence and preserved attempt identity."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from paper_atlas.collectors import rss
|
||||
from paper_atlas.ingestion.records import ingest_file
|
||||
|
||||
FIXTURES = Path(__file__).parent/'fixtures/rss2025'
|
||||
OBSERVED = '2026-11-09T10:00:00Z'
|
||||
|
||||
|
||||
def fixture(name):
|
||||
return (FIXTURES/name).read_text()
|
||||
|
||||
|
||||
class RssProceedingsTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.index = rss.parse_index(fixture('index.html'))
|
||||
self.accepted = rss.parse_program(fixture('accepted.html'), 'Accepted Papers', ('ID', 'Session', 'Title', 'Authors'))
|
||||
self.demos = rss.parse_program(fixture('demos.html'), 'Demos', ('ID', 'Location', 'Title', 'Day', 'Time'))
|
||||
|
||||
def paper(self, pid=1, html=None, entry=None):
|
||||
return rss.parse_paper(html or fixture(f'p{pid:03d}.html'), pid, entry or self.index[pid],
|
||||
self.accepted[pid], self.demos.get(pid), {'paper': {'observed_at': OBSERVED}})
|
||||
|
||||
def test_frozen_official_census_and_all_fixture_hashes(self):
|
||||
receipts = json.loads((FIXTURES/'receipts.json').read_bytes())['receipts']
|
||||
self.assertEqual(len(receipts), 6)
|
||||
for r in receipts:
|
||||
data = (FIXTURES/r['file']).read_bytes()
|
||||
self.assertEqual((len(data), hashlib.sha256(data).hexdigest()), (r['bytes'], r['sha256']))
|
||||
self.assertEqual(len(self.index), 163)
|
||||
self.assertEqual(set(self.index), set(self.accepted))
|
||||
self.assertEqual(len(self.demos), 15)
|
||||
rss.validate_program(self.index, self.accepted, self.demos, fixture('authorinfo.html'))
|
||||
self.assertFalse(self.demos[93]['Title'].startswith('Demonstrating'))
|
||||
|
||||
def test_complete_authors_abstract_and_source_metadata_are_preserved(self):
|
||||
row = self.paper()
|
||||
self.assertEqual(row['authors'], ['Pascal Roth', 'Jonas Frey', 'Cesar Cadena', 'Marco Hutter'])
|
||||
self.assertTrue(row['abstract'].startswith('Ensuring safe navigation'))
|
||||
self.assertTrue(row['abstract'].endswith('https://github.com/leggedrobotics/fdm.'))
|
||||
self.assertNotIn('Download:', row['abstract'])
|
||||
self.assertEqual(row['identifiers'], {'doi': '10.15607/rss.2025.xxi.001'})
|
||||
self.assertEqual(row['metadata']['publication_date'], '2025/06/21')
|
||||
self.assertIn('AUTHOR', row['metadata']['original_bibtex'])
|
||||
self.assertEqual(row['status'], 'accepted')
|
||||
self.assertNotIn('decision_date', row)
|
||||
self.assertEqual(row['metadata']['decision_date_precision'], 'not_supplied')
|
||||
|
||||
def test_demo_evidence_is_independent_of_title_and_session_topic(self):
|
||||
row = self.paper(2)
|
||||
self.assertEqual(row['track'], 'demo')
|
||||
self.assertEqual(row['metadata']['demo_program_entry']['ID'], '2')
|
||||
self.assertEqual(self.paper()['track'], 'proceedings_unspecified')
|
||||
self.assertEqual({t['namespace'] for t in row['tags']}, {'rss_proceedings:session', 'rss_proceedings:program_role'})
|
||||
self.assertFalse(any('research_type' in t['namespace'] for t in row['tags']))
|
||||
html = fixture('p001.html').replace('Learned Perceptive Forward Dynamics Model', 'Demonstrating a Learned Model')
|
||||
entry = {**self.index[1], 'title': self.index[1]['title'].replace('Learned Perceptive Forward Dynamics Model', 'Demonstrating a Learned Model')}
|
||||
self.assertEqual(self.paper(html=html, entry=entry)['track'], 'proceedings_unspecified')
|
||||
|
||||
def test_missing_abstract_is_explicit_and_multiple_paragraphs_are_complete(self):
|
||||
html = fixture('p001.html')
|
||||
beginning = html.index('<p style="text-align: justify;">')
|
||||
end = html.index('</p>', beginning)
|
||||
empty = html[:beginning]+'<p style="text-align: justify;">'+html[end:]
|
||||
row = self.paper(html=empty)
|
||||
self.assertIsNone(row['abstract'])
|
||||
self.assertEqual(row['metadata']['missing_abstract_reason'], 'empty_official_abstract_section')
|
||||
multi = html.replace('</p> \n<p> \n<b>Download:', '</p><p>Second abstract paragraph.</p><p><b>Download:', 1)
|
||||
self.assertTrue(self.paper(html=multi)['abstract'].endswith('Second abstract paragraph.'))
|
||||
|
||||
def test_metadata_drift_author_truncation_and_wrong_paper_doi_fail(self):
|
||||
original = fixture('p001.html')
|
||||
for altered in [original.replace('XXI.001}', 'XXI.002}'),
|
||||
original.replace('content="2025/06/21"', 'content="2026/06/21"'),
|
||||
original.replace('<meta name="citation_author" content="Marco Hutter" />', ''),
|
||||
original.replace('<b>Abstract:</b>', '<b>Summary:</b>'),
|
||||
original.replace('rss21/p001.pdf', 'rss21/p002.pdf'),
|
||||
original.replace('citation_volume" content="21', 'citation_volume" content="22')]:
|
||||
with self.subTest(altered=altered[:60]), self.assertRaises(ValueError):
|
||||
self.paper(html=altered)
|
||||
|
||||
def test_duplicate_and_unparsed_index_entries_fail(self):
|
||||
for html in [fixture('index.html').replace('p002.html', 'p001.html'),
|
||||
fixture('index.html')+'<a href="p999.html">Stray paper</a>',
|
||||
fixture('index.html').replace( rss.HEADING, 'A different volume')]:
|
||||
with self.subTest(html=html[:40]), self.assertRaises(ValueError):
|
||||
rss.parse_index(html)
|
||||
|
||||
def test_program_title_id_category_and_header_changes_require_review(self):
|
||||
for change in ['title', 'id', 'category']:
|
||||
index = {k: dict(v) for k, v in self.index.items()}
|
||||
demos = {k: dict(v) for k, v in self.demos.items()}
|
||||
if change == 'title':
|
||||
demos[2]['Title'] = 'Another paper'
|
||||
elif change == 'id':
|
||||
del index[1]
|
||||
else:
|
||||
del demos[93]
|
||||
with self.subTest(change=change), self.assertRaises(ValueError):
|
||||
rss.validate_program(index, self.accepted, demos, fixture('authorinfo.html'))
|
||||
with self.assertRaises(ValueError):
|
||||
rss.parse_program(fixture('demos.html').replace('>Location<', '>Other<'), 'Demos', ('ID', 'Location', 'Title', 'Day', 'Time'))
|
||||
|
||||
def snapshot(self, root):
|
||||
root.mkdir()
|
||||
data = (FIXTURES/'p001.html').read_bytes()
|
||||
(root/'paper.html').write_bytes(data)
|
||||
receipt = {'file': 'paper.html', 'url': self.index[1]['url'], 'final_url': self.index[1]['url'],
|
||||
'http_status': 200, 'content_type': 'text/html', 'bytes': len(data),
|
||||
'sha256': hashlib.sha256(data).hexdigest(), 'at': OBSERVED}
|
||||
(root/'receipts.json').write_text(json.dumps({'receipts': [receipt]}))
|
||||
return receipt
|
||||
|
||||
def test_frozen_receipt_hash_status_redirect_and_path_guards(self):
|
||||
for failure in ['hash', 'status', 'redirect', 'path', 'duplicate', 'symlink']:
|
||||
with self.subTest(failure=failure), tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)/'source'
|
||||
r = self.snapshot(root)
|
||||
receipts = [r]
|
||||
if failure == 'hash': r['sha256'] = '0'*64
|
||||
elif failure == 'status': r['http_status'] = 404
|
||||
elif failure == 'redirect': r['final_url'] = 'https://example.org/p001.html'
|
||||
elif failure == 'path': r['file'] = '../outside.html'
|
||||
elif failure == 'duplicate': receipts.append(dict(r))
|
||||
else:
|
||||
(root.parent/'outside.html').write_bytes((root/'paper.html').read_bytes())
|
||||
(root/'paper.html').unlink()
|
||||
(root/'paper.html').symlink_to(root.parent/'outside.html')
|
||||
(root/'receipts.json').write_text(json.dumps({'receipts': receipts}))
|
||||
with self.assertRaises(ValueError): rss.Pages(snapshots=root).read(self.index[1]['url'])
|
||||
|
||||
def test_unreviewed_edition_and_existing_output_refuse_work(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with self.assertRaises(ValueError): rss.collect(2026, tmp, snapshots=FIXTURES)
|
||||
(Path(tmp)/'rss-2025.json').write_text('{}')
|
||||
with self.assertRaises(FileExistsError): rss.collect(2025, tmp, snapshots=FIXTURES)
|
||||
|
||||
def test_native_roundtrip_is_idempotent_with_same_doi_and_distinct_tracks(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
original = self.paper()
|
||||
dblp = {**original, 'source': 'dblp', 'external_id': 'conf/rss/example', 'metadata': {}, 'tags': []}
|
||||
variant = {**original, 'external_id': '2025/other-track', 'track': 'workshop'}
|
||||
for n, record in enumerate([dblp, original, variant, original]):
|
||||
path = root/f'{n}.jsonl'
|
||||
path.write_text(json.dumps(record)+'\n')
|
||||
path.with_suffix('.manifest.json').write_text(json.dumps({'source': record['source'], 'list_url': rss.ROOT, 'count': 1}))
|
||||
ingest_file(root/'sample.sqlite', path)
|
||||
with sqlite3.connect(root/'sample.sqlite') as c:
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM papers').fetchone()[0], 1)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM source_records').fetchone()[0], 3)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM submissions').fetchone()[0], 2)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM decisions').fetchone()[0], 2)
|
||||
self.assertEqual(c.execute('SELECT count(*) FROM submission_sources').fetchone()[0], 3)
|
||||
self.assertEqual(c.execute('PRAGMA foreign_key_check').fetchall(), [])
|
||||
|
||||
def test_two_real_papers_collection_commits_evidence_only_after_success(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
index = {pid: self.index[pid] for pid in [1, 2]}
|
||||
accepted = {pid: self.accepted[pid] for pid in [1, 2]}
|
||||
demos = {2: self.demos[2]}
|
||||
with (patch.object(rss, 'EXPECTED', 2), patch.object(rss, 'DEMO_IDS', {2}),
|
||||
patch.object(rss, 'parse_index', return_value=index),
|
||||
patch.object(rss, 'parse_program', side_effect=[accepted, demos])):
|
||||
result = rss.collect(2025, root/'out', snapshots=FIXTURES)
|
||||
self.assertEqual((result['records'], len(result['files'])), (2, 2))
|
||||
self.assertFalse(result['database_written'])
|
||||
self.assertFalse(result['pdfs_downloaded'])
|
||||
self.assertEqual(len(result['source_evidence']), 6)
|
||||
for item in result['source_evidence']:
|
||||
self.assertEqual(hashlib.sha256((root/'out'/item['file']).read_bytes()).hexdigest(), item['sha256'])
|
||||
for item in result['files']:
|
||||
self.assertEqual(item['missing_abstract_count'], 0)
|
||||
self.assertEqual(hashlib.sha256(Path(item['file']).read_bytes()).hexdigest(), item['file_sha256'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user