Add paper metadata indexing and public submission collectors #1

Merged

Zayden-Jung merged 20 commits from codex/metadata-index into main 2026-11-06 09:21:33 +00:00

Conversation 0 Commits 20 Files Changed 42

+4623 -9

codex-bot commented 2026-11-05 19:02:23 +00:00

Collaborator

Copy Link

Copy Source

Paper Atlas is a long-term, on-demand research tool for querying paper metadata and public submission histories. This PR delivers the indexing and collection code; the full dataset is still being assembled.

Implemented:

• SQLite paper entities, retained source records, FTS queries, source-separated/custom tags, identifier-based merging and merge audit; checkpointed arXiv ZIP ingestion.
• Official collectors for PMLR, COLM, NeurIPS, ACL Anthology, ICLR public programs, CVF and ECVA, with year/track manifests and explicit gaps.
• Checksum-verified fixed snapshots and bounded download resumes. The public ICLR snapshot parser retains only public submissions and signed formal decisions, preserves unknowns and unmatched evidence, and excludes review text and author email fields.
• Same-year ICLR program/snapshot audits using real forum identities, with status/text differences and hash-checked backfill queues. Submission/event provenance survives source mirrors; contradictory identities fail instead of forcing merges.
• A streaming DBLP conference catalogue using a checksum-verified local DTD. Official dated DTD names and source encoding are supported. Multiple publication-volume fields remain reviewable without double-counting. Candidate families include workshops and historical names; they are not automatically assigned main-track acceptance labels.
• A coverage report that lists all 41 candidate venues, including targets with no imported records, and separates source-file validation from full historical coverage.
• Serialized maintenance with per-file transactions, foreign-key verification, and no restart of an already-complete arXiv batch.

Validation: ces-control remote CI run 902 passed all 96 unittest cases and query CLI checks at 16f428c. Actual recent ICLR snapshots also passed source checksum, declared inventory, public-evidence and same-edition official-program cross-checks. Tests cover repeat imports, conflict rollback, source evidence, track isolation, dated DTD/entity handling, gzip truncation, multi-volume counts and zero-coverage reporting.

Remaining dataset work: other target venue adapters and historical gaps, full-source ingestion, independent coverage checks, and the mandatory final cross-source duplicate/integrity audit. arXiv ingestion reached EOF; recent ICLR ingestion and broader conference collection are progressing through the single-writer pipeline. The candidate registry is a coverage plan, not a completed dataset or an objective ranking.

Base: main; head: codex/metadata-index. This is a reviewable code milestone. Waiting for user manual merge; no automatic merge is enabled.

Paper Atlas is a long-term, on-demand research tool for querying paper metadata and public submission histories. This PR delivers the indexing and collection code; the full dataset is still being assembled. Implemented: - SQLite paper entities, retained source records, FTS queries, source-separated/custom tags, identifier-based merging and merge audit; checkpointed arXiv ZIP ingestion. - Official collectors for PMLR, COLM, NeurIPS, ACL Anthology, ICLR public programs, CVF and ECVA, with year/track manifests and explicit gaps. - Checksum-verified fixed snapshots and bounded download resumes. The public ICLR snapshot parser retains only public submissions and signed formal decisions, preserves unknowns and unmatched evidence, and excludes review text and author email fields. - Same-year ICLR program/snapshot audits using real forum identities, with status/text differences and hash-checked backfill queues. Submission/event provenance survives source mirrors; contradictory identities fail instead of forcing merges. - A streaming DBLP conference catalogue using a checksum-verified local DTD. Official dated DTD names and source encoding are supported. Multiple publication-volume fields remain reviewable without double-counting. Candidate families include workshops and historical names; they are not automatically assigned main-track acceptance labels. - A coverage report that lists all 41 candidate venues, including targets with no imported records, and separates source-file validation from full historical coverage. - Serialized maintenance with per-file transactions, foreign-key verification, and no restart of an already-complete arXiv batch. Validation: ces-control remote CI run 902 passed all 96 unittest cases and query CLI checks at 16f428c. Actual recent ICLR snapshots also passed source checksum, declared inventory, public-evidence and same-edition official-program cross-checks. Tests cover repeat imports, conflict rollback, source evidence, track isolation, dated DTD/entity handling, gzip truncation, multi-volume counts and zero-coverage reporting. Remaining dataset work: other target venue adapters and historical gaps, full-source ingestion, independent coverage checks, and the mandatory final cross-source duplicate/integrity audit. arXiv ingestion reached EOF; recent ICLR ingestion and broader conference collection are progressing through the single-writer pipeline. The candidate registry is a coverage plan, not a completed dataset or an objective ranking. Base: main; head: codex/metadata-index. This is a reviewable code milestone. Waiting for user manual merge; no automatic merge is enabled.

codex-bot added 1 commit 2026-11-05 19:02:24 +00:00

Add provenance-aware paper metadata storage and PMLR collection 4d018a14c3

codex-bot added 1 commit 2026-11-05 19:12:56 +00:00

Collect explicit public ICLR decisions and refresh metadata indexes 4a4272219e

codex-bot added 1 commit 2026-11-05 22:11:57 +00:00

Fetch old PMLR metadata without entire PDF archives 66b98fa375

codex-bot added 1 commit 2026-11-06 03:35:10 +00:00

Verify historical conference metadata and expose consistent paper statistics 5d9d13a272

codex-bot added 1 commit 2026-11-06 03:40:48 +00:00

Index per-source tag refreshes for scalable bulk ingestion a3539be4b1

codex-bot changed title from WIP: Build paper metadata index and conference ingestion to WIP: Build provenance-aware paper index and conference history 2026-11-06 03:41:25 +00:00

codex-bot added 1 commit 2026-11-06 04:04:27 +00:00

Collect official COLM and NeurIPS abstracts with track-aware coverage ad9967c1b2

codex-bot added 1 commit 2026-11-06 04:35:34 +00:00

Add authoritative ACL metadata and protect publication identities adb4bf8f27

codex-bot added 1 commit 2026-11-06 04:58:25 +00:00

Collect official ICLR programs with presentation identity checks dba943f03e

codex-bot added 1 commit 2026-11-06 05:01:20 +00:00

Validate publication identities and completeness before bulk imports 1e6d64f541

codex-bot changed title from WIP: Build provenance-aware paper index and conference history to Draft: Build a paper metadata and submission history index 2026-11-06 05:02:19 +00:00

codex-bot changed title from Draft: Build a paper metadata and submission history index to WIP: Build a paper metadata and submission history index 2026-11-06 05:03:33 +00:00

codex-bot added 1 commit 2026-11-06 05:15:44 +00:00

Collect CVPR ICCV and ECCV official abstract metadata 396b7dbaf2

codex-bot added 1 commit 2026-11-06 05:30:04 +00:00

Download fixed metadata snapshots with checksum and resume safeguards a7a0729df2

codex-bot added 1 commit 2026-11-06 06:06:36 +00:00

Preserve canonical submission identities and decision source evidence 15bb4920c6

codex-bot added 1 commit 2026-11-06 07:03:05 +00:00

Retry transient metadata requests and handle legacy empty HTML attributes 1213369824

codex-bot added 1 commit 2026-11-06 07:50:31 +00:00

Extract public ICLR snapshot submissions and signed decision evidence 53a83a85a6

codex-bot added 1 commit 2026-11-06 08:32:16 +00:00

Reject contradictory submission identities and correct snapshot fixtures 8bfe52df68

codex-bot changed title from WIP: Build a paper metadata and submission history index to Add paper metadata indexing and public submission collectors 2026-11-06 08:34:11 +00:00

codex-bot added 1 commit 2026-11-06 08:39:24 +00:00

Audit public ICLR snapshots against official same-year programs a40efaa77c

codex-bot added 1 commit 2026-11-06 08:43:24 +00:00

Inventory pinned DBLP conference metadata before scope classification 4077cb4666

codex-bot added 1 commit 2026-11-06 08:47:13 +00:00

Resolve the official dated DBLP DTD locally and preserve source encoding f9bdab49f2

codex-bot added 1 commit 2026-11-06 08:53:37 +00:00

Preserve multi-volume DBLP metadata without inflating record counts 827ba17a99

codex-bot added 1 commit 2026-11-06 09:01:44 +00:00

Report every target venue and expose missing historical coverage 16f428cae9

Zayden-Jung merged commit dc7be9d6a4 into main 2026-11-06 09:21:33 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

codex-bot yilin-codex-bot Zayden-Jung

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Zayden-Jung/paper-atlas#1