Files
paper-atlas/docs/www-archive.md
T
codex-bot 9040649320
Metadata integrity checks / metadata (push) Successful in 1m10s
Preserve full source listing totals beside ready archive subsets
2026-11-08 21:54:43 +08:00

5.6 KiB

Early WWW archive collector

paper-atlas collect www-archive is a maintained metadata adapter. It reads official IW3C2 archive HTML, keeps exact source hashes, and separates importable rows from program entries that still need review. It fetches no PDF, follows no paper-body or reference link, and makes no database connection.

For a fresh source snapshot on the server:

paper-atlas collect www-archive \
  --http-cache /data/paper-index/cache/www-archive \
  --source-cache /data/paper-index/collected/www-archive-source-v2 \
  --out /data/paper-index/collected/www-archive-normalized-v2

For reproducible offline replay of the already verified snapshot:

paper-atlas collect www-archive \
  --source-cache /data/paper-index/collected/www-early-archive-source-v1 \
  --source-cache-sha256 c9388ccef44486e0684c9ee9e8620a465cdc4574e6bc9ea9391d743e01a3312c \
  --out /data/paper-index/collected/www-archive-normalized-v1

Use a new output directory for another batch. Existing snapshots and normalized batches are never overwritten. The manifest hash binds every source URL, byte count, response hash and cache filename. Plain files must remain directly within the cache; symlinks, escaping paths, duplicate URLs, non-200 responses and altered bytes fail. The command uses UTF-8 where valid and explicitly records an ISO-8859-1 fallback for older pages; original bytes remain authoritative for encoding corrections.

Evidence and outputs

  • WWW3 Spring 1995 papers use event_key=www-1995-spring, track=main. The author, session and abstract indexes must have the same nonempty entry-ID set. The ID is local to this exact event and proceedings section, never a DOI or OpenReview ID.
  • Posters remain track=poster. An abstract without a linked author-index entry stays in review even when another index has a placeholder # link for it.
  • Paper URLs must agree between author and session indexes. A malformed whitespace URL in the abstract list may use the exact event/entry-ID match only when both independent indexes corroborate the same clean URL. The original href is retained.
  • Canonical names come from the explicitly labeled author index. The source lists them alphabetically; this is not original byline order. Both the raw index names and the full abstract-page byline/affiliation text remain in metadata. No person ID or cross-source identity is inferred from a name. Conflicting titles or missing abstracts stay in review. Topic keywords are not treated as an abstract.
  • WWW1 1994 formal presentations and WWW2 1994 mixed categories remain review candidates. The first list contains uncertain business-track annotations and a count discrepancy; the second contains panels, tutorials, products and other activities. Neither presentation listing nor a missing full-text link establishes a regular-paper acceptance or a rejection. Raw category/title/name claims are kept.

The main and poster JSONL files have independent manifests with explicit event key, track, count and SHA256. source_listing_count records the full recognized abstract listing, while held_entry_count records excluded entries; neither is a ready-row count. Their status stays partial: a verified linked subset does not prove complete event coverage. review-candidates.jsonl contains nested candidates and reasons, with no importable decision status. It is excluded from import-queue.json. report.json accounts for ready/held records and every source used. Newlines delimit JSONL; Unicode U+2028/U+2029 inside metadata are not line breaks. The output is published only after all required source and scope checks pass.

Before importing

This independent collector branch starts from main; it does not include or silently install the event-aware writer. Its output requires the reviewed integrated runtime from PR31 (or a main revision containing that implementation). The production event schema migration has been independently accepted, preserving existing edition1212 and its 59 Fall submissions; that is not an import of the Spring proceedings.

Inspect held entries and original author names, preflight every ready row against current PostgreSQL stable identifiers and candidate identities, and verify the source/year/track/event coverage before running import records. When a ready subset excludes held entries, preserve the independently known listing total in the existing incremental-coverage protocol: bind a freshly checked parent count and expected count, and set the full expected count from this manifest. A new source uses previous count 0 and previous expected count null; an existing source requires its actual current values. These live parent values are not guessed by the offline collector. Do not automatically run this queue against an older writer. Preserve extended versions and independent submission/decision evidence, and keep unresolved identities held. A normalization report or passing tests alone do not establish production import, complete historical coverage or final global deduplication.

Initial fixed-cache replay (64ced7a, CI1057): 38 main papers and38 posters ready; 372 entries held (50 first-conference fields,307 mixed program entries,15 posters). Independent report44dfd070 verified source hashes,counts and author evidence, finding one existing exact-title candidate. That publication is explicitly linked to arXiv1811.12161 by its WWW1995/journal-reference declaration; arXiv1810.07232 is a different1997 invited chapter and remains separate despite the overlap note. This is a preparation observation, not evidence of completed production import or final global deduplication.