Files
paper-atlas/docs/www-archive.md
T
2026-11-08 21:41:02 +08:00

4.4 KiB

Early WWW archive collector

paper-atlas collect www-archive is a maintained metadata adapter. It reads official IW3C2 archive HTML, keeps exact source hashes, and separates importable rows from program entries that still need review. It fetches no PDF, follows no paper-body or reference link, and makes no database connection.

For a fresh source snapshot on the server:

paper-atlas collect www-archive \
  --http-cache /data/paper-index/cache/www-archive \
  --source-cache /data/paper-index/collected/www-archive-source-v2 \
  --out /data/paper-index/collected/www-archive-normalized-v2

For reproducible offline replay of the already verified snapshot:

paper-atlas collect www-archive \
  --source-cache /data/paper-index/collected/www-early-archive-source-v1 \
  --source-cache-sha256 c9388ccef44486e0684c9ee9e8620a465cdc4574e6bc9ea9391d743e01a3312c \
  --out /data/paper-index/collected/www-archive-normalized-v1

Use a new output directory for another batch. Existing snapshots and normalized batches are never overwritten. The manifest hash binds every source URL, byte count, response hash and cache filename. Plain files must remain directly within the cache; symlinks, escaping paths, duplicate URLs, non-200 responses and altered bytes fail. The command uses UTF-8 where valid and explicitly records an ISO-8859-1 fallback for older pages; original bytes remain authoritative for encoding corrections.

Evidence and outputs

  • WWW3 Spring 1995 papers use event_key=www-1995-spring, track=main. The author, session and abstract indexes must have the same nonempty entry-ID set. The ID is local to this exact event and proceedings section, never a DOI or OpenReview ID.
  • Posters remain track=poster. An abstract without a linked author-index entry stays in review even when another index has a placeholder # link for it.
  • Paper URLs must agree between author and session indexes. A malformed whitespace URL in the abstract list may use the exact event/entry-ID match only when both independent indexes corroborate the same clean URL. The original href is retained.
  • Canonical names come from the explicitly labeled author index. The source lists them alphabetically; this is not original byline order. Both the raw index names and the full abstract-page byline/affiliation text remain in metadata. No person ID or cross-source identity is inferred from a name. Conflicting titles or missing abstracts stay in review. Topic keywords are not treated as an abstract.
  • WWW1 1994 formal presentations and WWW2 1994 mixed categories remain review candidates. The first list contains uncertain business-track annotations and a count discrepancy; the second contains panels, tutorials, products and other activities. Neither presentation listing nor a missing full-text link establishes a regular-paper acceptance or a rejection. Raw category/title/name claims are kept.

The main and poster JSONL files have independent manifests with explicit event key, track, count and SHA256. Their status stays partial: a verified linked subset does not prove complete event coverage. review-candidates.jsonl contains nested candidates and reasons, with no importable decision status. It is excluded from import-queue.json. report.json accounts for ready/held records and every source used. Newlines delimit JSONL; Unicode U+2028/U+2029 inside metadata are not line breaks. The output is published only after all required source and scope checks pass.

Before importing

This independent collector branch starts from main; it does not include or silently install the event-aware writer. Its output requires the reviewed integrated runtime from PR31 (or a main revision containing that implementation). The production event schema migration has been independently accepted, preserving existing edition1212 and its 59 Fall submissions; that is not an import of the Spring proceedings.

Inspect held entries and original author names, preflight every ready row against current PostgreSQL stable identifiers and candidate identities, and verify the source/year/track/event coverage before running import records. Do not automatically run this queue against an older writer. Preserve extended versions and independent submission/decision evidence, and keep unresolved identities held. A normalization report or passing tests alone do not establish production import, complete historical coverage or final global deduplication.