77 lines
4.4 KiB
Markdown
77 lines
4.4 KiB
Markdown
# Early WWW archive collector
|
|
|
|
`paper-atlas collect www-archive` is a maintained metadata adapter. It reads official
|
|
IW3C2 archive HTML, keeps exact source hashes, and separates importable rows from
|
|
program entries that still need review. It fetches no PDF, follows no paper-body or
|
|
reference link, and makes no database connection.
|
|
|
|
For a fresh source snapshot on the server:
|
|
|
|
```sh
|
|
paper-atlas collect www-archive \
|
|
--http-cache /data/paper-index/cache/www-archive \
|
|
--source-cache /data/paper-index/collected/www-archive-source-v2 \
|
|
--out /data/paper-index/collected/www-archive-normalized-v2
|
|
```
|
|
|
|
For reproducible offline replay of the already verified snapshot:
|
|
|
|
```sh
|
|
paper-atlas collect www-archive \
|
|
--source-cache /data/paper-index/collected/www-early-archive-source-v1 \
|
|
--source-cache-sha256 c9388ccef44486e0684c9ee9e8620a465cdc4574e6bc9ea9391d743e01a3312c \
|
|
--out /data/paper-index/collected/www-archive-normalized-v1
|
|
```
|
|
|
|
Use a new output directory for another batch. Existing snapshots and normalized
|
|
batches are never overwritten. The manifest hash binds every source URL, byte count,
|
|
response hash and cache filename. Plain files must remain directly within the cache;
|
|
symlinks, escaping paths, duplicate URLs, non-200 responses and altered bytes fail.
|
|
The command uses UTF-8 where valid and explicitly records an ISO-8859-1 fallback for
|
|
older pages; original bytes remain authoritative for encoding corrections.
|
|
|
|
## Evidence and outputs
|
|
|
|
- WWW3 Spring 1995 papers use `event_key=www-1995-spring`, `track=main`. The author,
|
|
session and abstract indexes must have the same nonempty entry-ID set. The ID is
|
|
local to this exact event and proceedings section, never a DOI or OpenReview ID.
|
|
- Posters remain `track=poster`. An abstract without a linked author-index entry
|
|
stays in review even when another index has a placeholder `#` link for it.
|
|
- Paper URLs must agree between author and session indexes. A malformed whitespace
|
|
URL in the abstract list may use the exact event/entry-ID match only when both
|
|
independent indexes corroborate the same clean URL. The original href is retained.
|
|
- Canonical names come from the explicitly labeled author index. The source lists
|
|
them alphabetically; this is **not** original byline order. Both the raw index names
|
|
and the full abstract-page byline/affiliation text remain in metadata. No person ID
|
|
or cross-source identity is inferred from a name. Conflicting titles or missing
|
|
abstracts stay in review. Topic keywords are not treated as an abstract.
|
|
- WWW1 1994 formal presentations and WWW2 1994 mixed categories remain review
|
|
candidates. The first list contains uncertain business-track annotations and a
|
|
count discrepancy; the second contains panels, tutorials, products and other
|
|
activities. Neither presentation listing nor a missing full-text link establishes
|
|
a regular-paper acceptance or a rejection. Raw category/title/name claims are kept.
|
|
|
|
The main and poster JSONL files have independent manifests with explicit event key,
|
|
track, count and SHA256. Their status stays `partial`: a verified linked subset does
|
|
not prove complete event coverage. `review-candidates.jsonl` contains nested
|
|
candidates and reasons, with no importable decision status. It is excluded from
|
|
`import-queue.json`. `report.json` accounts for ready/held records and every source
|
|
used. Newlines delimit JSONL; Unicode U+2028/U+2029 inside metadata are not line breaks.
|
|
The output is published only after all required source and scope checks pass.
|
|
|
|
## Before importing
|
|
|
|
This independent collector branch starts from main; it does not include or silently
|
|
install the event-aware writer. Its output requires the reviewed integrated runtime
|
|
from PR31 (or a main revision containing that implementation). The production event
|
|
schema migration has been independently accepted, preserving existing edition1212
|
|
and its 59 Fall submissions; that is not an import of the Spring proceedings.
|
|
|
|
Inspect held entries and original author names, preflight every ready row against
|
|
current PostgreSQL stable identifiers and candidate identities, and verify the
|
|
source/year/track/event coverage before running `import records`. Do not automatically
|
|
run this queue against an older writer. Preserve extended versions and independent
|
|
submission/decision evidence, and keep unresolved identities held. A normalization
|
|
report or passing tests alone do not establish production import, complete historical
|
|
coverage or final global deduplication.
|