Package reproducible early WWW metadata collection with explicit review holds
Metadata integrity checks / metadata (push) Successful in 1m10s
Metadata integrity checks / metadata (push) Successful in 1m10s
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
# Early WWW archive collector
|
||||
|
||||
`paper-atlas collect www-archive` is a maintained metadata adapter. It reads official
|
||||
IW3C2 archive HTML, keeps exact source hashes, and separates importable rows from
|
||||
program entries that still need review. It fetches no PDF, follows no paper-body or
|
||||
reference link, and makes no database connection.
|
||||
|
||||
For a fresh source snapshot on the server:
|
||||
|
||||
```sh
|
||||
paper-atlas collect www-archive \
|
||||
--http-cache /data/paper-index/cache/www-archive \
|
||||
--source-cache /data/paper-index/collected/www-archive-source-v2 \
|
||||
--out /data/paper-index/collected/www-archive-normalized-v2
|
||||
```
|
||||
|
||||
For reproducible offline replay of the already verified snapshot:
|
||||
|
||||
```sh
|
||||
paper-atlas collect www-archive \
|
||||
--source-cache /data/paper-index/collected/www-early-archive-source-v1 \
|
||||
--source-cache-sha256 c9388ccef44486e0684c9ee9e8620a465cdc4574e6bc9ea9391d743e01a3312c \
|
||||
--out /data/paper-index/collected/www-archive-normalized-v1
|
||||
```
|
||||
|
||||
Use a new output directory for another batch. Existing snapshots and normalized
|
||||
batches are never overwritten. The manifest hash binds every source URL, byte count,
|
||||
response hash and cache filename. Plain files must remain directly within the cache;
|
||||
symlinks, escaping paths, duplicate URLs, non-200 responses and altered bytes fail.
|
||||
The command uses UTF-8 where valid and explicitly records an ISO-8859-1 fallback for
|
||||
older pages; original bytes remain authoritative for encoding corrections.
|
||||
|
||||
## Evidence and outputs
|
||||
|
||||
- WWW3 Spring 1995 papers use `event_key=www-1995-spring`, `track=main`. The author,
|
||||
session and abstract indexes must have the same nonempty entry-ID set. The ID is
|
||||
local to this exact event and proceedings section, never a DOI or OpenReview ID.
|
||||
- Posters remain `track=poster`. An abstract without a linked author-index entry
|
||||
stays in review even when another index has a placeholder `#` link for it.
|
||||
- Paper URLs must agree between author and session indexes. A malformed whitespace
|
||||
URL in the abstract list may use the exact event/entry-ID match only when both
|
||||
independent indexes corroborate the same clean URL. The original href is retained.
|
||||
- Canonical names come from the explicitly labeled author index. The source lists
|
||||
them alphabetically; this is **not** original byline order. Both the raw index names
|
||||
and the full abstract-page byline/affiliation text remain in metadata. No person ID
|
||||
or cross-source identity is inferred from a name. Conflicting titles or missing
|
||||
abstracts stay in review. Topic keywords are not treated as an abstract.
|
||||
- WWW1 1994 formal presentations and WWW2 1994 mixed categories remain review
|
||||
candidates. The first list contains uncertain business-track annotations and a
|
||||
count discrepancy; the second contains panels, tutorials, products and other
|
||||
activities. Neither presentation listing nor a missing full-text link establishes
|
||||
a regular-paper acceptance or a rejection. Raw category/title/name claims are kept.
|
||||
|
||||
The main and poster JSONL files have independent manifests with explicit event key,
|
||||
track, count and SHA256. Their status stays `partial`: a verified linked subset does
|
||||
not prove complete event coverage. `review-candidates.jsonl` contains nested
|
||||
candidates and reasons, with no importable decision status. It is excluded from
|
||||
`import-queue.json`. `report.json` accounts for ready/held records and every source
|
||||
used. Newlines delimit JSONL; Unicode U+2028/U+2029 inside metadata are not line breaks.
|
||||
The output is published only after all required source and scope checks pass.
|
||||
|
||||
## Before importing
|
||||
|
||||
This independent collector branch starts from main; it does not include or silently
|
||||
install the event-aware writer. Its output requires the reviewed integrated runtime
|
||||
from PR31 (or a main revision containing that implementation). The production event
|
||||
schema migration has been independently accepted, preserving existing edition1212
|
||||
and its 59 Fall submissions; that is not an import of the Spring proceedings.
|
||||
|
||||
Inspect held entries and original author names, preflight every ready row against
|
||||
current PostgreSQL stable identifiers and candidate identities, and verify the
|
||||
source/year/track/event coverage before running `import records`. Do not automatically
|
||||
run this queue against an older writer. Preserve extended versions and independent
|
||||
submission/decision evidence, and keep unresolved identities held. A normalization
|
||||
report or passing tests alone do not establish production import, complete historical
|
||||
coverage or final global deduplication.
|
||||
Reference in New Issue
Block a user