112 lines
7.6 KiB
Markdown
112 lines
7.6 KiB
Markdown
# Early WWW archive collector
|
|
|
|
`paper-atlas collect www-archive` is a maintained metadata adapter. It reads official
|
|
IW3C2 archive HTML, keeps exact source hashes, and separates importable rows from
|
|
program entries that still need review. It fetches no PDF, follows no paper-body or
|
|
reference link, and makes no database connection.
|
|
|
|
For a fresh source snapshot on the server:
|
|
|
|
```sh
|
|
paper-atlas collect www-archive \
|
|
--http-cache /data/paper-index/cache/www-archive \
|
|
--source-cache /data/paper-index/collected/www-archive-source-v2 \
|
|
--out /data/paper-index/collected/www-archive-normalized-v2
|
|
```
|
|
|
|
For reproducible offline replay of the already verified snapshot:
|
|
|
|
```sh
|
|
paper-atlas collect www-archive \
|
|
--source-cache /data/paper-index/collected/www-early-archive-source-v1 \
|
|
--source-cache-sha256 c9388ccef44486e0684c9ee9e8620a465cdc4574e6bc9ea9391d743e01a3312c \
|
|
--out /data/paper-index/collected/www-archive-normalized-v1
|
|
```
|
|
|
|
Use a new output directory for another batch. Existing snapshots and normalized
|
|
batches are never overwritten. The manifest hash binds every source URL, byte count,
|
|
response hash and cache filename. Plain files must remain directly within the cache;
|
|
symlinks, escaping paths, duplicate URLs, non-200 responses and altered bytes fail.
|
|
The command uses UTF-8 where valid and explicitly records an ISO-8859-1 fallback for
|
|
older pages; original bytes remain authoritative for encoding corrections.
|
|
|
|
## Evidence and outputs
|
|
|
|
- WWW3 Spring 1995 papers use `event_key=www-1995-spring`, `track=main`. The author,
|
|
session and abstract indexes must have the same nonempty entry-ID set. The ID is
|
|
local to this exact event and proceedings section, never a DOI or OpenReview ID.
|
|
- Posters remain `track=poster`. An abstract without a linked author-index entry
|
|
stays in review even when another index has a placeholder `#` link for it.
|
|
- Paper URLs must agree between author and session indexes. A malformed whitespace
|
|
URL in the abstract list may use the exact event/entry-ID match only when both
|
|
independent indexes corroborate the same clean URL. The original href is retained.
|
|
- Canonical names come from the explicitly labeled author index. The source lists
|
|
them alphabetically; this is **not** original byline order. Both the raw index names
|
|
and the full abstract-page byline/affiliation text remain in metadata. No person ID
|
|
or cross-source identity is inferred from a name. Conflicting titles or missing
|
|
abstracts stay in review. Topic keywords are not treated as an abstract.
|
|
- WWW1 1994 formal presentations and WWW2 1994 mixed categories remain review
|
|
candidates. The first list contains uncertain business-track annotations and a
|
|
count discrepancy; the second contains panels, tutorials, products and other
|
|
activities. Neither presentation listing nor a missing full-text link establishes
|
|
a regular-paper acceptance or a rejection. Raw category/title/name claims are kept.
|
|
|
|
The main and poster JSONL files have independent manifests with explicit event key,
|
|
track, count and SHA256. `source_listing_count` records the full recognized abstract
|
|
listing, while `held_entry_count` records excluded entries; neither is a ready-row
|
|
count. Their status stays `partial`: a verified linked subset does
|
|
not prove complete event coverage. `review-candidates.jsonl` contains nested
|
|
candidates and reasons, with no importable decision status. It is excluded from
|
|
`import-queue.json`. `report.json` accounts for ready/held records and every source
|
|
used. Newlines delimit JSONL; Unicode U+2028/U+2029 inside metadata are not line breaks.
|
|
The output is published only after all required source and scope checks pass.
|
|
|
|
## Before importing
|
|
|
|
This independent collector branch starts from main; it does not include or silently
|
|
install the event-aware writer. Its output requires the reviewed integrated runtime
|
|
from PR31 (or a main revision containing that implementation). The production event
|
|
schema migration has been independently accepted, preserving existing edition1212
|
|
and its 59 Fall submissions; that is not an import of the Spring proceedings.
|
|
|
|
Inspect held entries and original author names, preflight every ready row against
|
|
current PostgreSQL stable identifiers and candidate identities, and verify the
|
|
source/year/track/event coverage before running `import records`. When a ready subset
|
|
excludes held entries, preserve the independently known listing total in the existing
|
|
incremental-coverage protocol: bind a freshly checked parent count and expected count,
|
|
and set the full expected count from this manifest. A new source uses previous count
|
|
0 and previous expected count null; an existing source requires its actual current
|
|
values. These live parent values are not guessed by the offline collector. Do not automatically
|
|
run this queue against an older writer. Preserve extended versions and independent
|
|
submission/decision evidence, and keep unresolved identities held. A normalization
|
|
report or passing tests alone do not establish production import, complete historical
|
|
coverage or final global deduplication.
|
|
|
|
Initial fixed-cache replay (64ced7a, CI1057): 38 main papers and38 posters ready;
|
|
372 entries held (50 first-conference fields,307 mixed program entries,15 posters).
|
|
Independent report44dfd070 verified source hashes,counts and author evidence, finding
|
|
one existing exact-title candidate. That publication is explicitly linked to
|
|
arXiv1811.12161 by its WWW1995/journal-reference declaration; arXiv1810.07232 is a
|
|
different1997 invited chapter and remains separate despite the overlap note. This
|
|
is a preparation observation, not evidence of completed production import or final
|
|
global deduplication.
|
|
|
|
## 以单篇官方页面核实待处理 poster
|
|
|
|
作者索引漏链接、摘要目录使用短标题或标题存在拼写差异时，不放宽全局标题匹配。可以先取得单篇官方 HTML，人工核对 H1、署名区与原目录同一 entry URL，再提供带 SHA256 的逐条核查文件：
|
|
|
|
```sh
|
|
paper-atlas collect www-archive \
|
|
--source-cache /data/paper-index/collected/www-poster-reviewed-source-v2 \
|
|
--source-cache-sha256 8a65562f3490497cada2919ec77827134beabd5ec9c3690dec3625e2318aa54e \
|
|
--reviewed-posters docs/coverage/www-1995-posters-reviewed-2026-11-09.json \
|
|
--reviewed-posters-sha256 <核查文件的SHA256> \
|
|
--out <新的输出目录>
|
|
```
|
|
|
|
[1995 年15条poster的核查输入](coverage/www-1995-posters-reviewed-2026-11-09.json) 绑定61页来源快照：原46页和补取的15页单篇HTML。程序验证输入文件及快照哈希、单篇页面哈希、完全一致的 canonical H1、原摘要目录中的同一URL/entry ID及作者证据，不通过相似标题猜身份。规范标题采用单篇H1，摘要目录标题及作者索引标题均在 metadata 中保留。
|
|
|
|
`author_source=canonical_byline` 表示名字出现在H1之后、摘要/关键词或下一节标题之前的署名区域，由逐条核查明确选出，不从全文引用或机构猜作者；保留原始缩写、顺序、重音和拼写。只有 `author_source=author_index` 时使用原作者索引顺序并逐项核对；05的单篇页面写的是联系人，作者身份使用原作者索引证据，不由联系人角色推断。核查输入是审阅后的数据声明，程序的字符串/哈希检查不代替人工识别署名语义。
|
|
|
|
38和05的来源没有可明确引用的摘要。保留 `abstract=null` 和逐条缺失原因，manifest准确报告 missing_abstract_count，继续标partial；正文、联系方式及PostScript链接不当作摘要，也不下载正文。缺摘要不等于拒稿或未发表。本批输出仍包含原先已验收记录，正式导入前必须只取新增external ID，检查实时覆盖父记录并按incremental模式准备，不能直接重导入全部输出。
|