Merge remote-tracking branch 'origin/main' into codex/www-archive
Metadata integrity checks / metadata (push) Successful in 3m10s

# Conflicts:
#	README.md
This commit is contained in:
2026-11-09 00:48:14 +08:00
28 changed files with 1845 additions and 16 deletions
+107
View File
@@ -0,0 +1,107 @@
# Research types and custom annotations
Research type describes a claimed characteristic of the work (for example a survey,
benchmark, empirical study or replication). It is independent of topic, paper length,
conference track, presentation format and acceptance status. No type is inferred from
those fields. Multiple source-specific or human classifications may coexist; no
cross-platform vocabulary mapping or automatic classifier is provided.
`annotation_events` stores both kinds with an explicit `kind` discriminator:
`research_type` and `tag` are separate domains, even when namespace/value strings
happen to coincide. Existing platform labels remain in `tags` / `paper_tags`.
Each assertion carries namespace, value, actor, reason, evidence key, optional source
record and evidence URL, and the time it was recorded. This is observation time, not
paper publication time or a conference decision date.
Values are not forced into a global taxonomy: `custom:method=empirical` and
`platform:declared_type=empirical` retain separate provenance. An explicit platform
assertion API requires a source record belonging to that paper and matching the
namespace prefix. The manual CLI only accepts `custom:*`, and cannot retract platform
assertions. No source type has been automatically assigned to existing papers.
## Initialize explicitly
On the published PostgreSQL dataset, using its protected service file:
```sh
PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas annotate init \
--pg-service foundry-research --pg-schema research
```
This creates one optional table, its indexes, and an upgrade marker in one transaction,
using the existing dataset writer lock. It does not copy data, change existing papers
or decisions, or create a PostgreSQL instance. Repeating it is safe; an unexplained
pre-existing table/marker mismatch fails closed. Readers and ordinary imports do not
install the extension. Legacy datasets remain readable. A SQLite **sample** uses
`--db sample.sqlite`; the cutover SQLite backup remains protected against writes.
## Add and retract evidence
Use the canonical paper ID returned by `query`. For a reviewed classification:
```sh
paper-atlas annotate assert --pg-service foundry-research --pg-schema research \
--paper "$PAPER_ID" --kind research_type --namespace custom:method \
--value empirical --evidence-key manual:abstract-review \
--actor reviewer --reason 'Classified from the explicitly described study design' \
--request-id "$REQUEST_ID"
```
`$REQUEST_ID` must be a new, stable identifier for this action (for example a UUID).
Retry with the same request ID and identical content after an interrupted response;
it returns the original event ID. Reusing that ID with changed content is rejected.
A topic annotation uses `--kind tag --namespace custom:topic --value robotics` instead.
Optional `--source-record` and `--evidence-url` attach additional evidence; the source
record must belong to this paper. URLs are retained as evidence and are not fetched.
To withdraw one particular assertion, use its returned event ID:
```sh
paper-atlas annotate retract --pg-service foundry-research --pg-schema research \
--event "$EVENT_ID" --actor reviewer --reason 'Classification corrected after review' \
--request-id "$RETRACTION_REQUEST_ID"
```
Retraction adds an event referencing that assertion, preserving its original content.
It does not remove another contributor's assertion of the same label, platform tags,
source metadata or submission/decision history. A retracted assertion can be asserted
again with a new request ID and evidence; it is a new observation. The API does not
update or delete past annotation events. Database administrators still control direct
SQL access. Work reviewed as the same paper may subsequently be merged by the existing
identity workflow: annotation IDs, original paper IDs, evidence and retraction links
are retained while the canonical paper reference follows the surviving entity.
## Query and analyze
```sh
paper-atlas query --pg-service foundry-research --pg-schema research \
--research-type custom:method=empirical
paper-atlas query --pg-service foundry-research --pg-schema research \
--tag custom:topic=robotics
paper-atlas query --pg-service foundry-research --pg-schema research \
--paper "$PAPER_ID" --annotation-history
paper-atlas annotate list --pg-service foundry-research --pg-schema research \
--paper "$PAPER_ID" --history
```
`--tag` includes original platform labels and active annotation tags. It never matches
research types. `--research-type` matches active type assertions only. Paper details
return separate `tags` (existing provenance), `annotation_tags`, and `research_types`;
`--annotation-history` additionally includes retractions. A missing extension yields
empty annotation results rather than changing the database.
For direct SQL analysis, active research-type evidence is:
```sql
SELECT a.* FROM research.annotation_events a
WHERE a.kind = 'research_type' AND a.retracts_id IS NULL
AND NOT EXISTS (SELECT 1 FROM research.annotation_events r WHERE r.retracts_id = a.id);
```
Several independent assertions can support one label; count distinct papers/labels
when appropriate rather than counting evidence rows as papers. Namespaces stay
separate. Standard paper queries decode the PostgreSQL text codec; direct SQL consumers
must honor the dataset's documented codec, including NUL/SOH values. SQLite-to-PostgreSQL
migration recognizes an installed annotation extension and verifies its full row hashes,
constraints and identity sequence along with the original business tables. An old
21-table snapshot stays a 21-table migration; no synthetic annotations are generated.
+45
View File
@@ -0,0 +1,45 @@
# 按证据修正既有投稿的轨道
`paper-atlas maintain edition-tracks` 用于已导入整卷论文集、后来取得逐条官方分区证据的情形。它更正原投稿的 edition 归属,保留投稿 ID、所有决定和来源关系;不会重新导入论文、合并实体、生成研究方法标签或推断拒稿。支持独立 SQLite 样本和共享 PostgreSQL;不得写入已冻结的生产 SQLite 备份。
## 证据与完整范围
先核实官方目录、稳定 DOI、原始来源记录和分区,保存原始页面及 SHA256。输入 JSON 为 `{"format":1,"records":[...]}`,每条必须包含:
| 字段 | 含义 |
| --- | --- |
| `submission_id`、`source_record_id`、`external_id` | 原有投稿、主来源记录及主来源投稿标识 |
| `source`、`doi` | 原来源名称及其 metadata.record_identifiers 中的 DOI |
| `from_edition_id`、`year`、`venue` | 原 edition 及其真实年份和规范会议名 |
| `venue_names` | 规范会议名加全部已有别名;必须与库中一致 |
| `track` | 经核实的目标轨道,使用小写字母、数字、下划线;范围未明确可用 proceedings_unspecified |
| `section` | 官方分区原文或明确的未确定说明,不推断研究方法 |
| `evidence_url`、`evidence_sha256` | 已人工/独立核实的 HTTPS 原始证据和内容哈希 |
输入必须列出每个原 edition 的**全部**投稿,包含留在原轨道的记录。工具核对身份、DOI、单来源发表观察、现有别名、完整范围以及目标冲突;不自动解释 HTML,也不把填写了 URL/哈希当成已证明内容。核查者应先独立证明每条分区对应关系。
多来源共同拥有的投稿、有非 accepted 决定历史的投稿、已有分类修订、非空目标 edition、目标已有覆盖或标识等情形会停止,需要专门核对。每个原 edition 须有一个明确来源清单;其他来源或多清单覆盖需先另行协调。本功能不是任意批量移动或跨会合并工具。
## 只读计划、事务执行和验收
```sh
paper-atlas maintain edition-tracks \
--pg-service foundry-research --pg-schema research \
--inventory reviewed-inventory.json --out plan.json
sha256sum plan.json
paper-atlas maintain edition-tracks \
--pg-service foundry-research --pg-schema research \
--plan plan.json --plan-sha256 <上一步的完整SHA256> --out applied.json
```
报告路径必须是已有目录中的新文件。预检只读。执行使用 PostgreSQL 单写入 advisory lock,或 SQLite BEGIN IMMEDIATE;整个分类修正、逐来源证据、标识、覆盖及验收在一个事务中完成。计划绑定数据库身份、全部来源文本/非修订元数据、标签、可选 annotation_events、投稿/决定/来源/标识内容及原覆盖快照;过期即停止。失败回滚数据,PostgreSQL 序列可能留下正常间隙,不能用 max(id) 当数量。
目标 edition 沿用原会议、年份、event_key,不删除或重命名原 edition。只更正既有 submissions.edition_id 和对应 submission_identifiers.edition_id。原文、来源检索时间、决定 ID/状态/证据及各表来源关系不变。
分类证据写入来源元数据的保留字段 `edition_scope_curation_v1`,包括原范围、目标 edition、官方分区、URL、哈希和计划摘要。普通来源导入不能伪造或替换此字段;后续 record 刷新保留修订。相同来源投稿通过旧范围调用 decision 时仍路由到修正后的 edition。旧整卷 main 文件通过 import records 刷新会报错并回滚,要求先按核实轨道重新整理,避免覆盖正确的统计。
覆盖保留原清单 URL、cursor、成功时间,按实际目标分配 expected_count/imported_count 并标 partial;本工具核实分类,未核实摘要或所有来源完整性。原覆盖整行保留在计划中,不能继续将整卷数当 main 论文数。应用后校验原 edition 身份、全部受保护内容、分类、标识、修订字段和覆盖;相同计划再次执行仅在这些验收条件仍成立时返回 already_applied。
生产使用前必须在已确认 runner 完成软件验证。正式数据修正后仍需独立验收,并在最终全库审计中复核;单元测试和应用回执不能代替生产验收。
+1 -1
View File
@@ -17,7 +17,7 @@ PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain event-identi
--pg-service foundry-research --pg-schema research --out /data/paper-index/event-plan.json
```
检查计划的数据库身份、原 edition IDs、业务表数量、目标来源与历史指纹。精确计数在同一快照中按主键分页,每页最多5000行,只读取主键;不使用估计行数。CLI向stderr持续报告计数进度,避免大库长时间无输出。已有未标记 WWW1995 edition 必须逐投稿都有固定 DBLP Fall 卷证据,才会只为原 edition 填入 `www-1995-fall`;缺证或混合卷会拒绝计划。本文命令不表示生产已经迁移。
检查计划的数据库身份、原 edition IDs、业务表数量、目标来源与历史指纹。计数也包含已显式启用的可选注解表。精确计数在同一快照中按主键分页,每页最多5000行,只读取主键;不使用估计行数。CLI向stderr持续报告计数进度,避免大库长时间无输出。已有未标记 WWW1995 edition 必须逐投稿都有固定 DBLP Fall 卷证据,才会只为原 edition 填入 `www-1995-fall`;缺证或混合卷会拒绝计划。本文命令不表示生产已经迁移。
应用时传入同一文件及命令输出的文件 SHA256:
+67
View File
@@ -0,0 +1,67 @@
# Repairing joint conference membership
EURAI's [official history](https://www.eurai.org/ecai/history) identifies ECAI 2018
and 2022 as joint IJCAI/ECAI conferences. The pinned DBLP volumes
`conf/ijcai/2018` and `conf/ijcai/2022` use `IJCAI` as their booktitle and title.
Existing records can therefore already be present under IJCAI while an ECAI query
shows a missing year. Adding ECAI as a global alias to the ordinary IJCAI venue
would incorrectly include all other IJCAI years.
The existing canonical `ECAI+IJCAI` venue has both member aliases. This maintenance
command changes only the venue reference of the two existing `main` edition rows.
Edition IDs, submission IDs, decisions, source text, authors, identifiers, tags,
source links and coverage records stay intact. It does not merge or create papers
or submissions, resolve per-paper tracks, or turn reviews into negative decisions.
The DBLP collector applies the joint mapping only to these two exact main volume
keys; original booktitle/title remain in source metadata with the membership evidence.
The shared ingestion entry point also resolves these exact pinned source volumes before
creating an edition, so refreshing an older normalized file cannot recreate a separate
IJCAI edition. Other sources, years, workshop volumes and additional tracks retain their existing rules.
## Review a read-only plan
Use a runtime containing this repair command. The configured PostgreSQL writer lock
is used only for the apply operation; do not replace a newer ingestion runtime with
an older main-based feature branch.
```sh
PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain joint-editions \
--pg-service foundry-research --pg-schema research --out joint-plan.json
```
The preflight requires the existing joint venue and exactly its two member aliases,
one nonempty unambiguous edition per year, and matching pinned DBLP volume evidence
for every existing submission. A colliding target edition, missing provenance or
unexpected source scope fails closed. The plan binds current database/schema, IDs,
counts and full fingerprints of affected papers, identifiers, source records, tags,
submissions, decisions and their provenance, and coverage rows. Observation time is
recorded separately from conference year. These are scoped checks, not a final global
identity or conference completeness audit.
## Apply the exact reviewed plan
Compute the file's SHA256 after review and pass it explicitly:
```sh
PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain joint-editions \
--pg-service foundry-research --pg-schema research --plan joint-plan.json \
--plan-sha256 "$PLAN_SHA256" --out joint-applied.json
```
The plan hash is checked before opening a writer. Current evidence is checked again
under the dataset writer lock; both edition updates and an audit marker commit in one
transaction. Protected row hashes are checked again before commit. Changed evidence
requires a fresh reviewed plan. Existing output reports are preserved; choose a new
path for another run. SQLite samples use `--db sample.sqlite`; the frozen production
backup remains protected.
If a response is lost after commit, rerun with the same plan and a new output path.
When protected evidence is unchanged and both editions already point to the joint
venue, the result is `already_joint` with zero updates. It does not replay imports or
regenerate source records. Keep the plan, apply report and independent postcheck
alongside collection evidence. The audit marker is `joint_ijcai_ecai_2018_2022_v1`.
Publication counts can differ between the DBLP volume and the current publisher
page. This membership repair preserves all existing rows rather than deleting entries
to force an aggregate match; section-level main/invited/other classification and
publication completeness require separate per-paper evidence.
+49
View File
@@ -0,0 +1,49 @@
# Integrated research runtime
This branch composes the independently reviewed identity guards (PR25), incremental
coverage (PR27), joint-edition refresh guard (PR29), event identity (PR30), and optional
annotation support (PR28). It starts from main 1b81d1a and preserves their commits;
it is an explicit integration dependency, not evidence that those PRs have merged.
Users merge through Gitea manually. Review this combined diff against main, including
its conflict resolutions, before using it as the maintained runtime.
The shared Store preserves stable source/identifier anchors before title matching,
rejects ambiguous unanchored matches, resolves pinned IJCAI/ECAI volumes before event
lookup, and selects same-year events by event_key. Incremental manifests for a named
or ambiguous event must state event_key; their parent count and optimistic update
are checked against that exact event, not the first venue/year/track match.
Query filters compose event_key, status, tags and research type. An earlier rejected
attempt and a later known acceptance remain distinct; neither annotations nor track
names infer a research type. Installing this code does not initialize annotation_events.
Event plans count that optional table when present, alongside the original business
rows. Annotation initialization and event identity upgrade remain explicit commands
with their own transactional checks. Neither command imports historical records.
## Production sequencing
1. Verify the exact integrated commit on the approved ces-control runner, including
the real PostgreSQL interaction tests and wheel/installed CLI checks.
2. Stage that exact source in a separate release directory and verify package hashes;
leave the current runtime and frozen SQLite backup intact.
3. Inspect live writers and coordinate through the existing database writer lock.
Generate a fresh read-only event plan using that integrated release. Check its
database/schema identity, all business counts and preserved history hashes. A plan
from before further data or optional-schema changes must not be reused.
4. Apply only that reviewed file with its actual SHA256, in one transaction. Independently
verify the constraint, original edition IDs and fields, assigned Fall source, source
and history bytes, ordinary queries, full-text queries and foundry health.
5. Use the verified integrated release for subsequent writes. Older writers lacking
event identity support must not resume against the upgraded schema. Annotation
initialization requires a separate explicit operation and acceptance.
These are operational requirements, not claims of completed production deployment.
The production event-key migration and optional annotation activation have not occurred
at preparation time. Existing migration/import/repair acceptance remains valid; do not
rerun those jobs to install source code. Cross-feature tests use disposable fixtures
and PostgreSQL schemas, never the production research tables.
Coverage output also returns event_key for each independent edition ID. Two same-year
main events keep their own status/source counts; venue paper totals remain distinct
work counts rather than sums across events. Legacy datasets without the event column
return an empty event key and remain read-only compatible.