Merge remote-tracking branch 'origin/main' into codex/www-archive
Metadata integrity checks / metadata (push) Successful in 3m10s
Metadata integrity checks / metadata (push) Successful in 3m10s
# Conflicts: # README.md
This commit is contained in:
@@ -0,0 +1,107 @@
|
||||
# Research types and custom annotations
|
||||
|
||||
Research type describes a claimed characteristic of the work (for example a survey,
|
||||
benchmark, empirical study or replication). It is independent of topic, paper length,
|
||||
conference track, presentation format and acceptance status. No type is inferred from
|
||||
those fields. Multiple source-specific or human classifications may coexist; no
|
||||
cross-platform vocabulary mapping or automatic classifier is provided.
|
||||
|
||||
`annotation_events` stores both kinds with an explicit `kind` discriminator:
|
||||
`research_type` and `tag` are separate domains, even when namespace/value strings
|
||||
happen to coincide. Existing platform labels remain in `tags` / `paper_tags`.
|
||||
Each assertion carries namespace, value, actor, reason, evidence key, optional source
|
||||
record and evidence URL, and the time it was recorded. This is observation time, not
|
||||
paper publication time or a conference decision date.
|
||||
|
||||
Values are not forced into a global taxonomy: `custom:method=empirical` and
|
||||
`platform:declared_type=empirical` retain separate provenance. An explicit platform
|
||||
assertion API requires a source record belonging to that paper and matching the
|
||||
namespace prefix. The manual CLI only accepts `custom:*`, and cannot retract platform
|
||||
assertions. No source type has been automatically assigned to existing papers.
|
||||
|
||||
## Initialize explicitly
|
||||
|
||||
On the published PostgreSQL dataset, using its protected service file:
|
||||
|
||||
```sh
|
||||
PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas annotate init \
|
||||
--pg-service foundry-research --pg-schema research
|
||||
```
|
||||
|
||||
This creates one optional table, its indexes, and an upgrade marker in one transaction,
|
||||
using the existing dataset writer lock. It does not copy data, change existing papers
|
||||
or decisions, or create a PostgreSQL instance. Repeating it is safe; an unexplained
|
||||
pre-existing table/marker mismatch fails closed. Readers and ordinary imports do not
|
||||
install the extension. Legacy datasets remain readable. A SQLite **sample** uses
|
||||
`--db sample.sqlite`; the cutover SQLite backup remains protected against writes.
|
||||
|
||||
## Add and retract evidence
|
||||
|
||||
Use the canonical paper ID returned by `query`. For a reviewed classification:
|
||||
|
||||
```sh
|
||||
paper-atlas annotate assert --pg-service foundry-research --pg-schema research \
|
||||
--paper "$PAPER_ID" --kind research_type --namespace custom:method \
|
||||
--value empirical --evidence-key manual:abstract-review \
|
||||
--actor reviewer --reason 'Classified from the explicitly described study design' \
|
||||
--request-id "$REQUEST_ID"
|
||||
```
|
||||
|
||||
`$REQUEST_ID` must be a new, stable identifier for this action (for example a UUID).
|
||||
Retry with the same request ID and identical content after an interrupted response;
|
||||
it returns the original event ID. Reusing that ID with changed content is rejected.
|
||||
A topic annotation uses `--kind tag --namespace custom:topic --value robotics` instead.
|
||||
Optional `--source-record` and `--evidence-url` attach additional evidence; the source
|
||||
record must belong to this paper. URLs are retained as evidence and are not fetched.
|
||||
|
||||
To withdraw one particular assertion, use its returned event ID:
|
||||
|
||||
```sh
|
||||
paper-atlas annotate retract --pg-service foundry-research --pg-schema research \
|
||||
--event "$EVENT_ID" --actor reviewer --reason 'Classification corrected after review' \
|
||||
--request-id "$RETRACTION_REQUEST_ID"
|
||||
```
|
||||
|
||||
Retraction adds an event referencing that assertion, preserving its original content.
|
||||
It does not remove another contributor's assertion of the same label, platform tags,
|
||||
source metadata or submission/decision history. A retracted assertion can be asserted
|
||||
again with a new request ID and evidence; it is a new observation. The API does not
|
||||
update or delete past annotation events. Database administrators still control direct
|
||||
SQL access. Work reviewed as the same paper may subsequently be merged by the existing
|
||||
identity workflow: annotation IDs, original paper IDs, evidence and retraction links
|
||||
are retained while the canonical paper reference follows the surviving entity.
|
||||
|
||||
## Query and analyze
|
||||
|
||||
```sh
|
||||
paper-atlas query --pg-service foundry-research --pg-schema research \
|
||||
--research-type custom:method=empirical
|
||||
paper-atlas query --pg-service foundry-research --pg-schema research \
|
||||
--tag custom:topic=robotics
|
||||
paper-atlas query --pg-service foundry-research --pg-schema research \
|
||||
--paper "$PAPER_ID" --annotation-history
|
||||
paper-atlas annotate list --pg-service foundry-research --pg-schema research \
|
||||
--paper "$PAPER_ID" --history
|
||||
```
|
||||
|
||||
`--tag` includes original platform labels and active annotation tags. It never matches
|
||||
research types. `--research-type` matches active type assertions only. Paper details
|
||||
return separate `tags` (existing provenance), `annotation_tags`, and `research_types`;
|
||||
`--annotation-history` additionally includes retractions. A missing extension yields
|
||||
empty annotation results rather than changing the database.
|
||||
|
||||
For direct SQL analysis, active research-type evidence is:
|
||||
|
||||
```sql
|
||||
SELECT a.* FROM research.annotation_events a
|
||||
WHERE a.kind = 'research_type' AND a.retracts_id IS NULL
|
||||
AND NOT EXISTS (SELECT 1 FROM research.annotation_events r WHERE r.retracts_id = a.id);
|
||||
```
|
||||
|
||||
Several independent assertions can support one label; count distinct papers/labels
|
||||
when appropriate rather than counting evidence rows as papers. Namespaces stay
|
||||
separate. Standard paper queries decode the PostgreSQL text codec; direct SQL consumers
|
||||
must honor the dataset's documented codec, including NUL/SOH values. SQLite-to-PostgreSQL
|
||||
migration recognizes an installed annotation extension and verifies its full row hashes,
|
||||
constraints and identity sequence along with the original business tables. An old
|
||||
21-table snapshot stays a 21-table migration; no synthetic annotations are generated.
|
||||
@@ -0,0 +1,45 @@
|
||||
# 按证据修正既有投稿的轨道
|
||||
|
||||
`paper-atlas maintain edition-tracks` 用于已导入整卷论文集、后来取得逐条官方分区证据的情形。它更正原投稿的 edition 归属,保留投稿 ID、所有决定和来源关系;不会重新导入论文、合并实体、生成研究方法标签或推断拒稿。支持独立 SQLite 样本和共享 PostgreSQL;不得写入已冻结的生产 SQLite 备份。
|
||||
|
||||
## 证据与完整范围
|
||||
|
||||
先核实官方目录、稳定 DOI、原始来源记录和分区,保存原始页面及 SHA256。输入 JSON 为 `{"format":1,"records":[...]}`,每条必须包含:
|
||||
|
||||
| 字段 | 含义 |
|
||||
| --- | --- |
|
||||
| `submission_id`、`source_record_id`、`external_id` | 原有投稿、主来源记录及主来源投稿标识 |
|
||||
| `source`、`doi` | 原来源名称及其 metadata.record_identifiers 中的 DOI |
|
||||
| `from_edition_id`、`year`、`venue` | 原 edition 及其真实年份和规范会议名 |
|
||||
| `venue_names` | 规范会议名加全部已有别名;必须与库中一致 |
|
||||
| `track` | 经核实的目标轨道,使用小写字母、数字、下划线;范围未明确可用 proceedings_unspecified |
|
||||
| `section` | 官方分区原文或明确的未确定说明,不推断研究方法 |
|
||||
| `evidence_url`、`evidence_sha256` | 已人工/独立核实的 HTTPS 原始证据和内容哈希 |
|
||||
|
||||
输入必须列出每个原 edition 的**全部**投稿,包含留在原轨道的记录。工具核对身份、DOI、单来源发表观察、现有别名、完整范围以及目标冲突;不自动解释 HTML,也不把填写了 URL/哈希当成已证明内容。核查者应先独立证明每条分区对应关系。
|
||||
|
||||
多来源共同拥有的投稿、有非 accepted 决定历史的投稿、已有分类修订、非空目标 edition、目标已有覆盖或标识等情形会停止,需要专门核对。每个原 edition 须有一个明确来源清单;其他来源或多清单覆盖需先另行协调。本功能不是任意批量移动或跨会合并工具。
|
||||
|
||||
## 只读计划、事务执行和验收
|
||||
|
||||
```sh
|
||||
paper-atlas maintain edition-tracks \
|
||||
--pg-service foundry-research --pg-schema research \
|
||||
--inventory reviewed-inventory.json --out plan.json
|
||||
|
||||
sha256sum plan.json
|
||||
|
||||
paper-atlas maintain edition-tracks \
|
||||
--pg-service foundry-research --pg-schema research \
|
||||
--plan plan.json --plan-sha256 <上一步的完整SHA256> --out applied.json
|
||||
```
|
||||
|
||||
报告路径必须是已有目录中的新文件。预检只读。执行使用 PostgreSQL 单写入 advisory lock,或 SQLite BEGIN IMMEDIATE;整个分类修正、逐来源证据、标识、覆盖及验收在一个事务中完成。计划绑定数据库身份、全部来源文本/非修订元数据、标签、可选 annotation_events、投稿/决定/来源/标识内容及原覆盖快照;过期即停止。失败回滚数据,PostgreSQL 序列可能留下正常间隙,不能用 max(id) 当数量。
|
||||
|
||||
目标 edition 沿用原会议、年份、event_key,不删除或重命名原 edition。只更正既有 submissions.edition_id 和对应 submission_identifiers.edition_id。原文、来源检索时间、决定 ID/状态/证据及各表来源关系不变。
|
||||
|
||||
分类证据写入来源元数据的保留字段 `edition_scope_curation_v1`,包括原范围、目标 edition、官方分区、URL、哈希和计划摘要。普通来源导入不能伪造或替换此字段;后续 record 刷新保留修订。相同来源投稿通过旧范围调用 decision 时仍路由到修正后的 edition。旧整卷 main 文件通过 import records 刷新会报错并回滚,要求先按核实轨道重新整理,避免覆盖正确的统计。
|
||||
|
||||
覆盖保留原清单 URL、cursor、成功时间,按实际目标分配 expected_count/imported_count 并标 partial;本工具核实分类,未核实摘要或所有来源完整性。原覆盖整行保留在计划中,不能继续将整卷数当 main 论文数。应用后校验原 edition 身份、全部受保护内容、分类、标识、修订字段和覆盖;相同计划再次执行仅在这些验收条件仍成立时返回 already_applied。
|
||||
|
||||
生产使用前必须在已确认 runner 完成软件验证。正式数据修正后仍需独立验收,并在最终全库审计中复核;单元测试和应用回执不能代替生产验收。
|
||||
@@ -17,7 +17,7 @@ PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain event-identi
|
||||
--pg-service foundry-research --pg-schema research --out /data/paper-index/event-plan.json
|
||||
```
|
||||
|
||||
检查计划的数据库身份、原 edition IDs、业务表数量、目标来源与历史指纹。精确计数在同一快照中按主键分页,每页最多5000行,只读取主键;不使用估计行数。CLI向stderr持续报告计数进度,避免大库长时间无输出。已有未标记 WWW1995 edition 必须逐投稿都有固定 DBLP Fall 卷证据,才会只为原 edition 填入 `www-1995-fall`;缺证或混合卷会拒绝计划。本文命令不表示生产已经迁移。
|
||||
检查计划的数据库身份、原 edition IDs、业务表数量、目标来源与历史指纹。计数也包含已显式启用的可选注解表。精确计数在同一快照中按主键分页,每页最多5000行,只读取主键;不使用估计行数。CLI向stderr持续报告计数进度,避免大库长时间无输出。已有未标记 WWW1995 edition 必须逐投稿都有固定 DBLP Fall 卷证据,才会只为原 edition 填入 `www-1995-fall`;缺证或混合卷会拒绝计划。本文命令不表示生产已经迁移。
|
||||
|
||||
应用时传入同一文件及命令输出的文件 SHA256:
|
||||
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
# Repairing joint conference membership
|
||||
|
||||
EURAI's [official history](https://www.eurai.org/ecai/history) identifies ECAI 2018
|
||||
and 2022 as joint IJCAI/ECAI conferences. The pinned DBLP volumes
|
||||
`conf/ijcai/2018` and `conf/ijcai/2022` use `IJCAI` as their booktitle and title.
|
||||
Existing records can therefore already be present under IJCAI while an ECAI query
|
||||
shows a missing year. Adding ECAI as a global alias to the ordinary IJCAI venue
|
||||
would incorrectly include all other IJCAI years.
|
||||
|
||||
The existing canonical `ECAI+IJCAI` venue has both member aliases. This maintenance
|
||||
command changes only the venue reference of the two existing `main` edition rows.
|
||||
Edition IDs, submission IDs, decisions, source text, authors, identifiers, tags,
|
||||
source links and coverage records stay intact. It does not merge or create papers
|
||||
or submissions, resolve per-paper tracks, or turn reviews into negative decisions.
|
||||
The DBLP collector applies the joint mapping only to these two exact main volume
|
||||
keys; original booktitle/title remain in source metadata with the membership evidence.
|
||||
The shared ingestion entry point also resolves these exact pinned source volumes before
|
||||
creating an edition, so refreshing an older normalized file cannot recreate a separate
|
||||
IJCAI edition. Other sources, years, workshop volumes and additional tracks retain their existing rules.
|
||||
|
||||
## Review a read-only plan
|
||||
|
||||
Use a runtime containing this repair command. The configured PostgreSQL writer lock
|
||||
is used only for the apply operation; do not replace a newer ingestion runtime with
|
||||
an older main-based feature branch.
|
||||
|
||||
```sh
|
||||
PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain joint-editions \
|
||||
--pg-service foundry-research --pg-schema research --out joint-plan.json
|
||||
```
|
||||
|
||||
The preflight requires the existing joint venue and exactly its two member aliases,
|
||||
one nonempty unambiguous edition per year, and matching pinned DBLP volume evidence
|
||||
for every existing submission. A colliding target edition, missing provenance or
|
||||
unexpected source scope fails closed. The plan binds current database/schema, IDs,
|
||||
counts and full fingerprints of affected papers, identifiers, source records, tags,
|
||||
submissions, decisions and their provenance, and coverage rows. Observation time is
|
||||
recorded separately from conference year. These are scoped checks, not a final global
|
||||
identity or conference completeness audit.
|
||||
|
||||
## Apply the exact reviewed plan
|
||||
|
||||
Compute the file's SHA256 after review and pass it explicitly:
|
||||
|
||||
```sh
|
||||
PGSERVICEFILE=/etc/paper-atlas/pg_service.conf paper-atlas maintain joint-editions \
|
||||
--pg-service foundry-research --pg-schema research --plan joint-plan.json \
|
||||
--plan-sha256 "$PLAN_SHA256" --out joint-applied.json
|
||||
```
|
||||
|
||||
The plan hash is checked before opening a writer. Current evidence is checked again
|
||||
under the dataset writer lock; both edition updates and an audit marker commit in one
|
||||
transaction. Protected row hashes are checked again before commit. Changed evidence
|
||||
requires a fresh reviewed plan. Existing output reports are preserved; choose a new
|
||||
path for another run. SQLite samples use `--db sample.sqlite`; the frozen production
|
||||
backup remains protected.
|
||||
|
||||
If a response is lost after commit, rerun with the same plan and a new output path.
|
||||
When protected evidence is unchanged and both editions already point to the joint
|
||||
venue, the result is `already_joint` with zero updates. It does not replay imports or
|
||||
regenerate source records. Keep the plan, apply report and independent postcheck
|
||||
alongside collection evidence. The audit marker is `joint_ijcai_ecai_2018_2022_v1`.
|
||||
|
||||
Publication counts can differ between the DBLP volume and the current publisher
|
||||
page. This membership repair preserves all existing rows rather than deleting entries
|
||||
to force an aggregate match; section-level main/invited/other classification and
|
||||
publication completeness require separate per-paper evidence.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Integrated research runtime
|
||||
|
||||
This branch composes the independently reviewed identity guards (PR25), incremental
|
||||
coverage (PR27), joint-edition refresh guard (PR29), event identity (PR30), and optional
|
||||
annotation support (PR28). It starts from main 1b81d1a and preserves their commits;
|
||||
it is an explicit integration dependency, not evidence that those PRs have merged.
|
||||
Users merge through Gitea manually. Review this combined diff against main, including
|
||||
its conflict resolutions, before using it as the maintained runtime.
|
||||
|
||||
The shared Store preserves stable source/identifier anchors before title matching,
|
||||
rejects ambiguous unanchored matches, resolves pinned IJCAI/ECAI volumes before event
|
||||
lookup, and selects same-year events by event_key. Incremental manifests for a named
|
||||
or ambiguous event must state event_key; their parent count and optimistic update
|
||||
are checked against that exact event, not the first venue/year/track match.
|
||||
|
||||
Query filters compose event_key, status, tags and research type. An earlier rejected
|
||||
attempt and a later known acceptance remain distinct; neither annotations nor track
|
||||
names infer a research type. Installing this code does not initialize annotation_events.
|
||||
Event plans count that optional table when present, alongside the original business
|
||||
rows. Annotation initialization and event identity upgrade remain explicit commands
|
||||
with their own transactional checks. Neither command imports historical records.
|
||||
|
||||
## Production sequencing
|
||||
|
||||
1. Verify the exact integrated commit on the approved ces-control runner, including
|
||||
the real PostgreSQL interaction tests and wheel/installed CLI checks.
|
||||
2. Stage that exact source in a separate release directory and verify package hashes;
|
||||
leave the current runtime and frozen SQLite backup intact.
|
||||
3. Inspect live writers and coordinate through the existing database writer lock.
|
||||
Generate a fresh read-only event plan using that integrated release. Check its
|
||||
database/schema identity, all business counts and preserved history hashes. A plan
|
||||
from before further data or optional-schema changes must not be reused.
|
||||
4. Apply only that reviewed file with its actual SHA256, in one transaction. Independently
|
||||
verify the constraint, original edition IDs and fields, assigned Fall source, source
|
||||
and history bytes, ordinary queries, full-text queries and foundry health.
|
||||
5. Use the verified integrated release for subsequent writes. Older writers lacking
|
||||
event identity support must not resume against the upgraded schema. Annotation
|
||||
initialization requires a separate explicit operation and acceptance.
|
||||
|
||||
These are operational requirements, not claims of completed production deployment.
|
||||
The production event-key migration and optional annotation activation have not occurred
|
||||
at preparation time. Existing migration/import/repair acceptance remains valid; do not
|
||||
rerun those jobs to install source code. Cross-feature tests use disposable fixtures
|
||||
and PostgreSQL schemas, never the production research tables.
|
||||
|
||||
Coverage output also returns event_key for each independent edition ID. Two same-year
|
||||
main events keep their own status/source counts; venue paper totals remain distinct
|
||||
work counts rather than sums across events. Legacy datasets without the event column
|
||||
return an empty event key and remain read-only compatible.
|
||||
Reference in New Issue
Block a user