fix(pilot): freeze AI labels and enforce auditable execution
ci / python (pull_request) Successful in 22s
ci / lean (pull_request) Successful in 1m16s
ci / image (pull_request) Has been skipped

This commit is contained in:
2026-10-28 18:46:24 +08:00
parent 21d422fb35
commit 6875c79c40
9 changed files with 826 additions and 50 deletions
+25 -2
View File
@@ -16,16 +16,39 @@ unfaithful formalizations better than a single LLM judge, compile-only, or stati
## Steps
cp runs/p85-110/items.jsonl experiments/P0-pilot/data/items.jsonl
# Default exports omit source_text. Restore it from the matching private review.
python experiments/P0-pilot/prepare_items.py --items runs/p85-110/items.jsonl \
--review runs/p85-110/review.json --out experiments/P0-pilot/data/items.jsonl
python experiments/P0-pilot/make_sheet.py --items experiments/P0-pilot/data/items.jsonl \
--out experiments/P0-pilot/data --key-dir experiments/P0-pilot/labels --seed 7
# a person opens data/labelsheet.html, labels all 65 items, downloads labels.json into labels/
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
--out experiments/P0-pilot/data/results.jsonl --dry-run
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
--out experiments/P0-pilot/data/results.jsonl
--out experiments/P0-pilot/data/results.jsonl --labels experiments/P0-pilot/labels/labels.json
python experiments/P0-pilot/score.py --results experiments/P0-pilot/data/results.jsonl \
--labels experiments/P0-pilot/labels/labels.json --key experiments/P0-pilot/labels/key.json \
--summary experiments/P0-pilot/results/summary.json
Labeling must be finished before anyone opens `results.jsonl`.
## Execution safeguards
- Every item must have a nonempty source passage. `prepare_items.py` restores passages only
when all exported fields match the archived review; the exported cohort and Lean code stay intact.
- Freeze all labels before the paid run. The owner changed the labeler to the Codex assistant
on 2026-10-28. These are **AI labels, not human gold**; see the metadata in `labels/labels.json`.
- `--max-calls` counts actual HTTP attempts, including transport and JSON retries, cumulatively
across resumes. The hard ceiling is 400. A reservation is saved before each request; an interrupted
request may consume a reservation without a known response or token count.
- Metadata is stored beside the output as `results.meta.json`. Model, endpoint, temperature,
input hash, frozen-label hash and prompt hashes must match on resume. Usage is cumulative.
A file lock prevents concurrent writers. A budget stop returns exit status 2.
- The nominal successful path uses 130 calls (one source extraction and one judge per item);
malformed replies and network retries may increase the actual count.
- Scoring requires complete labels and results and no unresolved detector errors. The primary
comparison is wrong versus faithful; minor drift is included as positive in the secondary analysis.
The complementary-signal PARTIAL rule counts wrong items flagged by role alignment but missed
by the judge. No threshold, concept table or detector prompt was tuned for these labels.
- Raw passages, labeling HTML and model responses remain in the ignored `data/` directory.
Only verdicts, abstract annotation notes, hashes and text-free summaries belong in Git.