fix(pilot): freeze AI labels and enforce auditable execution
This commit is contained in:
@@ -16,16 +16,39 @@ unfaithful formalizations better than a single LLM judge, compile-only, or stati
|
||||
|
||||
## Steps
|
||||
|
||||
cp runs/p85-110/items.jsonl experiments/P0-pilot/data/items.jsonl
|
||||
# Default exports omit source_text. Restore it from the matching private review.
|
||||
python experiments/P0-pilot/prepare_items.py --items runs/p85-110/items.jsonl \
|
||||
--review runs/p85-110/review.json --out experiments/P0-pilot/data/items.jsonl
|
||||
python experiments/P0-pilot/make_sheet.py --items experiments/P0-pilot/data/items.jsonl \
|
||||
--out experiments/P0-pilot/data --key-dir experiments/P0-pilot/labels --seed 7
|
||||
# a person opens data/labelsheet.html, labels all 65 items, downloads labels.json into labels/
|
||||
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
|
||||
--out experiments/P0-pilot/data/results.jsonl --dry-run
|
||||
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
|
||||
--out experiments/P0-pilot/data/results.jsonl
|
||||
--out experiments/P0-pilot/data/results.jsonl --labels experiments/P0-pilot/labels/labels.json
|
||||
python experiments/P0-pilot/score.py --results experiments/P0-pilot/data/results.jsonl \
|
||||
--labels experiments/P0-pilot/labels/labels.json --key experiments/P0-pilot/labels/key.json \
|
||||
--summary experiments/P0-pilot/results/summary.json
|
||||
|
||||
Labeling must be finished before anyone opens `results.jsonl`.
|
||||
|
||||
## Execution safeguards
|
||||
|
||||
- Every item must have a nonempty source passage. `prepare_items.py` restores passages only
|
||||
when all exported fields match the archived review; the exported cohort and Lean code stay intact.
|
||||
- Freeze all labels before the paid run. The owner changed the labeler to the Codex assistant
|
||||
on 2026-10-28. These are **AI labels, not human gold**; see the metadata in `labels/labels.json`.
|
||||
- `--max-calls` counts actual HTTP attempts, including transport and JSON retries, cumulatively
|
||||
across resumes. The hard ceiling is 400. A reservation is saved before each request; an interrupted
|
||||
request may consume a reservation without a known response or token count.
|
||||
- Metadata is stored beside the output as `results.meta.json`. Model, endpoint, temperature,
|
||||
input hash, frozen-label hash and prompt hashes must match on resume. Usage is cumulative.
|
||||
A file lock prevents concurrent writers. A budget stop returns exit status 2.
|
||||
- The nominal successful path uses 130 calls (one source extraction and one judge per item);
|
||||
malformed replies and network retries may increase the actual count.
|
||||
- Scoring requires complete labels and results and no unresolved detector errors. The primary
|
||||
comparison is wrong versus faithful; minor drift is included as positive in the secondary analysis.
|
||||
The complementary-signal PARTIAL rule counts wrong items flagged by role alignment but missed
|
||||
by the judge. No threshold, concept table or detector prompt was tuned for these labels.
|
||||
- Raw passages, labeling HTML and model responses remain in the ignored `data/` directory.
|
||||
Only verdicts, abstract annotation notes, hashes and text-free summaries belong in Git.
|
||||
|
||||
Reference in New Issue
Block a user