# P0: pilot

Experiment ID **P0**. Spec, hypotheses, decision rule and costs: `docs/experiments/P0-pilot.md`.
Registry of all experiments: `docs/experiments/README.md`.

Question: on the 65 existing items, does role alignment (passage simplex vs. Lean simplex) find
unfaithful formalizations better than a single LLM judge, compile-only, or static lint?

## Layout

    data/      ignored by git: items.jsonl (copy of the 65 items), labelsheet.html, results.jsonl
               (all contain or paraphrase passages from a copyrighted book)
    labels/    tracked: key.json (blind id -> item id, no text), labels.json (verdicts and error
               types only; do not paste passage text into the notes field)
    results/   tracked: summary.json written by score.py (numbers only)

## Steps

    # Default exports omit source_text. Restore it from the matching private review.
    python experiments/P0-pilot/prepare_items.py --items runs/p85-110/items.jsonl \
        --review runs/p85-110/review.json --out experiments/P0-pilot/data/items.jsonl
    python experiments/P0-pilot/make_sheet.py --items experiments/P0-pilot/data/items.jsonl \
        --out experiments/P0-pilot/data --key-dir experiments/P0-pilot/labels --seed 7
    # a person opens data/labelsheet.html, labels all 65 items, downloads labels.json into labels/
    python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
        --out experiments/P0-pilot/data/results.jsonl --dry-run
    python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
        --out experiments/P0-pilot/data/results.jsonl --labels experiments/P0-pilot/labels/labels.json
    python experiments/P0-pilot/score.py --results experiments/P0-pilot/data/results.jsonl \
        --labels experiments/P0-pilot/labels/labels.json --key experiments/P0-pilot/labels/key.json \
        --summary experiments/P0-pilot/results/summary.json

Labeling must be finished before anyone opens `results.jsonl`.

## Execution safeguards

- Every item must have a nonempty source passage. `prepare_items.py` restores passages only
  when all exported fields match the archived review; the exported cohort and Lean code stay intact.
- Freeze all labels before the paid run. The owner changed the labeler to the Codex assistant
  on 2026-10-28. These are **AI labels, not human gold**; see the metadata in `labels/labels.json`.
- `--max-calls` counts actual HTTP attempts, including transport and JSON retries, cumulatively
  across resumes. The hard ceiling is 400. A reservation is saved before each request; an interrupted
  request may consume a reservation without a known response or token count.
- Metadata is stored beside the output as `results.meta.json`. Model, endpoint, temperature,
  input hash, frozen-label hash and prompt hashes must match on resume. Usage is cumulative.
  A file lock prevents concurrent writers. A budget stop returns exit status 2.
- The nominal successful path uses 130 calls (one source extraction and one judge per item);
  malformed replies and network retries may increase the actual count.
- Scoring requires complete labels and results and no unresolved detector errors. The primary
  comparison is wrong versus faithful; minor drift is included as positive in the secondary analysis.
  The complementary-signal PARTIAL rule counts wrong items flagged by role alignment but missed
  by the judge. No threshold, concept table or detector prompt was tuned for these labels.
- Raw passages, labeling HTML and model responses remain in the ignored `data/` directory.
  Only verdicts, abstract annotation notes, hashes and text-free summaries belong in Git.