# P0: pilot

Experiment ID **P0**. Spec, hypotheses, decision rule and costs: `docs/experiments/P0-pilot.md`.
Registry of all experiments: `docs/experiments/README.md`.

Question: on the 65 existing items, does role alignment (passage simplex vs. Lean simplex) find
unfaithful formalizations better than a single LLM judge, compile-only, or static lint?

## Layout

    data/      ignored by git: items.jsonl (copy of the 65 items), labelsheet.html, results.jsonl
               (all contain or paraphrase passages from a copyrighted book)
    labels/    tracked: key.json (blind id -> item id, no text), labels.json (verdicts and error
               types only; do not paste passage text into the notes field)
    results/   tracked: summary.json written by score.py (numbers only)

## Steps

    cp runs/p85-110/items.jsonl experiments/P0-pilot/data/items.jsonl
    python experiments/P0-pilot/make_sheet.py --items experiments/P0-pilot/data/items.jsonl \
        --out experiments/P0-pilot/data --key-dir experiments/P0-pilot/labels --seed 7
    # a person opens data/labelsheet.html, labels all 65 items, downloads labels.json into labels/
    python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
        --out experiments/P0-pilot/data/results.jsonl --dry-run
    python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
        --out experiments/P0-pilot/data/results.jsonl
    python experiments/P0-pilot/score.py --results experiments/P0-pilot/data/results.jsonl \
        --labels experiments/P0-pilot/labels/labels.json --key experiments/P0-pilot/labels/key.json \
        --summary experiments/P0-pilot/results/summary.json

Labeling must be finished before anyone opens `results.jsonl`.