lean-bench/experiments/P0-pilot

P0: pilot

Experiment ID P0. Spec, hypotheses, decision rule and costs: docs/experiments/P0-pilot.md. Registry of all experiments: docs/experiments/README.md.

Question: on the 65 existing items, does role alignment (passage simplex vs. Lean simplex) find unfaithful formalizations better than a single LLM judge, compile-only, or static lint?

Layout

data/      ignored by git: items.jsonl (copy of the 65 items), labelsheet.html, results.jsonl
           (all contain or paraphrase passages from a copyrighted book)
labels/    tracked: key.json (blind id -> item id, no text), labels.json (verdicts and error
           types only; do not paste passage text into the notes field)
results/   tracked: summary.json written by score.py (numbers only)

Steps

cp runs/p85-110/items.jsonl experiments/P0-pilot/data/items.jsonl
python experiments/P0-pilot/make_sheet.py --items experiments/P0-pilot/data/items.jsonl \
    --out experiments/P0-pilot/data --key-dir experiments/P0-pilot/labels --seed 7
# a person opens data/labelsheet.html, labels all 65 items, downloads labels.json into labels/
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
    --out experiments/P0-pilot/data/results.jsonl --dry-run
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
    --out experiments/P0-pilot/data/results.jsonl
python experiments/P0-pilot/score.py --results experiments/P0-pilot/data/results.jsonl \
    --labels experiments/P0-pilot/labels/labels.json --key experiments/P0-pilot/labels/key.json \
    --summary experiments/P0-pilot/results/summary.json

Labeling must be finished before anyone opens results.jsonl.