claude-bot
2b255a333a
Raw pilot data stays ignored (copyrighted text); blind key and numeric results are tracked. Runner notes move into docs/infrastructure. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FYqwNMyPUHeTbFWg83GQq6
1.7 KiB
1.7 KiB
P0: pilot
Experiment ID P0. Spec, hypotheses, decision rule and costs: docs/experiments/P0-pilot.md.
Registry of all experiments: docs/experiments/README.md.
Question: on the 65 existing items, does role alignment (passage simplex vs. Lean simplex) find unfaithful formalizations better than a single LLM judge, compile-only, or static lint?
Layout
data/ ignored by git: items.jsonl (copy of the 65 items), labelsheet.html, results.jsonl
(all contain or paraphrase passages from a copyrighted book)
labels/ tracked: key.json (blind id -> item id, no text), labels.json (verdicts and error
types only; do not paste passage text into the notes field)
results/ tracked: summary.json written by score.py (numbers only)
Steps
cp runs/p85-110/items.jsonl experiments/P0-pilot/data/items.jsonl
python experiments/P0-pilot/make_sheet.py --items experiments/P0-pilot/data/items.jsonl \
--out experiments/P0-pilot/data --key-dir experiments/P0-pilot/labels --seed 7
# a person opens data/labelsheet.html, labels all 65 items, downloads labels.json into labels/
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
--out experiments/P0-pilot/data/results.jsonl --dry-run
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
--out experiments/P0-pilot/data/results.jsonl
python experiments/P0-pilot/score.py --results experiments/P0-pilot/data/results.jsonl \
--labels experiments/P0-pilot/labels/labels.json --key experiments/P0-pilot/labels/key.json \
--summary experiments/P0-pilot/results/summary.json
Labeling must be finished before anyone opens results.jsonl.