claude-bot
2b255a333a
Raw pilot data stays ignored (copyrighted text); blind key and numeric results are tracked. Runner notes move into docs/infrastructure. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FYqwNMyPUHeTbFWg83GQq6
32 lines
1.7 KiB
Markdown
32 lines
1.7 KiB
Markdown
# P0: pilot
|
|
|
|
Experiment ID **P0**. Spec, hypotheses, decision rule and costs: `docs/experiments/P0-pilot.md`.
|
|
Registry of all experiments: `docs/experiments/README.md`.
|
|
|
|
Question: on the 65 existing items, does role alignment (passage simplex vs. Lean simplex) find
|
|
unfaithful formalizations better than a single LLM judge, compile-only, or static lint?
|
|
|
|
## Layout
|
|
|
|
data/ ignored by git: items.jsonl (copy of the 65 items), labelsheet.html, results.jsonl
|
|
(all contain or paraphrase passages from a copyrighted book)
|
|
labels/ tracked: key.json (blind id -> item id, no text), labels.json (verdicts and error
|
|
types only; do not paste passage text into the notes field)
|
|
results/ tracked: summary.json written by score.py (numbers only)
|
|
|
|
## Steps
|
|
|
|
cp runs/p85-110/items.jsonl experiments/P0-pilot/data/items.jsonl
|
|
python experiments/P0-pilot/make_sheet.py --items experiments/P0-pilot/data/items.jsonl \
|
|
--out experiments/P0-pilot/data --key-dir experiments/P0-pilot/labels --seed 7
|
|
# a person opens data/labelsheet.html, labels all 65 items, downloads labels.json into labels/
|
|
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
|
|
--out experiments/P0-pilot/data/results.jsonl --dry-run
|
|
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
|
|
--out experiments/P0-pilot/data/results.jsonl
|
|
python experiments/P0-pilot/score.py --results experiments/P0-pilot/data/results.jsonl \
|
|
--labels experiments/P0-pilot/labels/labels.json --key experiments/P0-pilot/labels/key.json \
|
|
--summary experiments/P0-pilot/results/summary.json
|
|
|
|
Labeling must be finished before anyone opens `results.jsonl`.
|