Give experiments IDs (P0, E0-E11): registry, per-experiment layout, spec moved to docs/experiments/P0-pilot.md
Raw pilot data stays ignored (copyrighted text); blind key and numeric results are tracked. Runner notes move into docs/infrastructure. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FYqwNMyPUHeTbFWg83GQq6
This commit is contained in:
@@ -0,0 +1,31 @@
|
||||
# P0: pilot
|
||||
|
||||
Experiment ID **P0**. Spec, hypotheses, decision rule and costs: `docs/experiments/P0-pilot.md`.
|
||||
Registry of all experiments: `docs/experiments/README.md`.
|
||||
|
||||
Question: on the 65 existing items, does role alignment (passage simplex vs. Lean simplex) find
|
||||
unfaithful formalizations better than a single LLM judge, compile-only, or static lint?
|
||||
|
||||
## Layout
|
||||
|
||||
data/ ignored by git: items.jsonl (copy of the 65 items), labelsheet.html, results.jsonl
|
||||
(all contain or paraphrase passages from a copyrighted book)
|
||||
labels/ tracked: key.json (blind id -> item id, no text), labels.json (verdicts and error
|
||||
types only; do not paste passage text into the notes field)
|
||||
results/ tracked: summary.json written by score.py (numbers only)
|
||||
|
||||
## Steps
|
||||
|
||||
cp runs/p85-110/items.jsonl experiments/P0-pilot/data/items.jsonl
|
||||
python experiments/P0-pilot/make_sheet.py --items experiments/P0-pilot/data/items.jsonl \
|
||||
--out experiments/P0-pilot/data --key-dir experiments/P0-pilot/labels --seed 7
|
||||
# a person opens data/labelsheet.html, labels all 65 items, downloads labels.json into labels/
|
||||
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
|
||||
--out experiments/P0-pilot/data/results.jsonl --dry-run
|
||||
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
|
||||
--out experiments/P0-pilot/data/results.jsonl
|
||||
python experiments/P0-pilot/score.py --results experiments/P0-pilot/data/results.jsonl \
|
||||
--labels experiments/P0-pilot/labels/labels.json --key experiments/P0-pilot/labels/key.json \
|
||||
--summary experiments/P0-pilot/results/summary.json
|
||||
|
||||
Labeling must be finished before anyone opens `results.jsonl`.
|
||||
Reference in New Issue
Block a user