Pilot prep: simplex schema, projection, alignment, lint, stats, pilot scripts, protocol v1

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FYqwNMyPUHeTbFWg83GQq6
2026-10-22 05:52:28 +00:00
parent 3e0782c62a
commit ca98703538
19 changed files with 2024 additions and 0 deletions
@@ -0,0 +1,15 @@
+# Pilot (Phase 0)
+
+Question: on the 65 existing items, does role alignment (passage simplex vs. Lean simplex) find
+unfaithful formalizations better than a single LLM judge, compile-only, or static lint?
+Full protocol and decision rule: `docs/experiments/PROTOCOL.md`.
+
+    python experiments/pilot/make_sheet.py --items runs/p85-110/items.jsonl --out runs/pilot/private
+    # a person opens runs/pilot/private/labelsheet.html, labels all items, saves labels.json
+    python experiments/pilot/run_pilot.py --items runs/p85-110/items.jsonl --out runs/pilot/results.jsonl --dry-run
+    python experiments/pilot/run_pilot.py --items runs/p85-110/items.jsonl --out runs/pilot/results.jsonl
+    python experiments/pilot/score.py --results runs/pilot/results.jsonl \
+        --labels runs/pilot/private/labels.json --key runs/pilot/private/key.json
+
+`runs/` and `**/private/` are gitignored: the passages come from a copyrighted book and must not be
+committed. Labeling must finish BEFORE `score.py` is run; do not look at `results.jsonl` while labeling.