Give experiments IDs (P0, E0-E11): registry, per-experiment layout, spec moved to docs/experiments/P0-pilot.md

Raw pilot data stays ignored (copyrighted text); blind key and numeric results are tracked. Runner notes move into docs/infrastructure. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FYqwNMyPUHeTbFWg83GQq6
2026-10-22 11:46:59 +00:00
parent 74a8162bb7
commit 2b255a333a
13 changed files with 231 additions and 42 deletions
@@ -0,0 +1,31 @@
+# P0: pilot
+
+Experiment ID **P0**. Spec, hypotheses, decision rule and costs: `docs/experiments/P0-pilot.md`.
+Registry of all experiments: `docs/experiments/README.md`.
+
+Question: on the 65 existing items, does role alignment (passage simplex vs. Lean simplex) find
+unfaithful formalizations better than a single LLM judge, compile-only, or static lint?
+
+## Layout
+
+    data/      ignored by git: items.jsonl (copy of the 65 items), labelsheet.html, results.jsonl
+               (all contain or paraphrase passages from a copyrighted book)
+    labels/    tracked: key.json (blind id -> item id, no text), labels.json (verdicts and error
+               types only; do not paste passage text into the notes field)
+    results/   tracked: summary.json written by score.py (numbers only)
+
+## Steps
+
+    cp runs/p85-110/items.jsonl experiments/P0-pilot/data/items.jsonl
+    python experiments/P0-pilot/make_sheet.py --items experiments/P0-pilot/data/items.jsonl \
+        --out experiments/P0-pilot/data --key-dir experiments/P0-pilot/labels --seed 7
+    # a person opens data/labelsheet.html, labels all 65 items, downloads labels.json into labels/
+    python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
+        --out experiments/P0-pilot/data/results.jsonl --dry-run
+    python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
+        --out experiments/P0-pilot/data/results.jsonl
+    python experiments/P0-pilot/score.py --results experiments/P0-pilot/data/results.jsonl \
+        --labels experiments/P0-pilot/labels/labels.json --key experiments/P0-pilot/labels/key.json \
+        --summary experiments/P0-pilot/results/summary.json
+
+Labeling must be finished before anyone opens `results.jsonl`.
@@ -0,0 +1,71 @@
+{
+  "seed": 7,
+  "items_file": "items.jsonl",
+  "key": {
+    "P01": "548ab1a767a685aefa146455",
+    "P02": "6ea85a17cb5decb0670c4afe",
+    "P03": "389a27cbae1957943fae2960",
+    "P04": "b63f0ed4d1ea1e11e34e6f8a",
+    "P05": "db4553333550bccbd3f0e21d",
+    "P06": "c3e75089fbfe8f386b4a22ec",
+    "P07": "91a6a0c1c21014bf08ffa3b6",
+    "P08": "675ac9dbf3ec8a0553149a96",
+    "P09": "8b39fc5b656bac00bc1e863f",
+    "P10": "3549ad65eee67f7b432a8b70",
+    "P11": "0b7db1a8100c33a50a08981f",
+    "P12": "e3d40d93cda13ec59c35d9a5",
+    "P13": "aaf1250a5104a4f7c1ca0d29",
+    "P14": "57966f39253aba58777a4ffd",
+    "P15": "6ef921c49672c73f8f3e0edf",
+    "P16": "371fe1b32fe04da354ad4478",
+    "P17": "a7396b1a5d5ebcd5b48ea7ba",
+    "P18": "59bb865c9bdfb8521e414743",
+    "P19": "283d407566264b02fbee6560",
+    "P20": "7de228e675b949cd64850c04",
+    "P21": "dc1c450987f3e3a5285260f7",
+    "P22": "c42dc687ecc042f5b7a00a9d",
+    "P23": "8491370b02906b6efdb3da5e",
+    "P24": "437d02fe69d580bde075648c",
+    "P25": "5fe4f3fbbead7e6988f015a9",
+    "P26": "8a55f59982e50e4af47b3d55",
+    "P27": "8d4d64143a8b214e38d7ba1b",
+    "P28": "6017927fdafd91ca22ed6ade",
+    "P29": "089c452ec2f9b850733a44f6",
+    "P30": "75009c3d616f02be4d7a5824",
+    "P31": "9091f6a277215174e821d373",
+    "P32": "0cd2f0368a0d8234c2593845",
+    "P33": "8e55e625fa9b10b7b65f59f4",
+    "P34": "534c6ffcf56931559e70657b",
+    "P35": "aa16561dceee591e35e42c4d",
+    "P36": "73ec95bfac2896218d2b1dab",
+    "P37": "6b5a55ec3bb3c85239919649",
+    "P38": "d6ddc1df3ce6b764f3fb8d73",
+    "P39": "ed87589326312890bea42d7c",
+    "P40": "d609c5bb746fc0e05625134a",
+    "P41": "4405f31187373d5e1de622d2",
+    "P42": "05a6defa23576b7da41f8fbf",
+    "P43": "47c39e88c29e3974e8864e90",
+    "P44": "fa61f8cb0e88b11e158c4eec",
+    "P45": "3c00c37b39285cbd7d0be732",
+    "P46": "186d922a4121917ac8643316",
+    "P47": "e845a95d7b5054a0dbdf34eb",
+    "P48": "e7a3cec23c2ecdc655dd12cf",
+    "P49": "e966f3b17e32318acd5f2762",
+    "P50": "2c420bf6ec32069ecd0043dc",
+    "P51": "25af34567cb2040bf41869e5",
+    "P52": "578ae89f353ecbdc0ffb76a2",
+    "P53": "1a8d6a5c9aa3cc5000dee474",
+    "P54": "450c7df89ecd284b0e08504a",
+    "P55": "f908ac15b1bf8228281bcf68",
+    "P56": "bcdfa01447e7060e1fd3d935",
+    "P57": "f86421f93f1381d2ce728bc1",
+    "P58": "89fe687db9136379cd276fff",
+    "P59": "da65aa56505f6c13e6553d0d",
+    "P60": "ab2990e9786d5bda5a04e067",
+    "P61": "a892779bfcaca35b91e7d611",
+    "P62": "7593062cab7c74cdecec08d0",
+    "P63": "c8204e9d19a6dd5d730dccc7",
+    "P64": "b40279b634fffd0841ff0f5a",
+    "P65": "f8d2e206681972927560be2c"
+  }
+}
@@ -1,13 +1,14 @@
 """Build the blind labeling sheet for the pilot.

-    python experiments/pilot/make_sheet.py --items runs/p85-110/items.jsonl \
-        --out runs/pilot/private --seed 7
+    python experiments/P0-pilot/make_sheet.py --items experiments/P0-pilot/data/items.jsonl \
+        --out experiments/P0-pilot/data --key-dir experiments/P0-pilot/labels --seed 7

-Writes, into a directory that must stay local (it contains passages from a
-copyrighted book; ``runs/`` and ``**/private/`` are gitignored):
+Writes:

-    labelsheet.html     open in a browser; label every item; press "Download labels.json"
-    key.json            blind id -> item id (keep it; the scorer needs it)
+    <out>/labelsheet.html    open in a browser; label every item; press "Download labels.json".
+                             Contains passages from a copyrighted book, so ``data/`` is gitignored.
+    <key-dir>/key.json       blind id -> item id. Text-free, so it is tracked in git; save the
+                             downloaded labels.json next to it.

 The sheet shows the passage and the Lean code. It deliberately hides whether Lean
 accepted the code, any model output, and any score, so the labels are independent of
@@ -146,7 +147,8 @@ def card(blind: str, item: dict) -> str:
 def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--items", required=True, type=Path)
-    parser.add_argument("--out", required=True, type=Path)
+    parser.add_argument("--out", required=True, type=Path, help="where labelsheet.html goes (ignored by git)")
+    parser.add_argument("--key-dir", type=Path, default=None, help="where key.json goes (default: ../labels)")
    parser.add_argument("--seed", type=int, default=7)
    args = parser.parse_args()

@@ -159,16 +161,18 @@ def main() -> int:

    legend = "".join(f"<li><b>{esc(n)}</b>: {esc(d)}</li>" for _, n, d in FIDELITY)
    cards = "\n".join(card(blind_ids[pos], items[idx]) for pos, idx in enumerate(order))
+    key_dir = args.key_dir or args.out.parent / "labels"
    args.out.mkdir(parents=True, exist_ok=True)
+    key_dir.mkdir(parents=True, exist_ok=True)
    (args.out / "labelsheet.html").write_text(
        PAGE.format(n=len(items), legend=legend, cards=cards, ids=json.dumps(blind_ids)),
        encoding="utf-8",
    )
-    (args.out / "key.json").write_text(
+    (key_dir / "key.json").write_text(
        json.dumps({"seed": args.seed, "items_file": args.items.name, "key": key}, indent=2),
        encoding="utf-8",
    )
-    print(f"wrote {args.out/'labelsheet.html'} ({len(items)} items) and key.json")
+    print(f"wrote {args.out/'labelsheet.html'} ({len(items)} items) and {key_dir/'key.json'}")
    return 0


@@ -1,8 +1,8 @@
 """Run the pilot detectors over the pilot items. Resumable; one JSON line per item.

-    python experiments/pilot/run_pilot.py --items runs/p85-110/items.jsonl \
-        --out runs/pilot/results.jsonl --dry-run          # no API calls: shows the plan
-    python experiments/pilot/run_pilot.py --items ... --out ... --max-calls 400
+    python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
+        --out experiments/P0-pilot/data/results.jsonl --dry-run          # no API calls: shows the plan
+    python experiments/P0-pilot/run_pilot.py --items ... --out ... --max-calls 400

 Detectors (all see the passage and/or the Lean code, never the human labels):
  lint    static patterns on the projected simplex          (no LLM)
@@ -1,7 +1,7 @@
 """Score the pilot: detectors vs. the blind human labels, with the pre-registered decision.

-    python experiments/pilot/score.py --results runs/pilot/results.jsonl \
-        --labels runs/pilot/private/labels.json --key runs/pilot/private/key.json
+    python experiments/P0-pilot/score.py --results experiments/P0-pilot/data/results.jsonl \
+        --labels experiments/P0-pilot/labels/labels.json --key experiments/P0-pilot/labels/key.json --summary experiments/P0-pilot/results/summary.json

 Positive class = unfaithful. Primary: verdict == "wrong", over ALL labeled items (role alignment and
 the judge do not need Lean to accept the code; only 10 of the 65 pilot items compiled, too few to
@@ -72,6 +72,8 @@ def main() -> int:
    ap.add_argument("--results", required=True, type=Path)
    ap.add_argument("--labels", required=True, type=Path)
    ap.add_argument("--key", required=True, type=Path)
+    ap.add_argument("--summary", type=Path, default=None,
+                    help="also write the (text-free) summary here, e.g. results/summary.json")
    args = ap.parse_args()
    rows = [r for r in load(args) if r["verdict"] != "not_formalizable"]
    compiled = [r for r in rows if r["compiled"]]
@@ -85,7 +87,11 @@ def main() -> int:
        "compiled_only": report(compiled, lambda r: r["verdict"] == "wrong"),
    }
    summary["decision"] = decide(summary["primary_all"])
-    print(json.dumps(summary, indent=2, default=str))
+    text = json.dumps(summary, indent=2, default=str)
+    if args.summary:
+        args.summary.parent.mkdir(parents=True, exist_ok=True)
+        args.summary.write_text(text + "\n", encoding="utf-8")
+    print(text)
    return 0


@@ -1,15 +0,0 @@
-# Pilot (Phase 0)
-
-Question: on the 65 existing items, does role alignment (passage simplex vs. Lean simplex) find
-unfaithful formalizations better than a single LLM judge, compile-only, or static lint?
-Full protocol and decision rule: `docs/experiments/PROTOCOL.md`.
-
-    python experiments/pilot/make_sheet.py --items runs/p85-110/items.jsonl --out runs/pilot/private
-    # a person opens runs/pilot/private/labelsheet.html, labels all items, saves labels.json
-    python experiments/pilot/run_pilot.py --items runs/p85-110/items.jsonl --out runs/pilot/results.jsonl --dry-run
-    python experiments/pilot/run_pilot.py --items runs/p85-110/items.jsonl --out runs/pilot/results.jsonl
-    python experiments/pilot/score.py --results runs/pilot/results.jsonl \
-        --labels runs/pilot/private/labels.json --key runs/pilot/private/key.json
-
-`runs/` and `**/private/` are gitignored: the passages come from a copyrighted book and must not be
-committed. Labeling must finish BEFORE `score.py` is run; do not look at `results.jsonl` while labeling.