55 lines
3.6 KiB
Markdown
55 lines
3.6 KiB
Markdown
# P0: pilot
|
|
|
|
Experiment ID **P0**. Spec, hypotheses, decision rule and costs: `docs/experiments/P0-pilot.md`.
|
|
Registry of all experiments: `docs/experiments/README.md`.
|
|
|
|
Question: on the 65 existing items, does role alignment (passage simplex vs. Lean simplex) find
|
|
unfaithful formalizations better than a single LLM judge, compile-only, or static lint?
|
|
|
|
## Layout
|
|
|
|
data/ ignored by git: items.jsonl (copy of the 65 items), labelsheet.html, results.jsonl
|
|
(all contain or paraphrase passages from a copyrighted book)
|
|
labels/ tracked: key.json (blind id -> item id, no text), labels.json (verdicts and error
|
|
types only; do not paste passage text into the notes field)
|
|
results/ tracked: summary.json written by score.py (numbers only)
|
|
|
|
## Steps
|
|
|
|
# Default exports omit source_text. Restore it from the matching private review.
|
|
python experiments/P0-pilot/prepare_items.py --items runs/p85-110/items.jsonl \
|
|
--review runs/p85-110/review.json --out experiments/P0-pilot/data/items.jsonl
|
|
python experiments/P0-pilot/make_sheet.py --items experiments/P0-pilot/data/items.jsonl \
|
|
--out experiments/P0-pilot/data --key-dir experiments/P0-pilot/labels --seed 7
|
|
# a person opens data/labelsheet.html, labels all 65 items, downloads labels.json into labels/
|
|
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
|
|
--out experiments/P0-pilot/data/results.jsonl --dry-run
|
|
python experiments/P0-pilot/run_pilot.py --items experiments/P0-pilot/data/items.jsonl \
|
|
--out experiments/P0-pilot/data/results.jsonl --labels experiments/P0-pilot/labels/labels.json
|
|
python experiments/P0-pilot/score.py --results experiments/P0-pilot/data/results.jsonl \
|
|
--labels experiments/P0-pilot/labels/labels.json --key experiments/P0-pilot/labels/key.json \
|
|
--summary experiments/P0-pilot/results/summary.json
|
|
|
|
Labeling must be finished before anyone opens `results.jsonl`.
|
|
|
|
## Execution safeguards
|
|
|
|
- Every item must have a nonempty source passage. `prepare_items.py` restores passages only
|
|
when all exported fields match the archived review; the exported cohort and Lean code stay intact.
|
|
- Freeze all labels before the paid run. The owner changed the labeler to the Codex assistant
|
|
on 2026-10-28. These are **AI labels, not human gold**; see the metadata in `labels/labels.json`.
|
|
- `--max-calls` counts actual HTTP attempts, including transport and JSON retries, cumulatively
|
|
across resumes. The hard ceiling is 400. A reservation is saved before each request; an interrupted
|
|
request may consume a reservation without a known response or token count.
|
|
- Metadata is stored beside the output as `results.meta.json`. Model, endpoint, temperature,
|
|
input hash, frozen-label hash and prompt hashes must match on resume. Usage is cumulative.
|
|
A file lock prevents concurrent writers. A budget stop returns exit status 2.
|
|
- The nominal successful path uses 130 calls (one source extraction and one judge per item);
|
|
malformed replies and network retries may increase the actual count.
|
|
- Scoring requires complete labels and results and no unresolved detector errors. The primary
|
|
comparison is wrong versus faithful; minor drift is included as positive in the secondary analysis.
|
|
The complementary-signal PARTIAL rule counts wrong items flagged by role alignment but missed
|
|
by the judge. No threshold, concept table or detector prompt was tuned for these labels.
|
|
- Raw passages, labeling HTML and model responses remain in the ignored `data/` directory.
|
|
Only verdicts, abstract annotation notes, hashes and text-free summaries belong in Git.
|