diff --git a/docs/TODO.md b/docs/TODO.md new file mode 100644 index 0000000..ecd2dc7 --- /dev/null +++ b/docs/TODO.md @@ -0,0 +1,87 @@ +# lean-bench paper TODO (target: ACL 2027 via ARR, deadline 4 Jan 2027 AoE; submit by 3 Jan) + +Companion to `docs/research-plan.md`. Priorities: **P0** blocks the paper, **P1** needed for a strong submission, **P2** only if time allows. Dates are targets. Updated 2026-10-21. + +Working claim: *a simplex schema derived from the Lean signature gives a cheap, structural, human-calibrated fidelity signal (role alignment) and guides formalization and retrieval; compiling alone overstates quality.* + +--- + +## A. Next 7 days (21-28 Oct) +- [ ] **P0** Read in full and take notes on the five closest papers: Beyond Compilation (2606.31002), ShadowBench (2608.29270), Lean-GAP (2606.02588), ATLAS (2605.29955), CriticLean (2507.06181). One page each: data, metric, numbers, what they do not do. +- [ ] **P0** Decide the minimal simplex implementation: re-implement `Role` / `Simplex` / `State` inside `leanbench/` (do not import the archived code). Write the design note. +- [ ] **P0** Fix the overloaded `unverified` label -> `rejected_by_lean` vs `lean_not_run` (+ test). +- [ ] **P0** `source_pages` coverage table (extractor, quality score, OCR ran/failed, timestamp) (+ test). +- [ ] **P0** Incremental per-item storage so a crash keeps finished work (+ test). +- [ ] **P0** Confirm licences for candidate open books (see C1). Shortlist 6-8. +- [ ] **P1** Check Lean-GAP's Kaggle terms: can the human-signed statements be used as an external gold set? +- [ ] **P1** Set up a weekly arXiv/Google Scholar alert for: autoformalization faithfulness; textbook Lean; semantic alignment Lean. +- [ ] **P1** Ask the owner for two annotators (Lean-literate) and agree pay/time. + +## B. Code: simplex schema and alignment (weeks 1-4) +- [ ] **P0** `leanbench/schema.py`: `Role(name, context, choices, attrs)`, `Simplex` (validation inside `__init__`, >= 2 roles, nesting), `Fact`/`Goal` with `lean_type`. Avoid the archived defects (dead `__post_init__`, missing `Role.__init__` call, absolute import). +- [ ] **P0** Projection `lean_code -> Simplex` extending `typeindex.py`: explicit binders, instance binders attached to their object, hypothesis binders as Fact roles, conclusion as Goal role, `binder_kind` for quantifiers. Tests on >= 100 real signatures. **Acceptance: >= 95% structural agreement with manual parse (E0).** +- [ ] **P0** Source-side extraction `prose -> Simplex` (one forced tool call, `minItems: 2`, search-first role reuse). Prompt versioned and hashed. +- [ ] **P0** Alignment `A(S_T, S_L)`: matching on name similarity, kind, type compatibility; outputs matched / missing / extra / conflicts + typed error list. Deterministic and unit-tested. +- [ ] **P1** Role registry in SQLite with `search_roles`, `find_by_roles`, `find_solvable`; migrate from archived `schema_registry.py` with tests for nesting depth > 1. +- [ ] **P1** Slot-constrained formalization mode (`--mode slots`): model fills binder strings per role, declaration assembled deterministically; slot-level repair from Lean diagnostics ranges. +- [ ] **P1** Ablation switches in the CLI: window, extractor, repair attempts, slot mode, closed vs open `choices`, schema key wording variants. +- [ ] **P2** Port `AnalogySchemaMiner`; run only after main results. +- [ ] **P1** Pin Lean/Mathlib image; add `lean-bench doctor` check that logs toolchain + Mathlib revision into every run record. +- [ ] **P1** Reproducibility: run record per ingest (model ids, prompts hash, seeds, image id) stored in the DB. + +## C. Data (weeks 1-6) +- [ ] **P0** C1. Choose 6-8 books (algebra, analysis, topology, number theory, combinatorics; >= 3 openly licensed; one Chinese-language for 7.7). Record licence, URL, hash. Candidates already used by others: Lebl *Basic Analysis*, McKay *Topology*, Doty *Abstract Algebra* (LaTeX, open) **[confirm licences]**; keep `uiowa-algebra.pdf` out of every release. +- [ ] **P0** Ingest all books (E1). Target >= 1,500 items. Record cost and time per page. +- [ ] **P0** Draw stratified samples: 150 pages for detection gold (E2), 600 items for the audited core (E5), 150 items for gold source-side simplices. +- [ ] **P0** Annotation tool: page crop + Lean code + both simplices side by side; blinded to model id and filter score; stores annotator id and time. +- [ ] **P0** Annotation guideline v1 with 30 worked examples and the error taxonomy (5.2). Pilot 50 items, compute kappa, revise. **Gate: pilot kappa >= 0.6.** +- [ ] **P1** Collect natural error examples (not synthetic) for the taxonomy figure. +- [ ] **P1** Build the release package structure (hashes + spans + Lean + simplex annotations; no copyrighted text). + +## D. Experiments (weeks 3-10), in run order +- [ ] **P0** E0 projection coverage (>= 95%). +- [ ] **P0** E2 detection: P/R/F1, boundary IoU, cross-page, 5-repeat Jaccard; ablate window/extractor/model. +- [ ] **P1** E3 OCR: embedded vs tesseract vs vision vs one math-OCR; `auto` heuristic ROC. +- [ ] **P0** E4 compile vs faithful across >= 4 general LLMs + 1 specialised formalizer, same budget; paired McNemar; bootstrap CIs; stratify by length/kind/domain/context dependency. +- [ ] **P0** E5 human labels on the 600 core; kappa on 200 overlap; third adjudicator. +- [ ] **P0** E6 filter comparison: no filter, type-check, single judge, two-judge, BEq+, static checkers, CriticLean/FormalAlign, trajectory length, role alignment, role alignment + judge. Metrics: P/R/F1, AUC, recall of wrong-but-compiles at 10% false-reject, per-error-type recall. Controls: shuffled spans, same-family judge. +- [ ] **P0** E7 slot-constrained vs free-form (same model, same budget). +- [ ] **P0** E8 repair ablation 0/1/3/5, whole vs slot-level; fraction of newly compiling items that are faithful. +- [ ] **P1** E9 retrieval/guidance: no retrieval, BM25, embedding, LeanSearch v2, structure-aware retriever baseline (2510.23637-style), type index, `find_solvable`, oracle; 2 open provers + 1 API; pass@1 / pass@8. +- [ ] **P1** E10 contamination check. +- [ ] **P1** E11 cost/time/reliability table, including the silent-failure checks. +- [ ] **P1** External check on Lean-GAP gold (if licence allows) and a spot check on ATLAS outputs. +- [ ] **P2** 7.1 role vocabulary growth/dedup; 7.2 dependency graph; 7.3 analogy mining; 7.4 judge-bias matrix; 7.5 robustness; 7.7 Chinese book + homogenization; 7.8 audit-time study. +- [ ] **P0** **Go/no-go on 8 Dec**: if role alignment does not beat a single judge on at least two error types at equal false-reject, drop contribution C2 as a headline and submit as evaluation + taxonomy + provenance. +- [ ] **P0** Freeze all tables on 24 Dec. + +## E. Writing (start 1 Dec; related work and method now) +- [ ] **P0** Related-work section from the notes in `research-plan.md` section 4 (all `[verify]` tags cleared). +- [ ] **P0** Method section: definitions (5.1), taxonomy (5.2), slot decoding (5.3), alignment (5.4), retrieval (5.5). +- [ ] **P0** Introduction (draft exists in section 3); rewrite after E4 numbers. +- [ ] **P0** Figures: F1 overview (page crop + two simplices), F2 gap by model, F3 error taxonomy, F4 ROC/recall at fixed false-reject, F5 retrieval/guidance. +- [ ] **P0** Tables: T1 stats, T2 gap, T3 filter comparison, T4 ablations. +- [ ] **P0** Limitations (LLM dependence of source-side extraction, one proof assistant, few books, annotator expertise, judges proprietary). +- [ ] **P0** Responsible NLP checklist, data statement, annotator pay and instructions, licence statement. +- [ ] **P1** Anonymized repository + artifact package; remove identity from images, paths, commits. +- [ ] **P1** Internal read-through by someone outside the project, 24-27 Dec. +- [ ] **P0** Format on ACL template, 8 pages; submit to ARR by 3 Jan 2027. + +## F. Risks to watch (check weekly) +- [ ] New arXiv papers overlapping role-level or structural fidelity checks. If one appears: read within 24 h, update the positioning table, decide whether C2 still stands. +- [ ] ICML 2027 / NeurIPS D&B 2027 official CFPs (dates are projected today). +- [ ] ITP 2027 CFP (Glasgow, ~2-7 Aug 2027) as fallback. +- [ ] Annotator availability; Lean compute (second runner if E4/E7/E8 exceed ~100 CPU-hours). +- [ ] API cost against the 135M-token estimate (recompute after pilot). + +## G. Milestones +| Date | Milestone | +|---|---| +| 3 Nov | Items in A and B-core done; E0 passes; books chosen | +| 17 Nov | All books ingested; annotation pilot kappa >= 0.6 | +| 1 Dec | E2, E3, E4 finished; core labelling under way; writing starts | +| 8 Dec | Go/no-go on role alignment | +| 15 Dec | E5-E8 finished | +| 24 Dec | Tables frozen; E9-E11 finished; full draft | +| 27 Dec | Internal read-through done | +| 3 Jan 2027 | Submit to ARR (deadline 4 Jan AoE) | diff --git a/docs/research-plan.md b/docs/research-plan.md index 54933df..0b163c3 100644 --- a/docs/research-plan.md +++ b/docs/research-plan.md @@ -93,7 +93,7 @@ Evidence base: repo README and code (`leanbench/`, 4,345 lines incl. tests), pro ### 3.1 Paragraph plan 1. **Hook.** Formal libraries grow slowly because turning human mathematics into Lean is expensive; textbooks hold most of undergraduate and graduate mathematics, and LLMs now translate statements fluently. -2. **The measurement problem.** Systems and benchmarks are scored by whether Lean accepts the output. But Lean checks types, not meaning: a statement missing a hypothesis, with `∃` for `∀`, or over `ℕ` instead of `ℤ` compiles and is wrong. Concurrent work already shows large gaps on curated statement sets (cite 2606.31002, 2608.29270 **[verify numbers]**); no one has measured it on raw textbooks where extraction error compounds with formalization error. +2. **The measurement problem.** Systems and benchmarks are scored by whether Lean accepts the output. But Lean checks types, not meaning: a statement missing a hypothesis, with `∃` for `∀`, or over `ℕ` instead of `ℤ` compiles and is wrong. Concurrent work already shows large gaps on curated statement sets (Beyond Compilation: up to 29.5 points; ShadowBench: 61.8% compile vs 11.2% SA-Pass), and textbook pipelines report faithfulness only through uncalibrated LLM scores (ATLAS, Lean-GAP). What is missing is a cheap, structural check that says *which parameter* is wrong, calibrated against humans, on pages where extraction error compounds with formalization error. 3. **Why textbooks are harder.** Font-mangled PDFs (`D` for `=`, `C` for `+`), items split across pages, definitions that depend on context. Each stage can break meaning silently: in our own pipeline the first server run reported success while storing 0 items. 4. **Our observation.** A Lean declaration's parameters are a small typed schema: given objects, hypotheses, a goal. The same schema exists in the prose (what is assumed, what is claimed). Comparing the two *role by role* is a structural check that localizes the error. 5. **Approach.** Represent both sides as simplices; use the schema to constrain formalization, to compare, and to retrieve. @@ -128,18 +128,25 @@ References marked **[verify]** are from memory/search snippets; confirm authors, - **miniF2F**, **PutnamBench**, **FormalMATH**, **CombiBench**, **Lean Workbook**: competition-style or synthetic. **[verify each]** - **ProofNet** (Azerbayev et al., 2023; undergraduate textbook theorems; [arXiv 2302.12433](https://arxiv.org/pdf/2302.12433)): closest prior. Hand-curated, statements only, no extraction from PDFs. - **ProofNet#, RLM25, ProofNetVerif** (Poiroux et al., [arXiv 2406.07222](https://arxiv.org/abs/2406.07222)): fixed 118 of 371 ProofNet entries; found type-check rate correlates only weakly (Pearson 0.655) with human accuracy; proposed BEq/BEq+ symbolic equivalence (BEq+ precision 98.0%, recall 48.3%). -- Concurrent 2026 work: **Beyond Compilation** ([arXiv 2606.31002](https://arxiv.org/abs/2606.31002); 227 graduate statements; faithfulness = compile + two-LLM agreement; judge vs human agreement 91.5%; compile-faithful gap from 1.3 to 29.5 points across 8 systems) and **ShadowBench / SA-Pass** ([arXiv 2608.29270](https://arxiv.org/abs/2608.29270); 178 postgraduate-research problems; "shadow" statements checked forward and backward; compile 61.8% vs SA-Pass 11.2% for the best agentic setup). **[verify numbers against the final versions]** -- *How we differ:* these start from a clean list of statements. We start from PDF pages, include definitions and exercises, keep page-level provenance, and measure how extraction error adds to formalization error. +- **Beyond Compilation** ([arXiv 2606.31002](https://arxiv.org/abs/2606.31002), read in full 2026-10-21). 227 graduate statements taken from **four open LaTeX textbooks** (Lebl *Basic Analysis* 48, Lebl *Complex Analysis* 19, McKay *Topology* 63, Doty *Abstract Algebra* 97). Faithful = compiles AND both judges (GPT-5.2, Gemini-2.5-Pro) grade >= 9/10. Human audit: positive precision 67/70 = 95.7%; of 67 compile-pass outputs the rule rejected, humans confirmed 54 (80.6%); a 59-case Aristotle batch gave 91.5% agreement. Beats LeanScorer (76.3% vs 91.5% on that batch; McNemar p = 0.0225). BEq cross-check certified 116/189 pairs (61.4%). Eight systems: full tool-augmented GPT-5.2 agent compiles 87.2% but is faithful on 57.7% (gap 29.5); one-shot general LLMs faithful 19.8-27.3%; specialised formalizers 10.1-13.8%; prover-oriented 4.6-6.2%. Ablation: Lean feedback adds +49.3 compile but +22.1 of that is unfaithful; **faithfulness falls from 82.9% (1-2 agent steps) to 11.5% (19-24 steps)**. Stated limits: proprietary judges, same-family orchestrator, 227 items, no kappa reported in the visible text. Code/data: github.com/grenadeComing/beyond-compilation. +- **ShadowBench / SA-Pass** ([arXiv 2608.29270](https://arxiv.org/abs/2608.29270), read 2026-10-21). 178 postgraduate/research problems from textbooks, notes, papers, Mathlib, Stacks, EGA. A *shadow* is a theorem implied by the intended statement; a candidate must compile, imply every shadow and be implied by their conjunction. Shadows are drafted by Qwen3-235B and completed by experts (706 checkers, ~10 min per problem, hidden from submitters). Best system (Claude Code Opus 4.8 + Numina) compile 61.8% vs SA-Pass 11.2%; non-agentic LLMs 0.0% SA-Pass. Agreement with two experts: SA-Pass 0.988, BEq+ 0.806, LLM-judge 0.785, compile 0.178. **On ProofNet the gap is only 2.1 points on average**, which the authors attribute to short single-conclusion targets. Cost of the method: per-problem shadow authoring. +- **Lean-GAP** ([arXiv 2606.02588](https://arxiv.org/html/2606.02588v1)). **Closest to our pipeline.** 430 of 1,966 Dummit & Foote *Abstract Algebra* exercises: scanned pages -> Mathpix OCR -> LLM -> Lean `sorry` statements, human contributors, two-stage human sign-off, 11 "suspicious-statement" static patterns (C1-C11), Claude Opus 4.7 five-axis judge (uncalibrated). Six systems; GPT-5 elaborates 44.1% and is judged faithful 45.1%; Codex agent elaborates 95.5% but faithful 51.2%. Released on Kaggle with Wiley's permission for non-commercial research **[confirm terms]**. +- **ATLAS / AutoformBot** ([arXiv 2605.29955](https://www.alphaxiv.org/abs/2605.29955)). 26 open-access graduate textbooks, > 45,000 Lean declarations, ~500k lines, 71.3% formalization rate; faithfulness only through an LLM 0-5 harness (threshold 3/5). No human calibration reported in the summary I read. 26 separate projects, not Mathlib-compatible. +- **LeanFlow** ([arXiv 2607.20503](https://arxiv.org/html/2607.20503v1)): paper -> Lean workflow with a blueprint mapping source spans to declarations and an LLM statement-review gate; states that "no automated faithfulness check exists". Two papers only. +- **Faults in Our Formal Benchmarking** ([arXiv 2606.29493](https://arxiv.org/html/2606.29493v1)): audits miniF2F, ProofNet, FormalMath, CombiBench, ProverBench (~10,000 problems), 4,833 findings, 398 with machine-checkable certificates; fidelity defects include missing hypotheses, vacuous hypotheses, `ℕ` subtraction truncation, division by zero. Its checkers are cheap static baselines for us. +- **How we differ, stated carefully.** Textbook-to-Lean pipelines (Lean-GAP, ATLAS, LeanFlow) and faithfulness measurement (Beyond Compilation, ShadowBench) both exist. What none of them does: (a) a **structural, role-level fidelity check that localizes which parameter is wrong and needs no per-item authoring** (shadows) **and no extra judge call**; (b) **calibrating** that check against humans at the role level; (c) **page-level provenance with a coverage record** so extraction and formalization errors are separable; (d) **schema-guided generation and retrieval** built from the same roles. The paper must not be sold as "first textbook autoformalization". ### 4.2 Autoformalization methods - Few-shot autoformalization (Wu et al., 2022 **[verify]**); sampling + type-check filtering + selection ([2406.07222](https://arxiv.org/abs/2406.07222)); Herald, Kimina-Autoformalizer, Goedel-Prover, DeepSeek-Prover-V2 **[verify each]**; agentic Lean workflows (e.g. Numina-Lean-Agent as used in ShadowBench **[verify]**). - *How we differ:* we do not propose another model. We propose a schema layer that constrains and audits any model, and treat these systems as baselines. ### 4.3 Faithfulness and equivalence checking -- **FormalAlign** (Lu et al., [ICLR 2025](https://proceedings.iclr.cc/paper_files/paper/2025/hash/fceedf8c9c0ff51f41b9fe0294ef0070-Abstract-Conference.html); [arXiv 2410.10135](https://arxiv.org/pdf/2410.10135)): learned alignment scorer. -- **BEq / BEq+**: symbolic, high precision, low recall (above). -- **LLM-judge ensembles** (Beyond Compilation) and **shadow statements** (ShadowBench): behavioural checks that need extra LLM calls or hand-written shadows. -- Back-translation / round-trip consistency **[verify specific papers]**. +- **FormalAlign** (Lu et al., [ICLR 2025](https://proceedings.iclr.cc/paper_files/paper/2025/hash/fceedf8c9c0ff51f41b9fe0294ef0070-Abstract-Conference.html); [arXiv 2410.10135](https://arxiv.org/pdf/2410.10135)): Mistral-7B fine-tuned with cross-entropy + contrastive loss; score = mean of certainty and cosine similarity, threshold 0.7; trained on synthetic misalignments (~21 negatives per positive); trails GPT-4 on FormL4-Random and MiniF2F-Test; human study of 80 items with Fleiss kappa 0.49. Lesson for us: **synthetic negatives do not match real errors, so evaluate on naturally occurring errors from textbook pages**. +- **BEq / BEq+** (Poiroux et al.): symbolic, precision 98.0%, recall 48.3%. **TransTED / ASSESS** ([arXiv 2509.22246](https://arxiv.org/html/2509.22246); 1,247 expert-annotated pairs; accuracy 67-70%; ~51 s per pair; threshold varies 0.38-0.61 by benchmark; false positives on near-misses). +- **CriticLean** ([arXiv 2507.06181](https://arxiv.org/pdf/2507.06181)): trained critic for faithfulness; CriticLeanBench 500 pairs; Qwen3-32B-RL 87.0% vs Gemini 2.5 Pro 89.2%; critic raised Kimina-Autoformalizer-7B from 54% (compiler feedback) to 84% on a 50-problem human check; ~47% of problems still failed in 200 attempts. +- **LLM-judge ensembles** (Beyond Compilation) and **shadow statements** (ShadowBench): behavioural checks that need extra LLM calls or authored shadows (~10 min per item). +- Static fidelity checkers: Lean-GAP's C1-C11 patterns and the checkers of Faults in Our Formal Benchmarking. +- Back-translation / round-trip consistency: used inside several of the above; **[locate a specific citation before claiming it]**. - *How we differ:* role alignment is structural and gives a typed explanation of *which* part of the statement is wrong. It is cheap (one extraction call + deterministic matching), and complements, not replaces, BEq and judges. We evaluate all of them side by side (E6). ### 4.4 Structured / schema-guided reasoning with LLMs @@ -161,8 +168,11 @@ References marked **[verify]** are from memory/search snippets; confirm authors, |---|---|---|---|---|---| | ProofNet(#) | curated | human | high | n/a | no | | BEq+ | statement pairs | symbolic | CPU | no | no | -| LLM judges | statement pairs | LLM | calls | no (text rationale) | no | -| ShadowBench | curated | shadows | authoring | partly | no | +| TransTED / ASSESS | statement pairs | tree edit + tactics | ~51 s/pair | no | no | +| LLM judges (Beyond Compilation) | LaTeX textbook statements | 2-judge agreement | 2 calls | no (text rationale) | LaTeX, not PDF | +| ShadowBench | curated | shadows | ~10 min authoring/item | partly | no | +| Lean-GAP | scanned pages (Mathpix) | human sign-off + uncalibrated LLM judge | high human cost | no | yes, exercises only | +| ATLAS | 26 open textbooks | uncalibrated LLM 0-5 | calls | no | yes, no human calibration | | **Ours** | **PDF pages** | **role alignment (+ judges)** | **1 call + CPU** | **yes, by role** | **yes** | --- @@ -276,6 +286,10 @@ Common protocol: Lean/Mathlib image pinned (Lean 4.33.1 image from the project); 6. shadow-style check on 50 hand-built items (small, illustrative) 7. **role alignment** (ours) 8. role alignment + judge + 9. static suspicious-pattern checkers (Lean-GAP C1-C11 style; Faults-in-Formal-Benchmarking style): cheap baseline that catches vacuity and `ℕ`-subtraction hazards + 10. a trained critic (CriticLean released model, if weights are available) and FormalAlign + 11. agent-trajectory-length signal (Beyond Compilation found faithfulness collapses with many repair steps): a free baseline from our own repair logs +- *External gold set (saves annotation):* if the Lean-GAP release permits, score role alignment against its human-signed exercise statements and its model outputs; if the terms forbid redistribution, use it only locally for evaluation and report aggregate numbers. **[confirm Kaggle terms]** - *Metrics:* precision, recall, F1 for "faithful"; ROC-AUC for alignment score; recall of "wrong-but-compiles" at fixed false-reject 10%; per-error-type recall (missing hypothesis, extra hypothesis, type, quantifier, goal); cost per item (calls, CPU). - *Controls:* shuffled spans (alignment must fail), identity (Lean vs its own back-translation), and judge self-preference (judge from same family as generator vs different). - *Quality of S_T itself:* role precision/recall of source-side extraction against the 150 gold simplices; sensitivity to extractor model. @@ -369,7 +383,8 @@ Writing starts 1 Dec in parallel; related work and method sections can be drafte | Archived simplex code needs more rework than planned | medium | Re-implement the minimal Role/Simplex/State in `leanbench/` rather than porting everything; analogy miner stays optional | | Compute for Lean at scale | medium | Second runner; reduce arms to core; cache by `sha1(code)` | | Licensing blocks release | medium | Open books only; hashes-only for others | -| Concurrent work (2606.31002, 2608.29270) overlaps | high | Cite and compare directly; emphasise PDF-to-Lean pipeline, provenance and role-level localization; recheck arXiv weekly until submission | +| Overlap with Beyond Compilation, ShadowBench, Lean-GAP, ATLAS (all 2026) | **high** | Do not claim first textbook pipeline or first gap measurement. Claim role-level, authoring-free, human-calibrated localization + schema guidance + provenance/coverage. Compare against their metrics directly. Recheck arXiv weekly (search: autoformalization faithfulness, textbook Lean) until submission | +| ShadowBench finds the gap is small on short ProofNet-style targets (2.1 pts) | medium | Stratify E4 by statement length/complexity and report where role alignment helps; do not generalise from long items to all items | | Dates shift | low-medium | ICML/NeurIPS dates are projected; recheck CFPs monthly | ### 8.3 ACL / ARR checklist