docs: verified literature review, gap analysis and paper TODO list
ci / python (push) Has been cancelled
ci / python (push) Has been cancelled
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FYqwNMyPUHeTbFWg83GQq6
This commit is contained in:
@@ -0,0 +1,87 @@
|
||||
# lean-bench paper TODO (target: ACL 2027 via ARR, deadline 4 Jan 2027 AoE; submit by 3 Jan)
|
||||
|
||||
Companion to `docs/research-plan.md`. Priorities: **P0** blocks the paper, **P1** needed for a strong submission, **P2** only if time allows. Dates are targets. Updated 2026-10-21.
|
||||
|
||||
Working claim: *a simplex schema derived from the Lean signature gives a cheap, structural, human-calibrated fidelity signal (role alignment) and guides formalization and retrieval; compiling alone overstates quality.*
|
||||
|
||||
---
|
||||
|
||||
## A. Next 7 days (21-28 Oct)
|
||||
- [ ] **P0** Read in full and take notes on the five closest papers: Beyond Compilation (2606.31002), ShadowBench (2608.29270), Lean-GAP (2606.02588), ATLAS (2605.29955), CriticLean (2507.06181). One page each: data, metric, numbers, what they do not do.
|
||||
- [ ] **P0** Decide the minimal simplex implementation: re-implement `Role` / `Simplex` / `State` inside `leanbench/` (do not import the archived code). Write the design note.
|
||||
- [ ] **P0** Fix the overloaded `unverified` label -> `rejected_by_lean` vs `lean_not_run` (+ test).
|
||||
- [ ] **P0** `source_pages` coverage table (extractor, quality score, OCR ran/failed, timestamp) (+ test).
|
||||
- [ ] **P0** Incremental per-item storage so a crash keeps finished work (+ test).
|
||||
- [ ] **P0** Confirm licences for candidate open books (see C1). Shortlist 6-8.
|
||||
- [ ] **P1** Check Lean-GAP's Kaggle terms: can the human-signed statements be used as an external gold set?
|
||||
- [ ] **P1** Set up a weekly arXiv/Google Scholar alert for: autoformalization faithfulness; textbook Lean; semantic alignment Lean.
|
||||
- [ ] **P1** Ask the owner for two annotators (Lean-literate) and agree pay/time.
|
||||
|
||||
## B. Code: simplex schema and alignment (weeks 1-4)
|
||||
- [ ] **P0** `leanbench/schema.py`: `Role(name, context, choices, attrs)`, `Simplex` (validation inside `__init__`, >= 2 roles, nesting), `Fact`/`Goal` with `lean_type`. Avoid the archived defects (dead `__post_init__`, missing `Role.__init__` call, absolute import).
|
||||
- [ ] **P0** Projection `lean_code -> Simplex` extending `typeindex.py`: explicit binders, instance binders attached to their object, hypothesis binders as Fact roles, conclusion as Goal role, `binder_kind` for quantifiers. Tests on >= 100 real signatures. **Acceptance: >= 95% structural agreement with manual parse (E0).**
|
||||
- [ ] **P0** Source-side extraction `prose -> Simplex` (one forced tool call, `minItems: 2`, search-first role reuse). Prompt versioned and hashed.
|
||||
- [ ] **P0** Alignment `A(S_T, S_L)`: matching on name similarity, kind, type compatibility; outputs matched / missing / extra / conflicts + typed error list. Deterministic and unit-tested.
|
||||
- [ ] **P1** Role registry in SQLite with `search_roles`, `find_by_roles`, `find_solvable`; migrate from archived `schema_registry.py` with tests for nesting depth > 1.
|
||||
- [ ] **P1** Slot-constrained formalization mode (`--mode slots`): model fills binder strings per role, declaration assembled deterministically; slot-level repair from Lean diagnostics ranges.
|
||||
- [ ] **P1** Ablation switches in the CLI: window, extractor, repair attempts, slot mode, closed vs open `choices`, schema key wording variants.
|
||||
- [ ] **P2** Port `AnalogySchemaMiner`; run only after main results.
|
||||
- [ ] **P1** Pin Lean/Mathlib image; add `lean-bench doctor` check that logs toolchain + Mathlib revision into every run record.
|
||||
- [ ] **P1** Reproducibility: run record per ingest (model ids, prompts hash, seeds, image id) stored in the DB.
|
||||
|
||||
## C. Data (weeks 1-6)
|
||||
- [ ] **P0** C1. Choose 6-8 books (algebra, analysis, topology, number theory, combinatorics; >= 3 openly licensed; one Chinese-language for 7.7). Record licence, URL, hash. Candidates already used by others: Lebl *Basic Analysis*, McKay *Topology*, Doty *Abstract Algebra* (LaTeX, open) **[confirm licences]**; keep `uiowa-algebra.pdf` out of every release.
|
||||
- [ ] **P0** Ingest all books (E1). Target >= 1,500 items. Record cost and time per page.
|
||||
- [ ] **P0** Draw stratified samples: 150 pages for detection gold (E2), 600 items for the audited core (E5), 150 items for gold source-side simplices.
|
||||
- [ ] **P0** Annotation tool: page crop + Lean code + both simplices side by side; blinded to model id and filter score; stores annotator id and time.
|
||||
- [ ] **P0** Annotation guideline v1 with 30 worked examples and the error taxonomy (5.2). Pilot 50 items, compute kappa, revise. **Gate: pilot kappa >= 0.6.**
|
||||
- [ ] **P1** Collect natural error examples (not synthetic) for the taxonomy figure.
|
||||
- [ ] **P1** Build the release package structure (hashes + spans + Lean + simplex annotations; no copyrighted text).
|
||||
|
||||
## D. Experiments (weeks 3-10), in run order
|
||||
- [ ] **P0** E0 projection coverage (>= 95%).
|
||||
- [ ] **P0** E2 detection: P/R/F1, boundary IoU, cross-page, 5-repeat Jaccard; ablate window/extractor/model.
|
||||
- [ ] **P1** E3 OCR: embedded vs tesseract vs vision vs one math-OCR; `auto` heuristic ROC.
|
||||
- [ ] **P0** E4 compile vs faithful across >= 4 general LLMs + 1 specialised formalizer, same budget; paired McNemar; bootstrap CIs; stratify by length/kind/domain/context dependency.
|
||||
- [ ] **P0** E5 human labels on the 600 core; kappa on 200 overlap; third adjudicator.
|
||||
- [ ] **P0** E6 filter comparison: no filter, type-check, single judge, two-judge, BEq+, static checkers, CriticLean/FormalAlign, trajectory length, role alignment, role alignment + judge. Metrics: P/R/F1, AUC, recall of wrong-but-compiles at 10% false-reject, per-error-type recall. Controls: shuffled spans, same-family judge.
|
||||
- [ ] **P0** E7 slot-constrained vs free-form (same model, same budget).
|
||||
- [ ] **P0** E8 repair ablation 0/1/3/5, whole vs slot-level; fraction of newly compiling items that are faithful.
|
||||
- [ ] **P1** E9 retrieval/guidance: no retrieval, BM25, embedding, LeanSearch v2, structure-aware retriever baseline (2510.23637-style), type index, `find_solvable`, oracle; 2 open provers + 1 API; pass@1 / pass@8.
|
||||
- [ ] **P1** E10 contamination check.
|
||||
- [ ] **P1** E11 cost/time/reliability table, including the silent-failure checks.
|
||||
- [ ] **P1** External check on Lean-GAP gold (if licence allows) and a spot check on ATLAS outputs.
|
||||
- [ ] **P2** 7.1 role vocabulary growth/dedup; 7.2 dependency graph; 7.3 analogy mining; 7.4 judge-bias matrix; 7.5 robustness; 7.7 Chinese book + homogenization; 7.8 audit-time study.
|
||||
- [ ] **P0** **Go/no-go on 8 Dec**: if role alignment does not beat a single judge on at least two error types at equal false-reject, drop contribution C2 as a headline and submit as evaluation + taxonomy + provenance.
|
||||
- [ ] **P0** Freeze all tables on 24 Dec.
|
||||
|
||||
## E. Writing (start 1 Dec; related work and method now)
|
||||
- [ ] **P0** Related-work section from the notes in `research-plan.md` section 4 (all `[verify]` tags cleared).
|
||||
- [ ] **P0** Method section: definitions (5.1), taxonomy (5.2), slot decoding (5.3), alignment (5.4), retrieval (5.5).
|
||||
- [ ] **P0** Introduction (draft exists in section 3); rewrite after E4 numbers.
|
||||
- [ ] **P0** Figures: F1 overview (page crop + two simplices), F2 gap by model, F3 error taxonomy, F4 ROC/recall at fixed false-reject, F5 retrieval/guidance.
|
||||
- [ ] **P0** Tables: T1 stats, T2 gap, T3 filter comparison, T4 ablations.
|
||||
- [ ] **P0** Limitations (LLM dependence of source-side extraction, one proof assistant, few books, annotator expertise, judges proprietary).
|
||||
- [ ] **P0** Responsible NLP checklist, data statement, annotator pay and instructions, licence statement.
|
||||
- [ ] **P1** Anonymized repository + artifact package; remove identity from images, paths, commits.
|
||||
- [ ] **P1** Internal read-through by someone outside the project, 24-27 Dec.
|
||||
- [ ] **P0** Format on ACL template, 8 pages; submit to ARR by 3 Jan 2027.
|
||||
|
||||
## F. Risks to watch (check weekly)
|
||||
- [ ] New arXiv papers overlapping role-level or structural fidelity checks. If one appears: read within 24 h, update the positioning table, decide whether C2 still stands.
|
||||
- [ ] ICML 2027 / NeurIPS D&B 2027 official CFPs (dates are projected today).
|
||||
- [ ] ITP 2027 CFP (Glasgow, ~2-7 Aug 2027) as fallback.
|
||||
- [ ] Annotator availability; Lean compute (second runner if E4/E7/E8 exceed ~100 CPU-hours).
|
||||
- [ ] API cost against the 135M-token estimate (recompute after pilot).
|
||||
|
||||
## G. Milestones
|
||||
| Date | Milestone |
|
||||
|---|---|
|
||||
| 3 Nov | Items in A and B-core done; E0 passes; books chosen |
|
||||
| 17 Nov | All books ingested; annotation pilot kappa >= 0.6 |
|
||||
| 1 Dec | E2, E3, E4 finished; core labelling under way; writing starts |
|
||||
| 8 Dec | Go/no-go on role alignment |
|
||||
| 15 Dec | E5-E8 finished |
|
||||
| 24 Dec | Tables frozen; E9-E11 finished; full draft |
|
||||
| 27 Dec | Internal read-through done |
|
||||
| 3 Jan 2027 | Submit to ARR (deadline 4 Jan AoE) |
|
||||
+25
-10
@@ -93,7 +93,7 @@ Evidence base: repo README and code (`leanbench/`, 4,345 lines incl. tests), pro
|
||||
|
||||
### 3.1 Paragraph plan
|
||||
1. **Hook.** Formal libraries grow slowly because turning human mathematics into Lean is expensive; textbooks hold most of undergraduate and graduate mathematics, and LLMs now translate statements fluently.
|
||||
2. **The measurement problem.** Systems and benchmarks are scored by whether Lean accepts the output. But Lean checks types, not meaning: a statement missing a hypothesis, with `∃` for `∀`, or over `ℕ` instead of `ℤ` compiles and is wrong. Concurrent work already shows large gaps on curated statement sets (cite 2606.31002, 2608.29270 **[verify numbers]**); no one has measured it on raw textbooks where extraction error compounds with formalization error.
|
||||
2. **The measurement problem.** Systems and benchmarks are scored by whether Lean accepts the output. But Lean checks types, not meaning: a statement missing a hypothesis, with `∃` for `∀`, or over `ℕ` instead of `ℤ` compiles and is wrong. Concurrent work already shows large gaps on curated statement sets (Beyond Compilation: up to 29.5 points; ShadowBench: 61.8% compile vs 11.2% SA-Pass), and textbook pipelines report faithfulness only through uncalibrated LLM scores (ATLAS, Lean-GAP). What is missing is a cheap, structural check that says *which parameter* is wrong, calibrated against humans, on pages where extraction error compounds with formalization error.
|
||||
3. **Why textbooks are harder.** Font-mangled PDFs (`D` for `=`, `C` for `+`), items split across pages, definitions that depend on context. Each stage can break meaning silently: in our own pipeline the first server run reported success while storing 0 items.
|
||||
4. **Our observation.** A Lean declaration's parameters are a small typed schema: given objects, hypotheses, a goal. The same schema exists in the prose (what is assumed, what is claimed). Comparing the two *role by role* is a structural check that localizes the error.
|
||||
5. **Approach.** Represent both sides as simplices; use the schema to constrain formalization, to compare, and to retrieve.
|
||||
@@ -128,18 +128,25 @@ References marked **[verify]** are from memory/search snippets; confirm authors,
|
||||
- **miniF2F**, **PutnamBench**, **FormalMATH**, **CombiBench**, **Lean Workbook**: competition-style or synthetic. **[verify each]**
|
||||
- **ProofNet** (Azerbayev et al., 2023; undergraduate textbook theorems; [arXiv 2302.12433](https://arxiv.org/pdf/2302.12433)): closest prior. Hand-curated, statements only, no extraction from PDFs.
|
||||
- **ProofNet#, RLM25, ProofNetVerif** (Poiroux et al., [arXiv 2406.07222](https://arxiv.org/abs/2406.07222)): fixed 118 of 371 ProofNet entries; found type-check rate correlates only weakly (Pearson 0.655) with human accuracy; proposed BEq/BEq+ symbolic equivalence (BEq+ precision 98.0%, recall 48.3%).
|
||||
- Concurrent 2026 work: **Beyond Compilation** ([arXiv 2606.31002](https://arxiv.org/abs/2606.31002); 227 graduate statements; faithfulness = compile + two-LLM agreement; judge vs human agreement 91.5%; compile-faithful gap from 1.3 to 29.5 points across 8 systems) and **ShadowBench / SA-Pass** ([arXiv 2608.29270](https://arxiv.org/abs/2608.29270); 178 postgraduate-research problems; "shadow" statements checked forward and backward; compile 61.8% vs SA-Pass 11.2% for the best agentic setup). **[verify numbers against the final versions]**
|
||||
- *How we differ:* these start from a clean list of statements. We start from PDF pages, include definitions and exercises, keep page-level provenance, and measure how extraction error adds to formalization error.
|
||||
- **Beyond Compilation** ([arXiv 2606.31002](https://arxiv.org/abs/2606.31002), read in full 2026-10-21). 227 graduate statements taken from **four open LaTeX textbooks** (Lebl *Basic Analysis* 48, Lebl *Complex Analysis* 19, McKay *Topology* 63, Doty *Abstract Algebra* 97). Faithful = compiles AND both judges (GPT-5.2, Gemini-2.5-Pro) grade >= 9/10. Human audit: positive precision 67/70 = 95.7%; of 67 compile-pass outputs the rule rejected, humans confirmed 54 (80.6%); a 59-case Aristotle batch gave 91.5% agreement. Beats LeanScorer (76.3% vs 91.5% on that batch; McNemar p = 0.0225). BEq cross-check certified 116/189 pairs (61.4%). Eight systems: full tool-augmented GPT-5.2 agent compiles 87.2% but is faithful on 57.7% (gap 29.5); one-shot general LLMs faithful 19.8-27.3%; specialised formalizers 10.1-13.8%; prover-oriented 4.6-6.2%. Ablation: Lean feedback adds +49.3 compile but +22.1 of that is unfaithful; **faithfulness falls from 82.9% (1-2 agent steps) to 11.5% (19-24 steps)**. Stated limits: proprietary judges, same-family orchestrator, 227 items, no kappa reported in the visible text. Code/data: github.com/grenadeComing/beyond-compilation.
|
||||
- **ShadowBench / SA-Pass** ([arXiv 2608.29270](https://arxiv.org/abs/2608.29270), read 2026-10-21). 178 postgraduate/research problems from textbooks, notes, papers, Mathlib, Stacks, EGA. A *shadow* is a theorem implied by the intended statement; a candidate must compile, imply every shadow and be implied by their conjunction. Shadows are drafted by Qwen3-235B and completed by experts (706 checkers, ~10 min per problem, hidden from submitters). Best system (Claude Code Opus 4.8 + Numina) compile 61.8% vs SA-Pass 11.2%; non-agentic LLMs 0.0% SA-Pass. Agreement with two experts: SA-Pass 0.988, BEq+ 0.806, LLM-judge 0.785, compile 0.178. **On ProofNet the gap is only 2.1 points on average**, which the authors attribute to short single-conclusion targets. Cost of the method: per-problem shadow authoring.
|
||||
- **Lean-GAP** ([arXiv 2606.02588](https://arxiv.org/html/2606.02588v1)). **Closest to our pipeline.** 430 of 1,966 Dummit & Foote *Abstract Algebra* exercises: scanned pages -> Mathpix OCR -> LLM -> Lean `sorry` statements, human contributors, two-stage human sign-off, 11 "suspicious-statement" static patterns (C1-C11), Claude Opus 4.7 five-axis judge (uncalibrated). Six systems; GPT-5 elaborates 44.1% and is judged faithful 45.1%; Codex agent elaborates 95.5% but faithful 51.2%. Released on Kaggle with Wiley's permission for non-commercial research **[confirm terms]**.
|
||||
- **ATLAS / AutoformBot** ([arXiv 2605.29955](https://www.alphaxiv.org/abs/2605.29955)). 26 open-access graduate textbooks, > 45,000 Lean declarations, ~500k lines, 71.3% formalization rate; faithfulness only through an LLM 0-5 harness (threshold 3/5). No human calibration reported in the summary I read. 26 separate projects, not Mathlib-compatible.
|
||||
- **LeanFlow** ([arXiv 2607.20503](https://arxiv.org/html/2607.20503v1)): paper -> Lean workflow with a blueprint mapping source spans to declarations and an LLM statement-review gate; states that "no automated faithfulness check exists". Two papers only.
|
||||
- **Faults in Our Formal Benchmarking** ([arXiv 2606.29493](https://arxiv.org/html/2606.29493v1)): audits miniF2F, ProofNet, FormalMath, CombiBench, ProverBench (~10,000 problems), 4,833 findings, 398 with machine-checkable certificates; fidelity defects include missing hypotheses, vacuous hypotheses, `ℕ` subtraction truncation, division by zero. Its checkers are cheap static baselines for us.
|
||||
- **How we differ, stated carefully.** Textbook-to-Lean pipelines (Lean-GAP, ATLAS, LeanFlow) and faithfulness measurement (Beyond Compilation, ShadowBench) both exist. What none of them does: (a) a **structural, role-level fidelity check that localizes which parameter is wrong and needs no per-item authoring** (shadows) **and no extra judge call**; (b) **calibrating** that check against humans at the role level; (c) **page-level provenance with a coverage record** so extraction and formalization errors are separable; (d) **schema-guided generation and retrieval** built from the same roles. The paper must not be sold as "first textbook autoformalization".
|
||||
|
||||
### 4.2 Autoformalization methods
|
||||
- Few-shot autoformalization (Wu et al., 2022 **[verify]**); sampling + type-check filtering + selection ([2406.07222](https://arxiv.org/abs/2406.07222)); Herald, Kimina-Autoformalizer, Goedel-Prover, DeepSeek-Prover-V2 **[verify each]**; agentic Lean workflows (e.g. Numina-Lean-Agent as used in ShadowBench **[verify]**).
|
||||
- *How we differ:* we do not propose another model. We propose a schema layer that constrains and audits any model, and treat these systems as baselines.
|
||||
|
||||
### 4.3 Faithfulness and equivalence checking
|
||||
- **FormalAlign** (Lu et al., [ICLR 2025](https://proceedings.iclr.cc/paper_files/paper/2025/hash/fceedf8c9c0ff51f41b9fe0294ef0070-Abstract-Conference.html); [arXiv 2410.10135](https://arxiv.org/pdf/2410.10135)): learned alignment scorer.
|
||||
- **BEq / BEq+**: symbolic, high precision, low recall (above).
|
||||
- **LLM-judge ensembles** (Beyond Compilation) and **shadow statements** (ShadowBench): behavioural checks that need extra LLM calls or hand-written shadows.
|
||||
- Back-translation / round-trip consistency **[verify specific papers]**.
|
||||
- **FormalAlign** (Lu et al., [ICLR 2025](https://proceedings.iclr.cc/paper_files/paper/2025/hash/fceedf8c9c0ff51f41b9fe0294ef0070-Abstract-Conference.html); [arXiv 2410.10135](https://arxiv.org/pdf/2410.10135)): Mistral-7B fine-tuned with cross-entropy + contrastive loss; score = mean of certainty and cosine similarity, threshold 0.7; trained on synthetic misalignments (~21 negatives per positive); trails GPT-4 on FormL4-Random and MiniF2F-Test; human study of 80 items with Fleiss kappa 0.49. Lesson for us: **synthetic negatives do not match real errors, so evaluate on naturally occurring errors from textbook pages**.
|
||||
- **BEq / BEq+** (Poiroux et al.): symbolic, precision 98.0%, recall 48.3%. **TransTED / ASSESS** ([arXiv 2509.22246](https://arxiv.org/html/2509.22246); 1,247 expert-annotated pairs; accuracy 67-70%; ~51 s per pair; threshold varies 0.38-0.61 by benchmark; false positives on near-misses).
|
||||
- **CriticLean** ([arXiv 2507.06181](https://arxiv.org/pdf/2507.06181)): trained critic for faithfulness; CriticLeanBench 500 pairs; Qwen3-32B-RL 87.0% vs Gemini 2.5 Pro 89.2%; critic raised Kimina-Autoformalizer-7B from 54% (compiler feedback) to 84% on a 50-problem human check; ~47% of problems still failed in 200 attempts.
|
||||
- **LLM-judge ensembles** (Beyond Compilation) and **shadow statements** (ShadowBench): behavioural checks that need extra LLM calls or authored shadows (~10 min per item).
|
||||
- Static fidelity checkers: Lean-GAP's C1-C11 patterns and the checkers of Faults in Our Formal Benchmarking.
|
||||
- Back-translation / round-trip consistency: used inside several of the above; **[locate a specific citation before claiming it]**.
|
||||
- *How we differ:* role alignment is structural and gives a typed explanation of *which* part of the statement is wrong. It is cheap (one extraction call + deterministic matching), and complements, not replaces, BEq and judges. We evaluate all of them side by side (E6).
|
||||
|
||||
### 4.4 Structured / schema-guided reasoning with LLMs
|
||||
@@ -161,8 +168,11 @@ References marked **[verify]** are from memory/search snippets; confirm authors,
|
||||
|---|---|---|---|---|---|
|
||||
| ProofNet(#) | curated | human | high | n/a | no |
|
||||
| BEq+ | statement pairs | symbolic | CPU | no | no |
|
||||
| LLM judges | statement pairs | LLM | calls | no (text rationale) | no |
|
||||
| ShadowBench | curated | shadows | authoring | partly | no |
|
||||
| TransTED / ASSESS | statement pairs | tree edit + tactics | ~51 s/pair | no | no |
|
||||
| LLM judges (Beyond Compilation) | LaTeX textbook statements | 2-judge agreement | 2 calls | no (text rationale) | LaTeX, not PDF |
|
||||
| ShadowBench | curated | shadows | ~10 min authoring/item | partly | no |
|
||||
| Lean-GAP | scanned pages (Mathpix) | human sign-off + uncalibrated LLM judge | high human cost | no | yes, exercises only |
|
||||
| ATLAS | 26 open textbooks | uncalibrated LLM 0-5 | calls | no | yes, no human calibration |
|
||||
| **Ours** | **PDF pages** | **role alignment (+ judges)** | **1 call + CPU** | **yes, by role** | **yes** |
|
||||
|
||||
---
|
||||
@@ -276,6 +286,10 @@ Common protocol: Lean/Mathlib image pinned (Lean 4.33.1 image from the project);
|
||||
6. shadow-style check on 50 hand-built items (small, illustrative)
|
||||
7. **role alignment** (ours)
|
||||
8. role alignment + judge
|
||||
9. static suspicious-pattern checkers (Lean-GAP C1-C11 style; Faults-in-Formal-Benchmarking style): cheap baseline that catches vacuity and `ℕ`-subtraction hazards
|
||||
10. a trained critic (CriticLean released model, if weights are available) and FormalAlign
|
||||
11. agent-trajectory-length signal (Beyond Compilation found faithfulness collapses with many repair steps): a free baseline from our own repair logs
|
||||
- *External gold set (saves annotation):* if the Lean-GAP release permits, score role alignment against its human-signed exercise statements and its model outputs; if the terms forbid redistribution, use it only locally for evaluation and report aggregate numbers. **[confirm Kaggle terms]**
|
||||
- *Metrics:* precision, recall, F1 for "faithful"; ROC-AUC for alignment score; recall of "wrong-but-compiles" at fixed false-reject 10%; per-error-type recall (missing hypothesis, extra hypothesis, type, quantifier, goal); cost per item (calls, CPU).
|
||||
- *Controls:* shuffled spans (alignment must fail), identity (Lean vs its own back-translation), and judge self-preference (judge from same family as generator vs different).
|
||||
- *Quality of S_T itself:* role precision/recall of source-side extraction against the 150 gold simplices; sensitivity to extractor model.
|
||||
@@ -369,7 +383,8 @@ Writing starts 1 Dec in parallel; related work and method sections can be drafte
|
||||
| Archived simplex code needs more rework than planned | medium | Re-implement the minimal Role/Simplex/State in `leanbench/` rather than porting everything; analogy miner stays optional |
|
||||
| Compute for Lean at scale | medium | Second runner; reduce arms to core; cache by `sha1(code)` |
|
||||
| Licensing blocks release | medium | Open books only; hashes-only for others |
|
||||
| Concurrent work (2606.31002, 2608.29270) overlaps | high | Cite and compare directly; emphasise PDF-to-Lean pipeline, provenance and role-level localization; recheck arXiv weekly until submission |
|
||||
| Overlap with Beyond Compilation, ShadowBench, Lean-GAP, ATLAS (all 2026) | **high** | Do not claim first textbook pipeline or first gap measurement. Claim role-level, authoring-free, human-calibrated localization + schema guidance + provenance/coverage. Compare against their metrics directly. Recheck arXiv weekly (search: autoformalization faithfulness, textbook Lean) until submission |
|
||||
| ShadowBench finds the gap is small on short ProofNet-style targets (2.1 pts) | medium | Stratify E4 by statement length/complexity and report where role alignment helps; do not generalise from long items to all items |
|
||||
| Dates shift | low-medium | ICML/NeurIPS dates are projected; recheck CFPs monthly |
|
||||
|
||||
### 8.3 ACL / ARR checklist
|
||||
|
||||
Reference in New Issue
Block a user