docs: research plan and paper brief (ACL 2027 target)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYqwNMyPUHeTbFWg83GQq6

docs/research-plan.md

@@ -0,0 +1,148 @@
+# lean-bench: research plan and paper brief
+
+Written 2026-10-21. Status of the system is taken from the repo README and the project notes (rebuild of 2026-09-19). Conference dates were checked on 2026-10-21; items marked *unverified* come from aggregator sites or projections and must be re-checked against the official call.
+
+## 0. Where the project stands, honestly
+
+What exists: a working pipeline (page-level detection with quote-anchored spans, vision-OCR for font-mangled PDFs, Lean 4 formalization with one repair pass, Lean verification, type-indexed SQLite store with provenance). One book (587 pages), one 26-page verified run (89 items, about 3.4 items per page).
+
+What a reviewer will say today:
+1. No quantitative result. The only numbers are operational (CI time, OCR failure rate).
+2. `verified` means Lean accepted the statement, not that the statement says what the book says. The notes already admit "a well-formed statement of the wrong theorem passes every check".
+3. Detection is not reproducible (80 vs 89 items on the same range).
+4. One book, one LLM, no baselines, no human annotation.
+5. The source book is copyrighted, so the data cannot be released as built.
+
+The plan below turns each of these into the paper's contribution or removes it.
+
+## 1. Venue decision
+
+| Venue | Deadline | Fit | Verdict |
+|---|---|---|---|
+| **ACL 2027** (Kyoto, Aug 20-22 2027), via ACL Rolling Review | **ARR submission 4 Jan 2027** (official site); commitment and notification TBA; 8-page long paper | Resource / evaluation paper, NLP + autoformalization. Optional theme on LLM homogenization | **Primary target.** About 11 weeks from today |
+| ICML 2027 | Abstract ~16 Jan, paper ~22 Jan 2027 (*unverified*, projected from the 2026 cycle; official CFP not out, location South America) | AI-for-math ML audience; bar for a benchmark with no new method is high | Backup if the faithfulness method is strong |
+| IJCAI 2027 | ~4 / 11 Jan 2027 (*unverified*) | General AI | Not recommended |
+| ITP 2027 (Glasgow, ~Aug 2-7 2027) | Not published yet | Formal-methods audience; cares about Lean/Mathlib correctness | Good fallback for a trimmed version; watch for the CFP |
+| COLM 2027 | ~30 Mar 2027 (*tentative*) | LLM-centred | Fallback if ACL misses |
+| NeurIPS 2027 Datasets & Benchmarks | ~14 / 21 May 2027 (*projected*) | Natural home for the full benchmark | Fallback with a full year of extra experiments |
+
+Already closed, so not options: ICLR 2027 (25 Sep 2026), CPP 2027 (10 Sep 2026), NAACL 2027 / COLING ARR cycle (12 Oct 2026), AAAI-27, NeurIPS 2026.
+
+**Recommendation:** target ACL 2027 on the 4 Jan ARR deadline. The work is a resource-plus-evaluation paper, which ACL accepts, and the date is official. Keep ICML (about 22 Jan) as a one-to-two-week-later alternative only if the experiments in 4.3 show a clear method gain. If ARR reviews arrive weak, commit to a later cycle or retarget NeurIPS D&B 2027.
+
+**What the ACL bar requires** (so the plan is built around it): a clear contribution beyond engineering; a meaningful evaluation with baselines and more than one model; human evaluation or annotation with agreement statistics; released artefacts that can be used (data and code, licence-clean); Limitations section and Responsible NLP checklist; reproducibility details; 8 pages double-blind.
+
+## 2. Paper framing
+
+**Working title:** *Compiled is not Faithful: Measuring and Closing the Fidelity Gap in Textbook-to-Lean Autoformalization*
+
+**One-sentence claim:** Lean acceptance badly overstates the quality of LLM autoformalization of real textbook mathematics; we release a page-grounded benchmark with human fidelity labels, show how large the gap is across models and pipeline stages, and show that a cheap round-trip fidelity check shrinks it.
+
+Contributions (all must be evidenced in the experiments):
+1. **lean-bench**, a page-grounded benchmark of textbook items (statements, definitions, exercises) with Lean 4 formalizations, each carrying source hash, page and character span, and human fidelity labels.
+2. **A measurement of the verified-vs-faithful gap** by model, by item type, by domain, and by pipeline stage (OCR, detection, formalization, repair).
+3. **A fidelity filter** (round-trip informalization plus judge ensemble, calibrated on human labels) that raises precision of the released set.
+4. **Type-indexed retrieval** over the verified set, tested as a downstream aid to provers.
+
+## 3. Brief
+
+### 3.1 Introduction (about 1 page)
+- Autoformalization of research-level and textbook mathematics is the bottleneck for formal libraries; existing benchmarks are competition problems or small curated sets.
+- Textbooks are the real input: PDFs with broken math fonts, items split over pages, definitions that depend on context.
+- Everyone reports compile rate. Compile rate cannot see a well-typed statement of the wrong theorem. We quantify that.
+- Preview of findings (to be filled with real numbers): gap size, which stage loses the most, what the filter recovers.
+- Contributions list as in section 2.
+
+### 3.2 Related work (about 0.75 page)
+Group into four threads. Entries are from memory and **must be verified and completed before submission**.
+- *Autoformalization benchmarks*: miniF2F, ProofNet (undergraduate textbook theorems, the closest prior), PutnamBench, FormalMATH, Lean Workbook, CombiBench. Gap we fill: page-grounded extraction from raw PDFs, human fidelity labels, definitions and exercises, not only theorem statements.
+- *Autoformalization models and training*: Wu et al. 2022 (few-shot with LLMs), Herald, Kimina-Autoformalizer, Goedel-Prover and DeepSeek-Prover family, repair with compiler feedback and type-check filtering (Poiroux et al. line of work). We use these as baselines, not competitors.
+- *Faithfulness / semantic equivalence checking*: FormalAlign, back-translation and LLM-judge approaches, ProofNet# style manual correction. We compare our filter against these.
+- *Document understanding for mathematics*: math OCR (Nougat, Mathpix-style systems), PDF-to-LaTeX pipelines, and statement mining from textbooks. We compare our vision-OCR route and anchor-based span resolution.
+- *Premise and library retrieval*: LeanDojo/ReProver, Lean search tools (Moogle, LeanSearch). We position type-index retrieval relative to embedding retrieval.
+
+### 3.3 Key strategy (about 1.5 pages)
+1. **Ground everything in the page.** Quote anchors, not model-counted offsets; per-item `source_hash`, span and `page_checksum`. This is what makes human audit and re-run cheap, and it is the methodological difference from ProofNet-style curation.
+2. **Separate four claims that today are conflated:** *detected* (an item exists at this span), *compiled* (Lean accepts), *proved* (no `sorry`), *faithful* (matches the source). The pipeline reports each separately. Fix the overloaded `unverified` label first, since "Lean rejected" and "Lean never ran" currently look identical.
+3. **Fidelity filter.** Informalize the Lean statement back to natural language with a model that never saw the source, then judge equivalence against the source span with a multi-model ensemble; calibrate thresholds against human labels. Report precision/recall of the filter against humans, not agreement among LLMs.
+4. **Make every stage ablatable.** OCR mode (text / auto / ocr), window size, repair attempts, model choice, each toggle changes one thing and gets measured.
+5. **Release without releasing the book.** Use openly licensed sources for the public set (see 4.1); for copyrighted books release only spans' hashes, Lean code and instructions to reproduce, never page text.
+
+### 3.4 Detailed experiments (about 2.5 pages)
+
+**E1. Benchmark construction and statistics.**
+Sources: at least 6-8 books across algebra, analysis, topology, number theory, combinatorics; at least 3 with an open licence. Report items per book, kinds, span lengths, cross-page rate, OCR failure rate. Target 1,500+ items, with a 600-item human-audited core.
+
+**E2. Detection quality.**
+Humans mark ground-truth items on 150 pages (stratified by book and by OCR difficulty). Report span precision, recall, boundary exact-match and IoU; breakdown for cross-page items. Ablate window size (1/2/3/5), extractor (text / auto / ocr), detection model. Report run-to-run variance over 5 repeats to address the 80-vs-89 instability.
+
+**E3. OCR / extraction.**
+On pages with known font damage, compare embedded text, tesseract, the vision model, and one dedicated math-OCR system on LaTeX token accuracy and on downstream item yield. Includes the auto-mode quality heuristic's accuracy against human judgement.
+
+**E4. Formalization: compile rate vs fidelity (the central experiment).**
+Models: at least four general LLMs (e.g. Qwen, one OpenAI, one Anthropic, one DeepSeek) plus at least one specialised autoformalizer. Metrics per model: compile rate, proved rate, human-judged fidelity, and the gap. Breakdown by item kind (definition / theorem / problem), domain, length, dependency on prior context. Report cost per verified-and-faithful item.
+
+**E5. Human annotation protocol.**
+Two Lean-literate annotators label each item {faithful, minor drift, wrong, wrong-but-compiles}; third resolves. Report Cohen's kappa; guideline and error taxonomy (missing hypothesis, wrong quantifier, vacuous statement, changed definition, over-general). Include a taxonomy figure; this is the analysis section reviewers will remember.
+
+**E6. Fidelity filter evaluation.**
+Compare: no filter, type-check only, single-LLM judge, judge ensemble, round-trip plus ensemble. Report precision/recall/F1 for "faithful" against the human core, and precision of the retained set at fixed recall. Include a negative control (shuffled spans) to show the filter is not rubber-stamping.
+
+**E7. Repair ablation.**
+Repair attempts 0/1/3/5. Key question: does repair improve fidelity or only compile rate? Expect compile rate up and fidelity flat or down; if so, that is a headline finding.
+
+**E8. Downstream use: type-indexed retrieval.**
+On held-out exercises, give a prover model (a) nothing, (b) embedding-retrieved items, (c) type-index retrieved items, from the verified-faithful set; report pass@k. Also run on faithful vs unfiltered store to show filtering matters downstream. Keep this modest: it supports a contribution, it does not carry the paper.
+
+**E9. Contamination check.**
+For each formalization model, test whether items exist verbatim in public Lean corpora (Mathlib docstrings, ProofNet); report results on the clean subset.
+
+**E10. Error analysis and cost.**
+Failure cases by pipeline stage, token cost, wall-clock per page, Lean time per item.
+
+### 3.5 More detailed experiments (second layer, if time allows)
+- **Scaling:** performance vs book length and vs number of dependencies on earlier chapters; a context-carrying variant that supplies earlier definitions.
+- **Cross-lingual / non-English books** (project has Chinese-language tooling already): detection and OCR on Chinese textbooks, for the ACL audience.
+- **Judge bias:** does the judge prefer outputs from its own family? Cross-family judging matrix.
+- **Human effort study:** time to audit with page-grounded span vs. without; supports the provenance design.
+- **Robustness to scan quality:** degrade page images (DPI, noise) and measure yield.
+- **Homogenization angle** (ACL 2027's optional theme): do models converge on the same wrong formalization? Measure agreement among wrong outputs across models.
+
+## 4. Work plan to the 4 Jan 2027 deadline
+
+### 4.1 Preconditions (week 1-2, 21 Oct - 3 Nov)
+- Fix the overloaded `unverified` label; add the `source_pages` coverage table; incremental per-item storage (all three already queued).
+- Choose the open-licence sources and confirm their licences (do not reuse the uiowa-algebra book for any released file).
+- Write annotation guideline; recruit two annotators; set up an annotation UI over the existing page-grounded provenance.
+- Freeze a pipeline version and a model list; log all prompts and model ids.
+
+### 4.2 Data and runs (weeks 3-6, 4 Nov - 1 Dec)
+- Ingest all books; produce the item pool; draw the 150-page detection sample and the 600-item core.
+- Run E2, E3, E4 for all models.
+
+### 4.3 Annotation and method (weeks 5-9, 18 Nov - 22 Dec)
+- Human labelling of the core; compute agreement.
+- Build and calibrate the fidelity filter; run E5-E7.
+- Go / no-go on 8 Dec: if the filter gives no clear gain over a single-LLM judge, submit as a pure evaluation-and-analysis paper and drop contribution 3.
+
+### 4.4 Downstream and writing (weeks 8-11, 9 Dec - 3 Jan)
+- Run E8-E10; write the paper in parallel from 1 Dec; internal review 24 Dec; release the licence-clean data and code package; ARR submission by 3 Jan to leave a day of slack.
+
+## 5. Risks and mitigations
+| Risk | Mitigation |
+|---|---|
+| Annotation too slow | Fix the audited core at 600; stratified sampling; second annotator on a 30% overlap only for kappa |
+| Fidelity filter shows no gain | Pure analysis paper; the gap measurement is the contribution |
+| Reviewers call it an engineering paper | Lead with the gap measurement and error taxonomy, not the pipeline |
+| Licensing blocks release | Open-licence books for the public split; hash-only releases for the rest |
+| Model API cost or drift | Pin model ids, cache outputs, report dates of calls |
+| Lean/Mathlib version drift | Pin the image (toolchain + Mathlib revision) per release |
+| Dates move | ICML and NeurIPS D&B dates are projected; re-check official CFPs before committing |
+
+## 6. Sources checked on 2026-10-21
+- ACL 2027 site (official): ARR deadline 4 Jan 2027, Kyoto 20-22 Aug 2027.
+- NAACL 2027 call: ARR deadline 12 Oct 2026 (closed).
+- ICLR 2027 call: paper deadline 25 Sep 2026 (closed).
+- CPP 2027 page: paper deadline 10 Sep 2026 (closed).
+- ITP 2027 site: Glasgow, early Aug 2027, deadlines not yet published.
+- OpenCurious / trybibby aggregators: ICML 2027, IJCAI 2027, COLM 2027, NeurIPS 2027 (projected, unofficial).