claude-bot
04d827b3b3
Restructures the textbook pipeline around the distinction the body-derived dataset had been blurring: a bench should be actual unsolved problems, and a reusable premise library should only ever contain statements that have really been proven, the way Mathlib itself works. - library_candidates/goodman-1.6.jsonl (renamed from datasets/goodman-1.6.jsonl): the book's own Theorem/Proposition/Lemma/Corollary claims from the main text. Not bench targets -- premises waiting to be proven once via a real lean-bench run, then assembled into the reusable library by build_library.py (which already only keeps success:true entries with their checked proof, unchanged by this commit). - datasets/goodman-1.6-exercises.jsonl (new): the actual bench, sourced from the book's Exercises 1.6 block via two new scripts, textbook/extract_exercises.py (slices the Exercises block, lists its numbered items, drops a known false-positive class where a wrapped cross-reference like "...Proposition\n1.6.19." looks like a real exercise heading) and textbook/formalize_exercises.py (LLM step, given both the exercises and body text together since many exercises say "Prove Proposition N.N.N" instead of restating the claim). - Both formalize.py and formalize_exercises.py's validate() now enforce that every checklist item is covered by at least one output entry (included or explicitly excluded with a reason); formalize_exercises.py did not have this yet and a first real run silently dropped 7 of 15 checklist items with no trace before this was added. - All 12 model-produced exercise entries needed real fixes before commit, the same review pass applied to the body pilot: 3 exercises literally ask to prove an already-stated body claim (Exercise 1.6.1 -> Proposition 1.6.7, 1.6.2 -> Proposition 1.6.9, 1.6.7 -> Proposition 1.6.18, 1.6.9 -> Corollary 1.6.20, 1.6.15 -> Lemma 1.6.26) and now reuse that claim's own already-checked statement verbatim instead of an independently re-derived (and in one case strictly weaker) restatement; two entries (1.6.2, 1.6.11) had literal math prose like "I(m, n)" or "gcd(n1,...,nk)" leaked into the statement field, which is not valid Lean; several had untyped/ambiguous gcd or Prime usage fixed the same way the earlier body fixes were (Nat.Prime, IsCoprime, explicit casts, IsGreatest instead of max on a Set); and Exercise 1.6.12 part (a) turned out to be a discussion question whose unconditional-identity reading is actually false (6, 10, 15 are jointly coprime but not pairwise coprime), so it was moved to the excluded file with a reason instead of committed as a false theorem. - Definitions can now be real, compiled Lean, not only JSON metadata: Definition 1.6.22 (gcd of a list of integers, previously a null placeholder since no single Mathlib constant matches it) now has a real hand-written `def IsGCDOfList`, and textbook/build_definitions.py compiles any manifest's non-null lean_code entries into lib/Definitions.lean. goodman-1-6-24 (library) and the new goodman-1-6-ex-11-b/c, goodman-1-6-ex-12 (bench) all now use IsGCDOfList instead of repeating an inline existential. - textbook/README.md rewritten around the two-track design and records how this is meant to connect to lean-agent's eventual goal-directed proof search against the growing library (not yet built). - New tests for extract_exercises.py and build_definitions.py; full local suite still green (18/19, the one failure is the pre-existing, unrelated lean_agent import error in test_dataset.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
45 lines
2.3 KiB
Markdown
45 lines
2.3 KiB
Markdown
# Lean Bench
|
||
|
||
独立评测项目，依赖 lean-agent；两者放在同一父目录。
|
||
|
||
```sh
|
||
python3 -m venv .venv
|
||
source .venv/bin/activate
|
||
pip install -e ../lean-agent -e .
|
||
export LLM_API_KEY='your-key'
|
||
lean-bench datasets/smoke.jsonl --output results/model-001 --model YOUR_MODEL --base-url https://YOUR_HOST/v1 --attempts 3
|
||
```
|
||
|
||
也支持 agent 的 --provider-command、--lean-project、--lean-command 和超时选项。协议见 [Lean Agent](https://gitea.entroia.com/codex-bot/lean-agent)。
|
||
|
||
输入 JSONL 每行包含唯一 id、statement，可选 prelude。数据集必须可信；拒绝空集、重复 ID 和缺少字段。
|
||
|
||
输出目录必须不存在：results.jsonl 逐题记录证明、Lean 诊断、耗时和错误；summary.json 汇总成功率、异常数、尝试数、模型、Lean 版本及数据集 SHA-256。异常题保留在分母。指标为修复预算内 solved/tasks，不是独立采样 pass@k。顺序运行，暂不支持断点续跑。
|
||
|
||
不同实验应固定模型、预算、Lean/Mathlib、数据集与适配器版本。三个 smoke 题只验证流程，不代表数学推理能力。
|
||
|
||
## 无 LLM 的真实 Lean smoke 测试
|
||
|
||
```sh
|
||
lean-bench datasets/smoke.jsonl --output results/smoke-001 --provider-command '["python3","tests/mock_provider.py"]'
|
||
python3 -m unittest discover -s tests -v
|
||
```
|
||
|
||
mock 返回固定证明，仍由真实 Lean 检查。真实 LLM 需配置 endpoint/model 或适配器。
|
||
|
||
## 教科书形式化流水线
|
||
|
||
`textbook/` 下有一套从教材 PDF 生成两类数据的工具：真正的 bench 目标（来自每节
|
||
的 Exercises，`datasets/`）和候选库（来自正文 Theorem/Proposition/Lemma/
|
||
Corollary，`library_candidates/`，只有被真正证明过之后才进最终的可复用 Lean 库
|
||
文件），配 Definition 清单（Mathlib 没有的概念会编译成真正可导入的 Lean 代码，
|
||
`lib/Definitions.lean`，不只是 JSON 元数据）。用法见
|
||
[textbook/README.md](textbook/README.md)。
|
||
|
||
## 状态（更新于 2026-09-18）
|
||
|
||
`library_candidates/goodman-1.6.jsonl`（正文来源的候选库，21 条）全部通过本地
|
||
Lean elaboration 检查。`datasets/goodman-1.6-exercises.jsonl`（Exercises 1.6 来
|
||
源的真正 bench 目标，14 条）刚生成，尚待你本机跑一遍 `check_statements.py` 验
|
||
证。两边定义清单中的幻觉 Mathlib 引用均已修正。
|