claude-bot
fbcb0c3383
The index had no proof field and every entry was an axiom, so the database asserted its premises rather than proving them. Adds the proof half, with the verification gate that makes the status trustworthy. - Database records now carry `proof` and `proof_checked`, and the search index exposes the proof, with status one of proved / proof_pending / asserted. build_database.py emits a real `theorem` only when a proof has been verified (by check_proofs.py, or by a lean-bench run via --results); a recorded but unverified proof stays an `axiom` and is reported as proof_pending, so the library never claims more than Lean has confirmed. A test pins that gating in both directions. - textbook/check_proofs.py verifies proofs by compiling `theorem ... := by <proof>`. Stricter than check_statements.py in three ways: it rejects sorry/admit/native_decide outright, and it reads `#print axioms` to reject any proof depending on more than Lean's own three axioms (propext, Classical.choice, Quot.sound) -- which in particular catches a proof leaning on one of this database's own book-asserted axioms, since that is not an independent proof. With --update it writes proof_checked back, promoting entries on the next build. - Ten candidate proofs recorded, all currently proof_checked=false pending a Lean run: goodman-1-6-1 (the eight algebraic identities are `ring`, plus a short argument for the nonzero conjunct), the four parts of Proposition 1.6.2 (Int.mul_eq_one_iff_eq_one_or_neg_one, associated_of_dvd_dvd with Int.associated_iff, dvd_trans, dvd_add with mul_left), 1-6-4 (Nat.exists_prime_and_dvd), 1-6-15 (Int.gcd_eq_one_iff_coprime, where IsCoprime unfolds to exactly the target existential), 1-6-17 (IsCoprime.mul_dvd via Nat.isCoprime_iff_coprime), 1-6-19 (Int.Prime.dvd_mul') and 1-6-20 (Prime.dvd_prod_iff). These are citations of existing Mathlib results, which is the realistic way to populate a textbook premise library -- the remaining 11 (1-6-6, -7, -9, -10, -11, -13, -18, -21, -23, -24, -26) are the ones needing actual work and carry no proof yet. Written without Lean available here, so expect a real failure rate on first compile -- run check_proofs.py --update and send back the FAILED ids with diagnostics, same loop as the statement fixes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
Lean Bench
独立评测项目，依赖 lean-agent；两者放在同一父目录。
python3 -m venv .venv
source .venv/bin/activate
pip install -e ../lean-agent -e .
export LLM_API_KEY='your-key'
lean-bench datasets/smoke.jsonl --output results/model-001 --model YOUR_MODEL --base-url https://YOUR_HOST/v1 --attempts 3
也支持 agent 的 --provider-command、--lean-project、--lean-command 和超时选项。协议见 Lean Agent。
输入 JSONL 每行包含唯一 id、statement，可选 prelude。数据集必须可信；拒绝空集、重复 ID 和缺少字段。
输出目录必须不存在：results.jsonl 逐题记录证明、Lean 诊断、耗时和错误；summary.json 汇总成功率、异常数、尝试数、模型、Lean 版本及数据集 SHA-256。异常题保留在分母。指标为修复预算内 solved/tasks，不是独立采样 pass@k。顺序运行，暂不支持断点续跑。
不同实验应固定模型、预算、Lean/Mathlib、数据集与适配器版本。三个 smoke 题只验证流程，不代表数学推理能力。
无 LLM 的真实 Lean smoke 测试
lean-bench datasets/smoke.jsonl --output results/smoke-001 --provider-command '["python3","tests/mock_provider.py"]'
python3 -m unittest discover -s tests -v
mock 返回固定证明，仍由真实 Lean 检查。真实 LLM 需配置 endpoint/model 或适配器。
教科书形式化流水线
textbook/ 下有一套从教材 PDF 生成两类数据的工具：真正的 bench 目标（来自每节
的 Exercises，datasets/）和候选库（来自正文 Theorem/Proposition/Lemma/
Corollary，database/，只有被真正证明过之后才进最终的可复用 Lean 库
文件），配 Definition 清单（Mathlib 没有的概念会编译成真正可导入的 Lean 代码，
lib/Definitions.lean，不只是 JSON 元数据）。用法见
textbook/README.md。
状态（更新于 2026-09-18）
database/goodman-1.6.jsonl（正文来源的候选库，21 条）全部通过本地
Lean elaboration 检查。datasets/goodman-1.6-exercises.jsonl（Exercises 1.6 来
源的真正 bench 目标，14 条）刚生成，尚待你本机跑一遍 check_statements.py 验
证。两边定义清单中的幻觉 Mathlib 引用均已修正。