claude-bot
79c728567a
All 21 statements in datasets/goodman-1.6.jsonl now elaborate cleanly
(verified locally via check_statements.py, 11/21 passing before this
change). Fixes applied, one per failing id:
- goodman-1-6-7: single-binder existential-unique over a pair (qr : ZxZ)
instead of the invalid two-binder 'exists unique' form.
- goodman-1-6-10: rewrote set-builder notation to the explicit
{v | exists a b, v = ...} form.
- goodman-1-6-11: replaced 'let d := ... in ...' with the required
semicolon form 'let d := ...; ...', and added the missing Nat-to-Int
casts on d.
- goodman-1-6-17: added explicit Nat-to-Int coercions on the three
divisibility relations.
- goodman-1-6-18: changed p from Z to N (Nat.Prime requires N), added
casts, and replaced the nonexistent Int.Coprime with IsCoprime.
- goodman-1-6-19: changed p from Z to N and added the matching casts.
- goodman-1-6-20: changed p from Z to N, added casts, and fixed the
quantifier structure so the prime-divides-product hypothesis is not
nested inside the per-element forall.
- goodman-1-6-23: replaced the Fin n literal comparison 'j = 0' (which
needs a NeZero instance) with 'j.val = 0'.
- goodman-1-6-24: replaced the invalid bare '(*)' operator section with
'(fun x y => x * y)'-equivalent '(. * .)', and replaced the
nonexistent GCD.gcd with an explicit existential statement of the gcd
property (divides every element; every common divisor divides it).
- goodman-1-6-26: replaced the nonexistent Int.Coprime with IsCoprime.
Also corrected the three parallel hallucinated Mathlib refs in
textbook/goodman-1.6.definitions.json (Int.Coprime, GCD.gcd,
Int.PairwiseCoprime -> IsCoprime, null with notes, and
List.Pairwise IsCoprime respectively), since the Lean compiler confirms
none of the original names exist.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
Lean Bench
独立评测项目，依赖 lean-agent；两者放在同一父目录。
python3 -m venv .venv
source .venv/bin/activate
pip install -e ../lean-agent -e .
export LLM_API_KEY='your-key'
lean-bench datasets/smoke.jsonl --output results/model-001 --model YOUR_MODEL --base-url https://YOUR_HOST/v1 --attempts 3
也支持 agent 的 --provider-command、--lean-project、--lean-command 和超时选项。协议见 Lean Agent。
输入 JSONL 每行包含唯一 id、statement，可选 prelude。数据集必须可信；拒绝空集、重复 ID 和缺少字段。
输出目录必须不存在：results.jsonl 逐题记录证明、Lean 诊断、耗时和错误；summary.json 汇总成功率、异常数、尝试数、模型、Lean 版本及数据集 SHA-256。异常题保留在分母。指标为修复预算内 solved/tasks，不是独立采样 pass@k。顺序运行，暂不支持断点续跑。
不同实验应固定模型、预算、Lean/Mathlib、数据集与适配器版本。三个 smoke 题只验证流程，不代表数学推理能力。
无 LLM 的真实 Lean smoke 测试
lean-bench datasets/smoke.jsonl --output results/smoke-001 --provider-command '["python3","tests/mock_provider.py"]'
python3 -m unittest discover -s tests -v
mock 返回固定证明，仍由真实 Lean 检查。真实 LLM 需配置 endpoint/model 或适配器。
教科书形式化流水线
textbook/ 下有一套从教材 PDF 生成 lean-bench 数据集的工具（LLM 辅助抽取
Theorem/Proposition/Lemma/Corollary，配 Definition 清单，事后把证出来的结果
组装成可复用的 Lean 库文件），用法见 textbook/README.md。