claude-bot
f98cb41f72
Generated with formalize.py against Qwen (qwen-max via DashScope), then hand-fixed before pushing, since these classes of bug pass elaboration but are still wrong and check_statements.py can't catch them without running Lean: - goodman-1-6-2 split into 4 separate theorems (1-6-2-a..d): the model's single merged statement had an operator-precedence bug (missing parens around an implicit right-associated AND after OR) that silently scoped 3 of the book's 4 independent claims inside the wrong disjunct. - goodman-1-6-1: dropped two vacuous "x + y : Nat is a Nat" clauses that confused Lean's type membership with set membership (element-of Nat on already-Nat-typed variables) and are automatically true by typing anyway. - goodman-1-6-9: swapped Mathlib's pointwise Set multiplication (x * I, which needs 'open scoped Pointwise' and is easy to get subtly wrong) for the equivalent elementwise statement. - goodman-1-6-13: replaced a misuse of min (a 2-argument function) applied to a Set with Mathlib's IsLeast predicate, matching what "least element" means in Int.gcd's characterization. - goodman-1-6-23: fixed Matrix index types (Fin n, not a bare Nat) and replaced an illegal literal '...' row with Matrix.vecMul; verified against the book's raw text that the n-by-n unimodular-matrix formulation is faithful, not a model invention. textbook/goodman-1.6.raw.jsonl keeps the unedited model output for reference. The remaining 13 statements are unedited and still need a real Lean elaboration pass (check_statements.py) plus a provability run (lean-bench) on a Mathlib-backed project -- neither ran here since this sandbox has no Lean.
Lean Bench
独立评测项目，依赖 lean-agent；两者放在同一父目录。
python3 -m venv .venv
source .venv/bin/activate
pip install -e ../lean-agent -e .
export LLM_API_KEY='your-key'
lean-bench datasets/smoke.jsonl --output results/model-001 --model YOUR_MODEL --base-url https://YOUR_HOST/v1 --attempts 3
也支持 agent 的 --provider-command、--lean-project、--lean-command 和超时选项。协议见 Lean Agent。
输入 JSONL 每行包含唯一 id、statement，可选 prelude。数据集必须可信；拒绝空集、重复 ID 和缺少字段。
输出目录必须不存在：results.jsonl 逐题记录证明、Lean 诊断、耗时和错误；summary.json 汇总成功率、异常数、尝试数、模型、Lean 版本及数据集 SHA-256。异常题保留在分母。指标为修复预算内 solved/tasks，不是独立采样 pass@k。顺序运行，暂不支持断点续跑。
不同实验应固定模型、预算、Lean/Mathlib、数据集与适配器版本。三个 smoke 题只验证流程，不代表数学推理能力。
无 LLM 的真实 Lean smoke 测试
lean-bench datasets/smoke.jsonl --output results/smoke-001 --provider-command '["python3","tests/mock_provider.py"]'
python3 -m unittest discover -s tests -v
mock 返回固定证明，仍由真实 Lean 检查。真实 LLM 需配置 endpoint/model 或适配器。
教科书形式化流水线
textbook/ 下有一套从教材 PDF 生成 lean-bench 数据集的工具（LLM 辅助抽取
Theorem/Proposition/Lemma/Corollary，配 Definition 清单，事后把证出来的结果
组装成可复用的 Lean 库文件），用法见 textbook/README.md。