claude-bot
caedf2e6f4
- extract_section.py: slice a book section out of a pdftotext -layout dump and list its labeled Theorem/Proposition/Lemma/Corollary/Definition items. - formalize.py: LLM-formalize a section into Lean 4 candidate statements (preferring Mathlib vocabulary) plus a definitions manifest. - check_statements.py: elaboration-only sanity check against a real Mathlib-backed Lean project (must run outside the Cowork sandbox, which has no Lean toolchain and can't run macOS binaries anyway). - build_library.py: assemble solved lean-bench results into a reusable .lean file, reusing each winning proof verbatim. - llm.py: standalone OpenAI-compatible chat client + .env loader, kept independent of lean_agent so this pipeline works without it installed. - Source PDFs and raw/layout text dumps are gitignored (copyrighted material); only code and the final short Lean statement datasets are tracked.
35 lines
1.7 KiB
Markdown
35 lines
1.7 KiB
Markdown
# Lean Bench
|
|
|
|
独立评测项目，依赖 lean-agent；两者放在同一父目录。
|
|
|
|
```sh
|
|
python3 -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -e ../lean-agent -e .
|
|
export LLM_API_KEY='your-key'
|
|
lean-bench datasets/smoke.jsonl --output results/model-001 --model YOUR_MODEL --base-url https://YOUR_HOST/v1 --attempts 3
|
|
```
|
|
|
|
也支持 agent 的 --provider-command、--lean-project、--lean-command 和超时选项。协议见 [Lean Agent](https://gitea.entroia.com/codex-bot/lean-agent)。
|
|
|
|
输入 JSONL 每行包含唯一 id、statement，可选 prelude。数据集必须可信；拒绝空集、重复 ID 和缺少字段。
|
|
|
|
输出目录必须不存在：results.jsonl 逐题记录证明、Lean 诊断、耗时和错误；summary.json 汇总成功率、异常数、尝试数、模型、Lean 版本及数据集 SHA-256。异常题保留在分母。指标为修复预算内 solved/tasks，不是独立采样 pass@k。顺序运行，暂不支持断点续跑。
|
|
|
|
不同实验应固定模型、预算、Lean/Mathlib、数据集与适配器版本。三个 smoke 题只验证流程，不代表数学推理能力。
|
|
|
|
## 无 LLM 的真实 Lean smoke 测试
|
|
|
|
```sh
|
|
lean-bench datasets/smoke.jsonl --output results/smoke-001 --provider-command '["python3","tests/mock_provider.py"]'
|
|
python3 -m unittest discover -s tests -v
|
|
```
|
|
|
|
mock 返回固定证明，仍由真实 Lean 检查。真实 LLM 需配置 endpoint/model 或适配器。
|
|
|
|
## 教科书形式化流水线
|
|
|
|
`textbook/` 下有一套从教材 PDF 生成 lean-bench 数据集的工具（LLM 辅助抽取
|
|
Theorem/Proposition/Lemma/Corollary，配 Definition 清单，事后把证出来的结果
|
|
组装成可复用的 Lean 库文件），用法见 [textbook/README.md](textbook/README.md)。
|