claude-bot
69c5b788b9
The database was the missing half: extraction and assembly existed, but nothing turned the book's main-content statements into something an agent could actually cite mid-proof, and nothing searched it. Two new scripts. textbook/build_database.py takes the main-content statements and emits both halves of a usable premise library: - an importable Lean file where every entry is a named declaration, so a proof can cite goodman_1_6_20 by name; - a JSONL search index carrying each entry's name, statement, mentioned identifiers and status. An entry is emitted as an `axiom` (asserted from the textbook) until a real lean-bench run proves it; passing --results upgrades those entries to real theorems carrying the exact verified proof. This mirrors how the book itself works -- its exercises are posed assuming its body results, the same way using Mathlib does not mean re-proving Mathlib. The cost is explicit: anything proven through an axiom inherits it, so a garbled extraction asserted as an axiom makes downstream proofs worthless. Guards: --require-checked refuses entries that have not passed check_statements.py (all 21 section 1.6 entries have, and are now marked accordingly), and the verifier's existing #print axioms check surfaces what a finished proof rests on. textbook/search_database.py is the retrieval step the proof loop calls at each line: given the current goal, it ranks premises by identifier and symbol overlap, weighting rare identifiers far above common ones (IsGCDOfList matching means much more than Int matching), preferring a proven premise over an asserted one, and formats the top few for the prompt. Verified against three real goals: the exactly-applicable premise tops each ranking by a wide margin (11.4 vs 1.6 for the gcd goal, 5.1 vs 3.7 for the prime-divides-product goal). It is a ranking heuristic, not a decision procedure -- only Lean can say whether a premise applies. Replacing score_premise() is the whole job of swapping in embeddings. Also: renamed library_candidates/ to database/ to match what it is; generated lib/Goodman/Chapter01Section06.lean (21 premises, 0 proven, 21 asserted) and database/goodman-1.6.index.jsonl; noted a latent bug in the older build_library.py, which attaches each entry's non-import prelude per-theorem and so would emit a shared `def` twice in one file -- build_database.py hoists and dedupes, and has a test pinning that. 13 new tests, all green (33 total, the one failure remains the pre-existing unrelated lean_agent import error in test_dataset.py). Still not done, and worth being clear about: nothing has been proven yet. The bench targets have not been elaboration-checked, no lean-bench run has happened over either file, and the goal-directed loop that would consume this retrieval does not exist -- lean-bench's solve() is still single-shot. Separately, lean-bench cannot currently run at all in this checkout, because lean_bench/cli.py imports lean_agent at module load and lean-agent is not installed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
45 lines
2.3 KiB
Markdown
45 lines
2.3 KiB
Markdown
# Lean Bench
|
||
|
||
独立评测项目，依赖 lean-agent；两者放在同一父目录。
|
||
|
||
```sh
|
||
python3 -m venv .venv
|
||
source .venv/bin/activate
|
||
pip install -e ../lean-agent -e .
|
||
export LLM_API_KEY='your-key'
|
||
lean-bench datasets/smoke.jsonl --output results/model-001 --model YOUR_MODEL --base-url https://YOUR_HOST/v1 --attempts 3
|
||
```
|
||
|
||
也支持 agent 的 --provider-command、--lean-project、--lean-command 和超时选项。协议见 [Lean Agent](https://gitea.entroia.com/codex-bot/lean-agent)。
|
||
|
||
输入 JSONL 每行包含唯一 id、statement，可选 prelude。数据集必须可信；拒绝空集、重复 ID 和缺少字段。
|
||
|
||
输出目录必须不存在：results.jsonl 逐题记录证明、Lean 诊断、耗时和错误；summary.json 汇总成功率、异常数、尝试数、模型、Lean 版本及数据集 SHA-256。异常题保留在分母。指标为修复预算内 solved/tasks，不是独立采样 pass@k。顺序运行，暂不支持断点续跑。
|
||
|
||
不同实验应固定模型、预算、Lean/Mathlib、数据集与适配器版本。三个 smoke 题只验证流程，不代表数学推理能力。
|
||
|
||
## 无 LLM 的真实 Lean smoke 测试
|
||
|
||
```sh
|
||
lean-bench datasets/smoke.jsonl --output results/smoke-001 --provider-command '["python3","tests/mock_provider.py"]'
|
||
python3 -m unittest discover -s tests -v
|
||
```
|
||
|
||
mock 返回固定证明，仍由真实 Lean 检查。真实 LLM 需配置 endpoint/model 或适配器。
|
||
|
||
## 教科书形式化流水线
|
||
|
||
`textbook/` 下有一套从教材 PDF 生成两类数据的工具：真正的 bench 目标（来自每节
|
||
的 Exercises，`datasets/`）和候选库（来自正文 Theorem/Proposition/Lemma/
|
||
Corollary，`database/`，只有被真正证明过之后才进最终的可复用 Lean 库
|
||
文件），配 Definition 清单（Mathlib 没有的概念会编译成真正可导入的 Lean 代码，
|
||
`lib/Definitions.lean`，不只是 JSON 元数据）。用法见
|
||
[textbook/README.md](textbook/README.md)。
|
||
|
||
## 状态（更新于 2026-09-18）
|
||
|
||
`database/goodman-1.6.jsonl`（正文来源的候选库，21 条）全部通过本地
|
||
Lean elaboration 检查。`datasets/goodman-1.6-exercises.jsonl`（Exercises 1.6 来
|
||
源的真正 bench 目标，14 条）刚生成，尚待你本机跑一遍 `check_statements.py` 验
|
||
证。两边定义清单中的幻觉 Mathlib 引用均已修正。
|