claude-bot
9e0457a0e1
The index file was 12KB of which ~96% was a byte-for-byte copy of
database/goodman-1.6.jsonl. Its only original content was three derived
fields: `name` (literally id.replace('-','_')), `status` (a two-line
computation over proof_checked), and `identifiers` (a regex over the
statement). It was a cache for work that costs microseconds, and it
created a real hazard: nothing checked that it matched its source, so
editing a statement and forgetting to regenerate would leave
search_database.py serving a premise whose text no longer matched the
generated .lean file.
- search_database.py now reads database/goodman-1.6.jsonl directly via a
new load_database(), deriving name/identifiers/status on the fly and
importing those functions from build_database.py so there is one
definition of each rather than two that can drift.
- build_database.py drops --out-index and index generation, and its
proof/status derivation is factored into resolve_proof() shared with
retrieval.
- database/goodman-1.6.index.jsonl deleted.
Also fixes a retrieval bug the removal surfaced: the tokenizer only
matched ASCII identifiers, so a goal that is pure algebra
("a * (b + c) = a * b + a * c") tokenized to nothing and retrieved
nothing at all -- it now keeps type and operator symbols, and drops
single-letter variable names, which say nothing when matched. That goal
now correctly finds Proposition 1.6.1, and the identifier-rich goals
still rank their exact premise first by a wide margin (9.3 vs 4.4).
Tests restructured accordingly: the proof gate is now pinned directly on
resolve_proof in both directions, plus new coverage for load_database
deriving status from proof_checked and for the symbol tokenization.
49 tests, 48 green (the one failure remains the pre-existing unrelated
lean_agent import error in test_dataset.py).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
Lean Bench
独立评测项目，依赖 lean-agent；两者放在同一父目录。
python3 -m venv .venv
source .venv/bin/activate
pip install -e ../lean-agent -e .
export LLM_API_KEY='your-key'
lean-bench datasets/smoke.jsonl --output results/model-001 --model YOUR_MODEL --base-url https://YOUR_HOST/v1 --attempts 3
也支持 agent 的 --provider-command、--lean-project、--lean-command 和超时选项。协议见 Lean Agent。
输入 JSONL 每行包含唯一 id、statement，可选 prelude。数据集必须可信；拒绝空集、重复 ID 和缺少字段。
输出目录必须不存在：results.jsonl 逐题记录证明、Lean 诊断、耗时和错误；summary.json 汇总成功率、异常数、尝试数、模型、Lean 版本及数据集 SHA-256。异常题保留在分母。指标为修复预算内 solved/tasks，不是独立采样 pass@k。顺序运行，暂不支持断点续跑。
不同实验应固定模型、预算、Lean/Mathlib、数据集与适配器版本。三个 smoke 题只验证流程，不代表数学推理能力。
无 LLM 的真实 Lean smoke 测试
lean-bench datasets/smoke.jsonl --output results/smoke-001 --provider-command '["python3","tests/mock_provider.py"]'
python3 -m unittest discover -s tests -v
mock 返回固定证明，仍由真实 Lean 检查。真实 LLM 需配置 endpoint/model 或适配器。
教科书形式化流水线
textbook/ 下有一套从教材 PDF 生成两类数据的工具：真正的 bench 目标（来自每节
的 Exercises，datasets/）和候选库（来自正文 Theorem/Proposition/Lemma/
Corollary，database/，只有被真正证明过之后才进最终的可复用 Lean 库
文件），配 Definition 清单（Mathlib 没有的概念会编译成真正可导入的 Lean 代码，
lib/Definitions.lean，不只是 JSON 元数据）。用法见
textbook/README.md。
状态（更新于 2026-09-18）
database/goodman-1.6.jsonl（正文来源的候选库，21 条）全部通过本地
Lean elaboration 检查。datasets/goodman-1.6-exercises.jsonl（Exercises 1.6 来
源的真正 bench 目标，14 条）刚生成，尚待你本机跑一遍 check_statements.py 验
证。两边定义清单中的幻觉 Mathlib 引用均已修正。