Split pipeline into real bench (exercises) vs library candidates (body)

Restructures the textbook pipeline around the distinction the body-derived
dataset had been blurring: a bench should be actual unsolved problems, and
a reusable premise library should only ever contain statements that have
really been proven, the way Mathlib itself works.

- library_candidates/goodman-1.6.jsonl (renamed from datasets/goodman-1.6.jsonl):
  the book's own Theorem/Proposition/Lemma/Corollary claims from the main
  text. Not bench targets -- premises waiting to be proven once via a real
  lean-bench run, then assembled into the reusable library by
  build_library.py (which already only keeps success:true entries with
  their checked proof, unchanged by this commit).
- datasets/goodman-1.6-exercises.jsonl (new): the actual bench, sourced
  from the book's Exercises 1.6 block via two new scripts,
  textbook/extract_exercises.py (slices the Exercises block, lists its
  numbered items, drops a known false-positive class where a wrapped
  cross-reference like "...Proposition\n1.6.19." looks like a real
  exercise heading) and textbook/formalize_exercises.py (LLM step, given
  both the exercises and body text together since many exercises say
  "Prove Proposition N.N.N" instead of restating the claim).
- Both formalize.py and formalize_exercises.py's validate() now enforce
  that every checklist item is covered by at least one output entry
  (included or explicitly excluded with a reason); formalize_exercises.py
  did not have this yet and a first real run silently dropped 7 of 15
  checklist items with no trace before this was added.
- All 12 model-produced exercise entries needed real fixes before commit,
  the same review pass applied to the body pilot: 3 exercises literally
  ask to prove an already-stated body claim (Exercise 1.6.1 -> Proposition
  1.6.7, 1.6.2 -> Proposition 1.6.9, 1.6.7 -> Proposition 1.6.18, 1.6.9 ->
  Corollary 1.6.20, 1.6.15 -> Lemma 1.6.26) and now reuse that claim's own
  already-checked statement verbatim instead of an independently
  re-derived (and in one case strictly weaker) restatement; two entries
  (1.6.2, 1.6.11) had literal math prose like "I(m, n)" or
  "gcd(n1,...,nk)" leaked into the statement field, which is not valid
  Lean; several had untyped/ambiguous gcd or Prime usage fixed the same
  way the earlier body fixes were (Nat.Prime, IsCoprime, explicit casts,
  IsGreatest instead of max on a Set); and Exercise 1.6.12 part (a) turned
  out to be a discussion question whose unconditional-identity reading is
  actually false (6, 10, 15 are jointly coprime but not pairwise coprime),
  so it was moved to the excluded file with a reason instead of committed
  as a false theorem.
- Definitions can now be real, compiled Lean, not only JSON metadata:
  Definition 1.6.22 (gcd of a list of integers, previously a null
  placeholder since no single Mathlib constant matches it) now has a real
  hand-written `def IsGCDOfList`, and textbook/build_definitions.py
  compiles any manifest's non-null lean_code entries into
  lib/Definitions.lean. goodman-1-6-24 (library) and the new
  goodman-1-6-ex-11-b/c, goodman-1-6-ex-12 (bench) all now use
  IsGCDOfList instead of repeating an inline existential.
- textbook/README.md rewritten around the two-track design and records
  how this is meant to connect to lean-agent's eventual goal-directed
  proof search against the growing library (not yet built).
- New tests for extract_exercises.py and build_definitions.py; full
  local suite still green (18/19, the one failure is the pre-existing,
  unrelated lean_agent import error in test_dataset.py).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
This commit is contained in:
2026-09-18 19:27:40 +00:00
parent 5aa048593c
commit 04d827b3b3
14 changed files with 702 additions and 81 deletions
+160 -72
View File
@@ -1,35 +1,62 @@
# Textbook formalization pipeline
把一本教科书(目前是 Goodman, *Algebra: Abstract and Concrete*)里正文的
Theorem/Proposition/Lemma/Corollary 转成 lean-bench 能跑的任务 JSONL,配套
Definition 清单(优先复用 Mathlib,Mathlib 没有的才手写),再由 lean-agent 在
一个真正带 Mathlib 的 Lean 项目里验证/证明。这条流水线本身不跑 Lean,也不联网
以外没有第三方依赖。
Turns one book (currently Goodman, *Algebra: Abstract and Concrete*) into two
distinct things `lean-bench`/`lean-agent` can use, not one:
## 这条流水线不做什么
- **The bench** (`datasets/goodman-1.6-exercises.jsonl`): real problems the reader
is meant to solve, sourced from the book's own **Exercises** at the end of each
section. These are the actual targets `lean-bench` scores an agent against.
- **The library candidates** (`library_candidates/goodman-1.6.jsonl`): the book's
stated Theorem/Proposition/Lemma/Corollary claims from the section's main text.
These are NOT bench targets -- they are premises waiting to be proven once and
then reused, the way Mathlib itself is a library of already-proven facts an agent
can cite while proving something new. An entry only belongs in the trustworthy,
reusable library after `lean-bench` has actually proven it for real (see step 5);
`build_library.py` already enforces this -- it only assembles entries whose
`results.jsonl` says `success: true`, with the exact proof that was checked, never
anything from this file directly.
- **Definitions** (`textbook/*.definitions.json` + `lib/Definitions.lean`): a
concept a bench/library statement depends on. When Mathlib already has it
(`mathlib_ref` set), statements just use that name directly. When it doesn't
(`mathlib_ref: null`), the manifest also carries real, hand-written Lean code
(`lean_code`), compiled by `build_definitions.py` into an actual importable
`.lean` file -- Lean can only ever verify a proof about a concept that exists as
real Lean code somewhere, a JSON description of a definition is not enough for it
to typecheck anything that mentions it.
- 不产出证明。生成的只是**待证命题**(`statement` 字段),证明留给
`lean-agent`/`lean-bench` 去跑。
- 不在 Cowork 沙盒里验证 Lean 语句能不能通过类型检查——沙盒里没有装
Lean/Lake/Mathlib(架构也不兼容:本机 `~/.elan` 下是 macOS 二进制,沙盒是
Linux VM,跑不了),所以 `check_statements.py` 必须在你本机、且是一个已经
`lake exe cache get` 过的 Mathlib 项目里跑,见下面"本机环境准备"。
- 不会把教科书 PDF 或者大段原文提取结果提交进 git——`.gitignore` 里已经排除
了 `*.pdf` 和 `textbook/*.txt`,这些是版权内容,只应该留在你本机。只有代码、
生成的最终数据集(短的 Lean 语句 + 自己改写的简短英文释义)会进仓库。
This pipeline does not itself prove anything or run Lean; see "What this pipeline
does not do" below and `lean-agent`'s own README for how proving actually happens
(goal-directed proof search against the library, not a single-shot proof attempt --
see "How this is meant to connect to lean-agent" at the end of this file).
## 用法
## What this pipeline does not do
### 1. 从 PDF 提取一章的版式文本
- Does not produce proofs. Every `statement` field is an unproven target; a Lean
`sorry` would technically "complete" it, which is exactly why proving is left to
`lean-agent`/`lean-bench`, not this pipeline.
- Does not verify Lean statements type-check inside the Cowork sandbox -- there is
no Lean/Lake/Mathlib there (also architecturally incompatible: your local
`~/.elan` has macOS binaries, the sandbox is a Linux VM). `check_statements.py`
must run on your machine against a Mathlib-backed Lean project; see "Local
environment setup" below.
- Does not commit the textbook PDF or large raw/derived full-text extraction dumps
to git -- `.gitignore` excludes `*.pdf` and `textbook/*.txt`, since that's
copyrighted material that should only ever live on your machine. Only code, the
generated JSON/JSONL datasets (short Lean statements + your own short English
glosses), and the compiled `Definitions.lean` go into git.
## Usage
### 1. Extract one chapter's layout text from the PDF
```sh
pdftotext -layout -f <起始pdf页码> -l <结束pdf页码> uiowa-algebra.pdf textbook/chapterNN_layout.txt
pdftotext -layout -f <start pdf page> -l <end pdf page> uiowa-algebra.pdf textbook/chapterNN_layout.txt
```
页码用 `pdfinfo`/`pypdf` 先核对——书里印的页码和 PDF 物理页码不是一回事(前
面有目录、序言)。
Check page numbers with `pdfinfo`/`pypdf` first -- the book's printed page numbers
and the PDF's physical page numbers differ (front matter, table of contents).
### 2. 切出一个 section,列出带编号的条目清单
### 2. Slice out a section's body and its exercises separately
```sh
python3 textbook/extract_section.py textbook/chapterNN_layout.txt \
@@ -38,96 +65,157 @@ python3 textbook/extract_section.py textbook/chapterNN_layout.txt \
--section-prefix "1.6." \
--out-text textbook/section_1_6.txt \
--out-index textbook/section_1_6.index.json
python3 textbook/extract_exercises.py textbook/chapterNN_layout.txt \
--start-heading "Exercises 1.6" \
--end-heading "1.7. Modular Arithmetic" \
--section-prefix "1.6." \
--out-text textbook/exercises_1_6.txt \
--out-index textbook/exercises_1_6.index.json
```
`--start-heading`/`--end-heading` 必须和版式文本里独占一行的章节标题**逐字**
一致(提取时打印出的书本身的标题行);`--end-heading` 留空就切到文件末尾。
输出的 index JSON 是这个 section 里所有 `Theorem/Proposition/Lemma/
Corollary/Definition N.N.N` 开头的条目清单(自动去重了跨页重复的标签),用来
让下一步的 LLM 不漏项、不编项目。
Headings must match the layout text's own section-title lines **verbatim**.
`extract_exercises.py` drops obvious false positives (a sentence like "...converse
of Proposition\n1.6.19." can wrap so the bare number lands alone at a line start,
looking exactly like a real exercise heading) -- anything dropped is printed to
stderr, never silently lost.
### 3. 用 LLM 把 section 形式化成候选 Lean 语句
### 3. Formalize the body (-> library candidates) and the exercises (-> bench) separately
```sh
export DASHSCOPE_API_KEY=... # 或者写进 lean-bench/.env(已 gitignore)
export DASHSCOPE_API_KEY=... # or put it in lean-bench/.env (gitignored)
python3 textbook/formalize.py \
textbook/section_1_6.txt textbook/section_1_6.index.json \
--model qwen-max --base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
--out-dataset datasets/goodman-1.6.jsonl \
--out-dataset library_candidates/goodman-1.6.jsonl \
--out-definitions textbook/goodman-1.6.definitions.json \
--out-excluded textbook/goodman-1.6.excluded.jsonl
python3 textbook/formalize_exercises.py \
textbook/section_1_6.txt textbook/exercises_1_6.txt textbook/exercises_1_6.index.json \
--model qwen-max --base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
--out-dataset datasets/goodman-1.6-exercises.jsonl \
--out-excluded textbook/goodman-1.6-exercises.excluded.jsonl
```
模型会为每条 Theorem/Proposition/Lemma/Corollary 产出一个 Lean 4 命题
(`statement`)、需要的 `prelude`(通常就是 `import Mathlib`)、一句自己改写
的英文释义,以及用到的 Mathlib 标识符列表;为每条相关的 Definition 产出一个
清单条目——如果 Mathlib 已经有对应概念(`mathlib_ref` 非空)就不写 Lean 代
码,只有 Mathlib 没有的概念才会给出手写的 `def`。判断这条目根本没法形式化
(纯计算题、内容太模糊/被 PDF 提取污染到读不出原意)的条目,会带着
`include_in_bench: false` 和原因单独写进 `--out-excluded`,不会进真正的数据
集,但也不会被静默丢弃——都留痕方便你复核。
`formalize_exercises.py` is given the body text too, because many exercises say
things like "Prove Proposition 1.6.9" instead of restating the claim -- the model
needs the body to know what that actually says, and is told to write out the full
claim, never a placeholder like "the statement of Proposition 1.6.9".
源 PDF 的数学符号提取质量很差(≡、∈、上下标经常丢失或乱码),这一步依赖模
型自己对标准教材内容的知识去猜测/还原真实含义,**不是逐字翻译提取出来的文
本**。所以生成的候选语句必须经过下一步验证,不能直接信。
Both steps validate that every checklist item is covered by at least one output
entry (included or explicitly excluded with a reason) and raise an error if the
model silently drops one instead -- this caught a real bug during the exercises
pilot (7 of 15 checklist items were dropped with no trace on the first attempt).
### 4. 在你本机验证候选语句能不能通过类型检查
The source PDF's math symbol extraction is unreliable (missing/garbled ≡, ∈,
superscripts), so this step leans on the model's own knowledge of standard textbook
mathematics rather than literal transcription. Candidate statements must go through
step 4 before you trust them -- in practice, even after that, expect to still need
to read and fix a meaningful fraction by hand (both the exercises and the original
body pilot needed real fixes to type mismatches, invalid syntax, and even one
mathematically false claim smuggled in as a "prove or disprove" exercise that got
mis-included as an unconditional theorem).
见下面"本机环境准备"先搭好一个带 Mathlib 的 Lake 项目,然后:
### 4. Check candidate statements type-check on your machine
```sh
python3 textbook/check_statements.py datasets/goodman-1.6.jsonl \
python3 textbook/check_statements.py library_candidates/goodman-1.6.jsonl \
--lean-project /path/to/your/mathlib-project \
--lean-command '["lake","env","lean"]' \
--report textbook/goodman-1.6.elab-report.jsonl
python3 textbook/check_statements.py datasets/goodman-1.6-exercises.jsonl \
--lean-project /path/to/your/mathlib-project \
--lean-command '["lake","env","lean"]' \
--report textbook/goodman-1.6-exercises.elab-report.jsonl
```
这一步只检查"这句 Lean 能不能通过类型检查"(用 `sorry` 占位),**不检查证明
是否存在、也不检查是否忠实于原书**——分别对应两件不同的事:语法/引用是否正
确,跟数学内容是否正确。失败的条目(`FAIL`)要么是模型编造了不存在的
Mathlib 标识符,要么是语句本身写错了,需要你告诉我具体哪条、Lean 报的什么
错,我再改。
This only checks "does this Lean statement type-check" (via `sorry`), not "is it
provable" or "is it faithful to the book" -- report any `FAIL` id and its Lean
diagnostics and they'll get fixed and pushed back.
### 5. 用 lean-agent/lean-bench 真正证明
### 5. Prove the bench targets for real with lean-agent/lean-bench
```sh
lean-bench datasets/goodman-1.6.jsonl --output results/goodman-1.6-001 \
lean-bench datasets/goodman-1.6-exercises.jsonl --output results/goodman-1.6-001 \
--lean-project /path/to/your/mathlib-project --lean-command '["lake","env","lean"]' \
--model ... --base-url ...
```
### 6. 把证出来的定理组装成一份可复用的库文件
### 6. Compile hand-written definitions into real, importable Lean
```sh
python3 textbook/build_library.py datasets/goodman-1.6.jsonl \
results/goodman-1.6-001/results.jsonl --out lib/Chapter01Section06.lean
python3 textbook/build_definitions.py textbook/goodman-1.6.definitions.json --out lib/Definitions.lean
```
只有这次跑真的证出来的定理(`results.jsonl` 里 `success: true`)才会被收进
去,直接复用验证时确认过的那个证明,不会引入新的证明风险。后续章节要用这些
已证定理时,`import` 这份文件——注意这只是把已证结果攒成一个文件,不是一个真
正的 Lake 包/模块边界。
Only definitions with no Mathlib equivalent (`mathlib_ref: null`) contribute
anything here -- everything else is used directly by name. Any statement whose
`prelude` needs one of these also inlines the `def` itself (self-contained per
task, matching how `check_statements.py` elaborates one task at a time), so
`lib/Definitions.lean` is the single canonical copy and the inlined ones must stay
in sync with it by construction (generated from the same manifest).
## 本机环境准备:一个带 Mathlib 的 Lake 项目
Cowork 这边的沙盒没有 Lean,`check_statements.py` 和真正跑 `lean-agent`/
`lean-bench` 都需要在你本机(或者别的有 Lean 的环境)完成。如果还没有这样一
个项目:
### 7. Assemble proven library candidates into a reusable library file
```sh
cd ~/ProjectResearch # 随便一个地方
python3 textbook/build_library.py library_candidates/goodman-1.6.jsonl \
results/goodman-1.6-001-library/results.jsonl --out lib/Chapter01Section06.lean
```
(This step needs its own `lean-bench` run over `library_candidates/`, proving those
statements the same way step 5 proves the exercises -- not yet done for section
1.6.) Only entries that were actually proven in that run go in, with the exact
proof that was checked, verbatim. Later chapters `import` this file to build on
already-proven results -- it's one flat `.lean` file, not a real Lake
package/module boundary.
## How this is meant to connect to lean-agent
This pipeline only produces data files; it is not the proof search itself. The
intended `lean-agent` workflow this feeds:
1. Given a target statement, check whether it's even valid Lean (this is what
`check_statements.py` does ahead of time, so `lean-agent` doesn't waste attempts
on a statement that can't type-check no matter what).
2. Track the current proof state as a list of open goals (the same thing the
[vscode-lean4](https://github.com/leanprover/vscode-lean4) plugin shows you
live) -- not attempt the whole theorem in one LLM call.
3. At each step, search a database of already-proven statements (`lib/*.lean`,
built by `build_library.py`, plus Mathlib itself) for something applicable to
the current goal -- the equivalent of Lean's `exact?`/`apply?` tactics, or a
semantic/embedding search over the library -- and if something is used, apply it
and fold the resulting subgoals back into the open-goals list.
3. Repeat until the goal list is empty (theorem proved) or a budget is exhausted
(forced stop, left as an open goal / `sorry`).
That proof-search loop isn't built yet -- `lean-bench`'s current `solve()` call is
still single-shot (one LLM call, checked by Lean, retried up to `--attempts` times
from scratch), not goal-directed. This pipeline's two-track split (bench targets
from exercises, growing library from proven body content) is what that loop will
need once it exists.
## Local environment setup: a Lake project with Mathlib
The Cowork sandbox has no Lean; `check_statements.py` and real `lean-agent`/
`lean-bench` runs both need to happen on your machine (or anywhere else with Lean).
If you don't have such a project yet:
```sh
cd ~/ProjectResearch # anywhere
lake +leanprover/lean4:v4.33.1 new mathlib-bench math
cd mathlib-bench
lake exe cache get # 下载 Mathlib 预编译缓存,比从源码编译快得多
lake build
lake exe cache get # downloads Mathlib's precompiled cache -- much faster than building from source
lake build Mathlib # compiles Mathlib's own top-level .olean aggregator (cache get alone doesn't)
```
之后 `--lean-project` 就指到 `mathlib-bench` 这个目录,`--lean-command` 用
`'["lake","env","lean"]'`。
Then point `--lean-project` at `mathlib-bench` and `--lean-command` at
`'["lake","env","lean"]'`.
## 反馈循环
## Feedback loop
我(claude-bot)在 ProjectResearch 里生成/修改数据集和代码 → 提交并 push 到
`Zayden-Jung/lean-bench`(用 claude-bot 的 token) → 你在本机 `git pull` →
跑 `check_statements.py` / `lean-bench` → 把失败的条目 id 和 Lean 报的错发
给我 → 我改完再 push。不需要额外工具,就是正常的 git 往返。
I (claude-bot) generate/edit files in ProjectResearch -> commit and push to
`Zayden-Jung/lean-bench` (using claude-bot's token) -> you `git pull` locally ->
run `check_statements.py` / `lean-bench` -> send back the failing ids and Lean's
diagnostics -> I fix and push again. Ordinary git round-trip, no extra tooling.