Split pipeline into real bench (exercises) vs library candidates (body)
Restructures the textbook pipeline around the distinction the body-derived dataset had been blurring: a bench should be actual unsolved problems, and a reusable premise library should only ever contain statements that have really been proven, the way Mathlib itself works. - library_candidates/goodman-1.6.jsonl (renamed from datasets/goodman-1.6.jsonl): the book's own Theorem/Proposition/Lemma/Corollary claims from the main text. Not bench targets -- premises waiting to be proven once via a real lean-bench run, then assembled into the reusable library by build_library.py (which already only keeps success:true entries with their checked proof, unchanged by this commit). - datasets/goodman-1.6-exercises.jsonl (new): the actual bench, sourced from the book's Exercises 1.6 block via two new scripts, textbook/extract_exercises.py (slices the Exercises block, lists its numbered items, drops a known false-positive class where a wrapped cross-reference like "...Proposition\n1.6.19." looks like a real exercise heading) and textbook/formalize_exercises.py (LLM step, given both the exercises and body text together since many exercises say "Prove Proposition N.N.N" instead of restating the claim). - Both formalize.py and formalize_exercises.py's validate() now enforce that every checklist item is covered by at least one output entry (included or explicitly excluded with a reason); formalize_exercises.py did not have this yet and a first real run silently dropped 7 of 15 checklist items with no trace before this was added. - All 12 model-produced exercise entries needed real fixes before commit, the same review pass applied to the body pilot: 3 exercises literally ask to prove an already-stated body claim (Exercise 1.6.1 -> Proposition 1.6.7, 1.6.2 -> Proposition 1.6.9, 1.6.7 -> Proposition 1.6.18, 1.6.9 -> Corollary 1.6.20, 1.6.15 -> Lemma 1.6.26) and now reuse that claim's own already-checked statement verbatim instead of an independently re-derived (and in one case strictly weaker) restatement; two entries (1.6.2, 1.6.11) had literal math prose like "I(m, n)" or "gcd(n1,...,nk)" leaked into the statement field, which is not valid Lean; several had untyped/ambiguous gcd or Prime usage fixed the same way the earlier body fixes were (Nat.Prime, IsCoprime, explicit casts, IsGreatest instead of max on a Set); and Exercise 1.6.12 part (a) turned out to be a discussion question whose unconditional-identity reading is actually false (6, 10, 15 are jointly coprime but not pairwise coprime), so it was moved to the excluded file with a reason instead of committed as a false theorem. - Definitions can now be real, compiled Lean, not only JSON metadata: Definition 1.6.22 (gcd of a list of integers, previously a null placeholder since no single Mathlib constant matches it) now has a real hand-written `def IsGCDOfList`, and textbook/build_definitions.py compiles any manifest's non-null lean_code entries into lib/Definitions.lean. goodman-1-6-24 (library) and the new goodman-1-6-ex-11-b/c, goodman-1-6-ex-12 (bench) all now use IsGCDOfList instead of repeating an inline existential. - textbook/README.md rewritten around the two-track design and records how this is meant to connect to lean-agent's eventual goal-directed proof search against the growing library (not yet built). - New tests for extract_exercises.py and build_definitions.py; full local suite still green (18/19, the one failure is the pre-existing, unrelated lean_agent import error in test_dataset.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
This commit is contained in:
+160
-72
@@ -1,35 +1,62 @@
|
||||
# Textbook formalization pipeline
|
||||
|
||||
把一本教科书(目前是 Goodman, *Algebra: Abstract and Concrete*)里正文的
|
||||
Theorem/Proposition/Lemma/Corollary 转成 lean-bench 能跑的任务 JSONL,配套
|
||||
Definition 清单(优先复用 Mathlib,Mathlib 没有的才手写),再由 lean-agent 在
|
||||
一个真正带 Mathlib 的 Lean 项目里验证/证明。这条流水线本身不跑 Lean,也不联网
|
||||
以外没有第三方依赖。
|
||||
Turns one book (currently Goodman, *Algebra: Abstract and Concrete*) into two
|
||||
distinct things `lean-bench`/`lean-agent` can use, not one:
|
||||
|
||||
## 这条流水线不做什么
|
||||
- **The bench** (`datasets/goodman-1.6-exercises.jsonl`): real problems the reader
|
||||
is meant to solve, sourced from the book's own **Exercises** at the end of each
|
||||
section. These are the actual targets `lean-bench` scores an agent against.
|
||||
- **The library candidates** (`library_candidates/goodman-1.6.jsonl`): the book's
|
||||
stated Theorem/Proposition/Lemma/Corollary claims from the section's main text.
|
||||
These are NOT bench targets -- they are premises waiting to be proven once and
|
||||
then reused, the way Mathlib itself is a library of already-proven facts an agent
|
||||
can cite while proving something new. An entry only belongs in the trustworthy,
|
||||
reusable library after `lean-bench` has actually proven it for real (see step 5);
|
||||
`build_library.py` already enforces this -- it only assembles entries whose
|
||||
`results.jsonl` says `success: true`, with the exact proof that was checked, never
|
||||
anything from this file directly.
|
||||
- **Definitions** (`textbook/*.definitions.json` + `lib/Definitions.lean`): a
|
||||
concept a bench/library statement depends on. When Mathlib already has it
|
||||
(`mathlib_ref` set), statements just use that name directly. When it doesn't
|
||||
(`mathlib_ref: null`), the manifest also carries real, hand-written Lean code
|
||||
(`lean_code`), compiled by `build_definitions.py` into an actual importable
|
||||
`.lean` file -- Lean can only ever verify a proof about a concept that exists as
|
||||
real Lean code somewhere, a JSON description of a definition is not enough for it
|
||||
to typecheck anything that mentions it.
|
||||
|
||||
- 不产出证明。生成的只是**待证命题**(`statement` 字段),证明留给
|
||||
`lean-agent`/`lean-bench` 去跑。
|
||||
- 不在 Cowork 沙盒里验证 Lean 语句能不能通过类型检查——沙盒里没有装
|
||||
Lean/Lake/Mathlib(架构也不兼容:本机 `~/.elan` 下是 macOS 二进制,沙盒是
|
||||
Linux VM,跑不了),所以 `check_statements.py` 必须在你本机、且是一个已经
|
||||
`lake exe cache get` 过的 Mathlib 项目里跑,见下面"本机环境准备"。
|
||||
- 不会把教科书 PDF 或者大段原文提取结果提交进 git——`.gitignore` 里已经排除
|
||||
了 `*.pdf` 和 `textbook/*.txt`,这些是版权内容,只应该留在你本机。只有代码、
|
||||
生成的最终数据集(短的 Lean 语句 + 自己改写的简短英文释义)会进仓库。
|
||||
This pipeline does not itself prove anything or run Lean; see "What this pipeline
|
||||
does not do" below and `lean-agent`'s own README for how proving actually happens
|
||||
(goal-directed proof search against the library, not a single-shot proof attempt --
|
||||
see "How this is meant to connect to lean-agent" at the end of this file).
|
||||
|
||||
## 用法
|
||||
## What this pipeline does not do
|
||||
|
||||
### 1. 从 PDF 提取一章的版式文本
|
||||
- Does not produce proofs. Every `statement` field is an unproven target; a Lean
|
||||
`sorry` would technically "complete" it, which is exactly why proving is left to
|
||||
`lean-agent`/`lean-bench`, not this pipeline.
|
||||
- Does not verify Lean statements type-check inside the Cowork sandbox -- there is
|
||||
no Lean/Lake/Mathlib there (also architecturally incompatible: your local
|
||||
`~/.elan` has macOS binaries, the sandbox is a Linux VM). `check_statements.py`
|
||||
must run on your machine against a Mathlib-backed Lean project; see "Local
|
||||
environment setup" below.
|
||||
- Does not commit the textbook PDF or large raw/derived full-text extraction dumps
|
||||
to git -- `.gitignore` excludes `*.pdf` and `textbook/*.txt`, since that's
|
||||
copyrighted material that should only ever live on your machine. Only code, the
|
||||
generated JSON/JSONL datasets (short Lean statements + your own short English
|
||||
glosses), and the compiled `Definitions.lean` go into git.
|
||||
|
||||
## Usage
|
||||
|
||||
### 1. Extract one chapter's layout text from the PDF
|
||||
|
||||
```sh
|
||||
pdftotext -layout -f <起始pdf页码> -l <结束pdf页码> uiowa-algebra.pdf textbook/chapterNN_layout.txt
|
||||
pdftotext -layout -f <start pdf page> -l <end pdf page> uiowa-algebra.pdf textbook/chapterNN_layout.txt
|
||||
```
|
||||
|
||||
页码用 `pdfinfo`/`pypdf` 先核对——书里印的页码和 PDF 物理页码不是一回事(前
|
||||
面有目录、序言)。
|
||||
Check page numbers with `pdfinfo`/`pypdf` first -- the book's printed page numbers
|
||||
and the PDF's physical page numbers differ (front matter, table of contents).
|
||||
|
||||
### 2. 切出一个 section,列出带编号的条目清单
|
||||
### 2. Slice out a section's body and its exercises separately
|
||||
|
||||
```sh
|
||||
python3 textbook/extract_section.py textbook/chapterNN_layout.txt \
|
||||
@@ -38,96 +65,157 @@ python3 textbook/extract_section.py textbook/chapterNN_layout.txt \
|
||||
--section-prefix "1.6." \
|
||||
--out-text textbook/section_1_6.txt \
|
||||
--out-index textbook/section_1_6.index.json
|
||||
|
||||
python3 textbook/extract_exercises.py textbook/chapterNN_layout.txt \
|
||||
--start-heading "Exercises 1.6" \
|
||||
--end-heading "1.7. Modular Arithmetic" \
|
||||
--section-prefix "1.6." \
|
||||
--out-text textbook/exercises_1_6.txt \
|
||||
--out-index textbook/exercises_1_6.index.json
|
||||
```
|
||||
|
||||
`--start-heading`/`--end-heading` 必须和版式文本里独占一行的章节标题**逐字**
|
||||
一致(提取时打印出的书本身的标题行);`--end-heading` 留空就切到文件末尾。
|
||||
输出的 index JSON 是这个 section 里所有 `Theorem/Proposition/Lemma/
|
||||
Corollary/Definition N.N.N` 开头的条目清单(自动去重了跨页重复的标签),用来
|
||||
让下一步的 LLM 不漏项、不编项目。
|
||||
Headings must match the layout text's own section-title lines **verbatim**.
|
||||
`extract_exercises.py` drops obvious false positives (a sentence like "...converse
|
||||
of Proposition\n1.6.19." can wrap so the bare number lands alone at a line start,
|
||||
looking exactly like a real exercise heading) -- anything dropped is printed to
|
||||
stderr, never silently lost.
|
||||
|
||||
### 3. 用 LLM 把 section 形式化成候选 Lean 语句
|
||||
### 3. Formalize the body (-> library candidates) and the exercises (-> bench) separately
|
||||
|
||||
```sh
|
||||
export DASHSCOPE_API_KEY=... # 或者写进 lean-bench/.env(已 gitignore)
|
||||
export DASHSCOPE_API_KEY=... # or put it in lean-bench/.env (gitignored)
|
||||
|
||||
python3 textbook/formalize.py \
|
||||
textbook/section_1_6.txt textbook/section_1_6.index.json \
|
||||
--model qwen-max --base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
|
||||
--out-dataset datasets/goodman-1.6.jsonl \
|
||||
--out-dataset library_candidates/goodman-1.6.jsonl \
|
||||
--out-definitions textbook/goodman-1.6.definitions.json \
|
||||
--out-excluded textbook/goodman-1.6.excluded.jsonl
|
||||
|
||||
python3 textbook/formalize_exercises.py \
|
||||
textbook/section_1_6.txt textbook/exercises_1_6.txt textbook/exercises_1_6.index.json \
|
||||
--model qwen-max --base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
|
||||
--out-dataset datasets/goodman-1.6-exercises.jsonl \
|
||||
--out-excluded textbook/goodman-1.6-exercises.excluded.jsonl
|
||||
```
|
||||
|
||||
模型会为每条 Theorem/Proposition/Lemma/Corollary 产出一个 Lean 4 命题
|
||||
(`statement`)、需要的 `prelude`(通常就是 `import Mathlib`)、一句自己改写
|
||||
的英文释义,以及用到的 Mathlib 标识符列表;为每条相关的 Definition 产出一个
|
||||
清单条目——如果 Mathlib 已经有对应概念(`mathlib_ref` 非空)就不写 Lean 代
|
||||
码,只有 Mathlib 没有的概念才会给出手写的 `def`。判断这条目根本没法形式化
|
||||
(纯计算题、内容太模糊/被 PDF 提取污染到读不出原意)的条目,会带着
|
||||
`include_in_bench: false` 和原因单独写进 `--out-excluded`,不会进真正的数据
|
||||
集,但也不会被静默丢弃——都留痕方便你复核。
|
||||
`formalize_exercises.py` is given the body text too, because many exercises say
|
||||
things like "Prove Proposition 1.6.9" instead of restating the claim -- the model
|
||||
needs the body to know what that actually says, and is told to write out the full
|
||||
claim, never a placeholder like "the statement of Proposition 1.6.9".
|
||||
|
||||
源 PDF 的数学符号提取质量很差(≡、∈、上下标经常丢失或乱码),这一步依赖模
|
||||
型自己对标准教材内容的知识去猜测/还原真实含义,**不是逐字翻译提取出来的文
|
||||
本**。所以生成的候选语句必须经过下一步验证,不能直接信。
|
||||
Both steps validate that every checklist item is covered by at least one output
|
||||
entry (included or explicitly excluded with a reason) and raise an error if the
|
||||
model silently drops one instead -- this caught a real bug during the exercises
|
||||
pilot (7 of 15 checklist items were dropped with no trace on the first attempt).
|
||||
|
||||
### 4. 在你本机验证候选语句能不能通过类型检查
|
||||
The source PDF's math symbol extraction is unreliable (missing/garbled ≡, ∈,
|
||||
superscripts), so this step leans on the model's own knowledge of standard textbook
|
||||
mathematics rather than literal transcription. Candidate statements must go through
|
||||
step 4 before you trust them -- in practice, even after that, expect to still need
|
||||
to read and fix a meaningful fraction by hand (both the exercises and the original
|
||||
body pilot needed real fixes to type mismatches, invalid syntax, and even one
|
||||
mathematically false claim smuggled in as a "prove or disprove" exercise that got
|
||||
mis-included as an unconditional theorem).
|
||||
|
||||
见下面"本机环境准备"先搭好一个带 Mathlib 的 Lake 项目,然后:
|
||||
### 4. Check candidate statements type-check on your machine
|
||||
|
||||
```sh
|
||||
python3 textbook/check_statements.py datasets/goodman-1.6.jsonl \
|
||||
python3 textbook/check_statements.py library_candidates/goodman-1.6.jsonl \
|
||||
--lean-project /path/to/your/mathlib-project \
|
||||
--lean-command '["lake","env","lean"]' \
|
||||
--report textbook/goodman-1.6.elab-report.jsonl
|
||||
|
||||
python3 textbook/check_statements.py datasets/goodman-1.6-exercises.jsonl \
|
||||
--lean-project /path/to/your/mathlib-project \
|
||||
--lean-command '["lake","env","lean"]' \
|
||||
--report textbook/goodman-1.6-exercises.elab-report.jsonl
|
||||
```
|
||||
|
||||
这一步只检查"这句 Lean 能不能通过类型检查"(用 `sorry` 占位),**不检查证明
|
||||
是否存在、也不检查是否忠实于原书**——分别对应两件不同的事:语法/引用是否正
|
||||
确,跟数学内容是否正确。失败的条目(`FAIL`)要么是模型编造了不存在的
|
||||
Mathlib 标识符,要么是语句本身写错了,需要你告诉我具体哪条、Lean 报的什么
|
||||
错,我再改。
|
||||
This only checks "does this Lean statement type-check" (via `sorry`), not "is it
|
||||
provable" or "is it faithful to the book" -- report any `FAIL` id and its Lean
|
||||
diagnostics and they'll get fixed and pushed back.
|
||||
|
||||
### 5. 用 lean-agent/lean-bench 真正证明
|
||||
### 5. Prove the bench targets for real with lean-agent/lean-bench
|
||||
|
||||
```sh
|
||||
lean-bench datasets/goodman-1.6.jsonl --output results/goodman-1.6-001 \
|
||||
lean-bench datasets/goodman-1.6-exercises.jsonl --output results/goodman-1.6-001 \
|
||||
--lean-project /path/to/your/mathlib-project --lean-command '["lake","env","lean"]' \
|
||||
--model ... --base-url ...
|
||||
```
|
||||
|
||||
### 6. 把证出来的定理组装成一份可复用的库文件
|
||||
### 6. Compile hand-written definitions into real, importable Lean
|
||||
|
||||
```sh
|
||||
python3 textbook/build_library.py datasets/goodman-1.6.jsonl \
|
||||
results/goodman-1.6-001/results.jsonl --out lib/Chapter01Section06.lean
|
||||
python3 textbook/build_definitions.py textbook/goodman-1.6.definitions.json --out lib/Definitions.lean
|
||||
```
|
||||
|
||||
只有这次跑真的证出来的定理(`results.jsonl` 里 `success: true`)才会被收进
|
||||
去,直接复用验证时确认过的那个证明,不会引入新的证明风险。后续章节要用这些
|
||||
已证定理时,`import` 这份文件——注意这只是把已证结果攒成一个文件,不是一个真
|
||||
正的 Lake 包/模块边界。
|
||||
Only definitions with no Mathlib equivalent (`mathlib_ref: null`) contribute
|
||||
anything here -- everything else is used directly by name. Any statement whose
|
||||
`prelude` needs one of these also inlines the `def` itself (self-contained per
|
||||
task, matching how `check_statements.py` elaborates one task at a time), so
|
||||
`lib/Definitions.lean` is the single canonical copy and the inlined ones must stay
|
||||
in sync with it by construction (generated from the same manifest).
|
||||
|
||||
## 本机环境准备:一个带 Mathlib 的 Lake 项目
|
||||
|
||||
Cowork 这边的沙盒没有 Lean,`check_statements.py` 和真正跑 `lean-agent`/
|
||||
`lean-bench` 都需要在你本机(或者别的有 Lean 的环境)完成。如果还没有这样一
|
||||
个项目:
|
||||
### 7. Assemble proven library candidates into a reusable library file
|
||||
|
||||
```sh
|
||||
cd ~/ProjectResearch # 随便一个地方
|
||||
python3 textbook/build_library.py library_candidates/goodman-1.6.jsonl \
|
||||
results/goodman-1.6-001-library/results.jsonl --out lib/Chapter01Section06.lean
|
||||
```
|
||||
|
||||
(This step needs its own `lean-bench` run over `library_candidates/`, proving those
|
||||
statements the same way step 5 proves the exercises -- not yet done for section
|
||||
1.6.) Only entries that were actually proven in that run go in, with the exact
|
||||
proof that was checked, verbatim. Later chapters `import` this file to build on
|
||||
already-proven results -- it's one flat `.lean` file, not a real Lake
|
||||
package/module boundary.
|
||||
|
||||
## How this is meant to connect to lean-agent
|
||||
|
||||
This pipeline only produces data files; it is not the proof search itself. The
|
||||
intended `lean-agent` workflow this feeds:
|
||||
|
||||
1. Given a target statement, check whether it's even valid Lean (this is what
|
||||
`check_statements.py` does ahead of time, so `lean-agent` doesn't waste attempts
|
||||
on a statement that can't type-check no matter what).
|
||||
2. Track the current proof state as a list of open goals (the same thing the
|
||||
[vscode-lean4](https://github.com/leanprover/vscode-lean4) plugin shows you
|
||||
live) -- not attempt the whole theorem in one LLM call.
|
||||
3. At each step, search a database of already-proven statements (`lib/*.lean`,
|
||||
built by `build_library.py`, plus Mathlib itself) for something applicable to
|
||||
the current goal -- the equivalent of Lean's `exact?`/`apply?` tactics, or a
|
||||
semantic/embedding search over the library -- and if something is used, apply it
|
||||
and fold the resulting subgoals back into the open-goals list.
|
||||
3. Repeat until the goal list is empty (theorem proved) or a budget is exhausted
|
||||
(forced stop, left as an open goal / `sorry`).
|
||||
|
||||
That proof-search loop isn't built yet -- `lean-bench`'s current `solve()` call is
|
||||
still single-shot (one LLM call, checked by Lean, retried up to `--attempts` times
|
||||
from scratch), not goal-directed. This pipeline's two-track split (bench targets
|
||||
from exercises, growing library from proven body content) is what that loop will
|
||||
need once it exists.
|
||||
|
||||
## Local environment setup: a Lake project with Mathlib
|
||||
|
||||
The Cowork sandbox has no Lean; `check_statements.py` and real `lean-agent`/
|
||||
`lean-bench` runs both need to happen on your machine (or anywhere else with Lean).
|
||||
If you don't have such a project yet:
|
||||
|
||||
```sh
|
||||
cd ~/ProjectResearch # anywhere
|
||||
lake +leanprover/lean4:v4.33.1 new mathlib-bench math
|
||||
cd mathlib-bench
|
||||
lake exe cache get # 下载 Mathlib 预编译缓存,比从源码编译快得多
|
||||
lake build
|
||||
lake exe cache get # downloads Mathlib's precompiled cache -- much faster than building from source
|
||||
lake build Mathlib # compiles Mathlib's own top-level .olean aggregator (cache get alone doesn't)
|
||||
```
|
||||
|
||||
之后 `--lean-project` 就指到 `mathlib-bench` 这个目录,`--lean-command` 用
|
||||
`'["lake","env","lean"]'`。
|
||||
Then point `--lean-project` at `mathlib-bench` and `--lean-command` at
|
||||
`'["lake","env","lean"]'`.
|
||||
|
||||
## 反馈循环
|
||||
## Feedback loop
|
||||
|
||||
我(claude-bot)在 ProjectResearch 里生成/修改数据集和代码 → 提交并 push 到
|
||||
`Zayden-Jung/lean-bench`(用 claude-bot 的 token) → 你在本机 `git pull` →
|
||||
跑 `check_statements.py` / `lean-bench` → 把失败的条目 id 和 Lean 报的错发
|
||||
给我 → 我改完再 push。不需要额外工具,就是正常的 git 往返。
|
||||
I (claude-bot) generate/edit files in ProjectResearch -> commit and push to
|
||||
`Zayden-Jung/lean-bench` (using claude-bot's token) -> you `git pull` locally ->
|
||||
run `check_statements.py` / `lean-bench` -> send back the failing ids and Lean's
|
||||
diagnostics -> I fix and push again. Ordinary git round-trip, no extra tooling.
|
||||
|
||||
Reference in New Issue
Block a user