The database was the missing half: extraction and assembly existed, but
nothing turned the book's main-content statements into something an agent
could actually cite mid-proof, and nothing searched it. Two new scripts.
textbook/build_database.py takes the main-content statements and emits
both halves of a usable premise library:
- an importable Lean file where every entry is a named declaration, so a
proof can cite goodman_1_6_20 by name;
- a JSONL search index carrying each entry's name, statement, mentioned
identifiers and status.
An entry is emitted as an `axiom` (asserted from the textbook) until a
real lean-bench run proves it; passing --results upgrades those entries
to real theorems carrying the exact verified proof. This mirrors how the
book itself works -- its exercises are posed assuming its body results,
the same way using Mathlib does not mean re-proving Mathlib. The cost is
explicit: anything proven through an axiom inherits it, so a garbled
extraction asserted as an axiom makes downstream proofs worthless.
Guards: --require-checked refuses entries that have not passed
check_statements.py (all 21 section 1.6 entries have, and are now marked
accordingly), and the verifier's existing #print axioms check surfaces
what a finished proof rests on.
textbook/search_database.py is the retrieval step the proof loop calls at
each line: given the current goal, it ranks premises by identifier and
symbol overlap, weighting rare identifiers far above common ones
(IsGCDOfList matching means much more than Int matching), preferring a
proven premise over an asserted one, and formats the top few for the
prompt. Verified against three real goals: the exactly-applicable premise
tops each ranking by a wide margin (11.4 vs 1.6 for the gcd goal, 5.1 vs
3.7 for the prime-divides-product goal). It is a ranking heuristic, not a
decision procedure -- only Lean can say whether a premise applies.
Replacing score_premise() is the whole job of swapping in embeddings.
Also: renamed library_candidates/ to database/ to match what it is;
generated lib/Goodman/Chapter01Section06.lean (21 premises, 0 proven, 21
asserted) and database/goodman-1.6.index.jsonl; noted a latent bug in the
older build_library.py, which attaches each entry's non-import prelude
per-theorem and so would emit a shared `def` twice in one file --
build_database.py hoists and dedupes, and has a test pinning that.
13 new tests, all green (33 total, the one failure remains the
pre-existing unrelated lean_agent import error in test_dataset.py).
Still not done, and worth being clear about: nothing has been proven yet.
The bench targets have not been elaboration-checked, no lean-bench run
has happened over either file, and the goal-directed loop that would
consume this retrieval does not exist -- lean-bench's solve() is still
single-shot. Separately, lean-bench cannot currently run at all in this
checkout, because lean_bench/cli.py imports lean_agent at module load and
lean-agent is not installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
Restructures the textbook pipeline around the distinction the body-derived
dataset had been blurring: a bench should be actual unsolved problems, and
a reusable premise library should only ever contain statements that have
really been proven, the way Mathlib itself works.
- library_candidates/goodman-1.6.jsonl (renamed from datasets/goodman-1.6.jsonl):
the book's own Theorem/Proposition/Lemma/Corollary claims from the main
text. Not bench targets -- premises waiting to be proven once via a real
lean-bench run, then assembled into the reusable library by
build_library.py (which already only keeps success:true entries with
their checked proof, unchanged by this commit).
- datasets/goodman-1.6-exercises.jsonl (new): the actual bench, sourced
from the book's Exercises 1.6 block via two new scripts,
textbook/extract_exercises.py (slices the Exercises block, lists its
numbered items, drops a known false-positive class where a wrapped
cross-reference like "...Proposition\n1.6.19." looks like a real
exercise heading) and textbook/formalize_exercises.py (LLM step, given
both the exercises and body text together since many exercises say
"Prove Proposition N.N.N" instead of restating the claim).
- Both formalize.py and formalize_exercises.py's validate() now enforce
that every checklist item is covered by at least one output entry
(included or explicitly excluded with a reason); formalize_exercises.py
did not have this yet and a first real run silently dropped 7 of 15
checklist items with no trace before this was added.
- All 12 model-produced exercise entries needed real fixes before commit,
the same review pass applied to the body pilot: 3 exercises literally
ask to prove an already-stated body claim (Exercise 1.6.1 -> Proposition
1.6.7, 1.6.2 -> Proposition 1.6.9, 1.6.7 -> Proposition 1.6.18, 1.6.9 ->
Corollary 1.6.20, 1.6.15 -> Lemma 1.6.26) and now reuse that claim's own
already-checked statement verbatim instead of an independently
re-derived (and in one case strictly weaker) restatement; two entries
(1.6.2, 1.6.11) had literal math prose like "I(m, n)" or
"gcd(n1,...,nk)" leaked into the statement field, which is not valid
Lean; several had untyped/ambiguous gcd or Prime usage fixed the same
way the earlier body fixes were (Nat.Prime, IsCoprime, explicit casts,
IsGreatest instead of max on a Set); and Exercise 1.6.12 part (a) turned
out to be a discussion question whose unconditional-identity reading is
actually false (6, 10, 15 are jointly coprime but not pairwise coprime),
so it was moved to the excluded file with a reason instead of committed
as a false theorem.
- Definitions can now be real, compiled Lean, not only JSON metadata:
Definition 1.6.22 (gcd of a list of integers, previously a null
placeholder since no single Mathlib constant matches it) now has a real
hand-written `def IsGCDOfList`, and textbook/build_definitions.py
compiles any manifest's non-null lean_code entries into
lib/Definitions.lean. goodman-1-6-24 (library) and the new
goodman-1-6-ex-11-b/c, goodman-1-6-ex-12 (bench) all now use
IsGCDOfList instead of repeating an inline existential.
- textbook/README.md rewritten around the two-track design and records
how this is meant to connect to lean-agent's eventual goal-directed
proof search against the growing library (not yet built).
- New tests for extract_exercises.py and build_definitions.py; full
local suite still green (18/19, the one failure is the pre-existing,
unrelated lean_agent import error in test_dataset.py).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
- extract_section.py: slice a book section out of a pdftotext -layout dump and
list its labeled Theorem/Proposition/Lemma/Corollary/Definition items.
- formalize.py: LLM-formalize a section into Lean 4 candidate statements
(preferring Mathlib vocabulary) plus a definitions manifest.
- check_statements.py: elaboration-only sanity check against a real
Mathlib-backed Lean project (must run outside the Cowork sandbox, which has
no Lean toolchain and can't run macOS binaries anyway).
- build_library.py: assemble solved lean-bench results into a reusable .lean
file, reusing each winning proof verbatim.
- llm.py: standalone OpenAI-compatible chat client + .env loader, kept
independent of lean_agent so this pipeline works without it installed.
- Source PDFs and raw/layout text dumps are gitignored (copyrighted material);
only code and the final short Lean statement datasets are tracked.
claude-bot
codex-bot