69c5b788b92c5876ea0bfe3ff3e5772564b98d7f
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 claude-bot
|
69c5b788b9 |
Generate the citable premise database, and goal-driven retrieval over it
The database was the missing half: extraction and assembly existed, but nothing turned the book's main-content statements into something an agent could actually cite mid-proof, and nothing searched it. Two new scripts. textbook/build_database.py takes the main-content statements and emits both halves of a usable premise library: - an importable Lean file where every entry is a named declaration, so a proof can cite goodman_1_6_20 by name; - a JSONL search index carrying each entry's name, statement, mentioned identifiers and status. An entry is emitted as an `axiom` (asserted from the textbook) until a real lean-bench run proves it; passing --results upgrades those entries to real theorems carrying the exact verified proof. This mirrors how the book itself works -- its exercises are posed assuming its body results, the same way using Mathlib does not mean re-proving Mathlib. The cost is explicit: anything proven through an axiom inherits it, so a garbled extraction asserted as an axiom makes downstream proofs worthless. Guards: --require-checked refuses entries that have not passed check_statements.py (all 21 section 1.6 entries have, and are now marked accordingly), and the verifier's existing #print axioms check surfaces what a finished proof rests on. textbook/search_database.py is the retrieval step the proof loop calls at each line: given the current goal, it ranks premises by identifier and symbol overlap, weighting rare identifiers far above common ones (IsGCDOfList matching means much more than Int matching), preferring a proven premise over an asserted one, and formats the top few for the prompt. Verified against three real goals: the exactly-applicable premise tops each ranking by a wide margin (11.4 vs 1.6 for the gcd goal, 5.1 vs 3.7 for the prime-divides-product goal). It is a ranking heuristic, not a decision procedure -- only Lean can say whether a premise applies. Replacing score_premise() is the whole job of swapping in embeddings. Also: renamed library_candidates/ to database/ to match what it is; generated lib/Goodman/Chapter01Section06.lean (21 premises, 0 proven, 21 asserted) and database/goodman-1.6.index.jsonl; noted a latent bug in the older build_library.py, which attaches each entry's non-import prelude per-theorem and so would emit a shared `def` twice in one file -- build_database.py hoists and dedupes, and has a test pinning that. 13 new tests, all green (33 total, the one failure remains the pre-existing unrelated lean_agent import error in test_dataset.py). Still not done, and worth being clear about: nothing has been proven yet. The bench targets have not been elaboration-checked, no lean-bench run has happened over either file, and the goal-directed loop that would consume this retrieval does not exist -- lean-bench's solve() is still single-shot. Separately, lean-bench cannot currently run at all in this checkout, because lean_bench/cli.py imports lean_agent at module load and lean-agent is not installed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x |
||
|
claude-bot
|
04d827b3b3 |
Split pipeline into real bench (exercises) vs library candidates (body)
Restructures the textbook pipeline around the distinction the body-derived dataset had been blurring: a bench should be actual unsolved problems, and a reusable premise library should only ever contain statements that have really been proven, the way Mathlib itself works. - library_candidates/goodman-1.6.jsonl (renamed from datasets/goodman-1.6.jsonl): the book's own Theorem/Proposition/Lemma/Corollary claims from the main text. Not bench targets -- premises waiting to be proven once via a real lean-bench run, then assembled into the reusable library by build_library.py (which already only keeps success:true entries with their checked proof, unchanged by this commit). - datasets/goodman-1.6-exercises.jsonl (new): the actual bench, sourced from the book's Exercises 1.6 block via two new scripts, textbook/extract_exercises.py (slices the Exercises block, lists its numbered items, drops a known false-positive class where a wrapped cross-reference like "...Proposition\n1.6.19." looks like a real exercise heading) and textbook/formalize_exercises.py (LLM step, given both the exercises and body text together since many exercises say "Prove Proposition N.N.N" instead of restating the claim). - Both formalize.py and formalize_exercises.py's validate() now enforce that every checklist item is covered by at least one output entry (included or explicitly excluded with a reason); formalize_exercises.py did not have this yet and a first real run silently dropped 7 of 15 checklist items with no trace before this was added. - All 12 model-produced exercise entries needed real fixes before commit, the same review pass applied to the body pilot: 3 exercises literally ask to prove an already-stated body claim (Exercise 1.6.1 -> Proposition 1.6.7, 1.6.2 -> Proposition 1.6.9, 1.6.7 -> Proposition 1.6.18, 1.6.9 -> Corollary 1.6.20, 1.6.15 -> Lemma 1.6.26) and now reuse that claim's own already-checked statement verbatim instead of an independently re-derived (and in one case strictly weaker) restatement; two entries (1.6.2, 1.6.11) had literal math prose like "I(m, n)" or "gcd(n1,...,nk)" leaked into the statement field, which is not valid Lean; several had untyped/ambiguous gcd or Prime usage fixed the same way the earlier body fixes were (Nat.Prime, IsCoprime, explicit casts, IsGreatest instead of max on a Set); and Exercise 1.6.12 part (a) turned out to be a discussion question whose unconditional-identity reading is actually false (6, 10, 15 are jointly coprime but not pairwise coprime), so it was moved to the excluded file with a reason instead of committed as a false theorem. - Definitions can now be real, compiled Lean, not only JSON metadata: Definition 1.6.22 (gcd of a list of integers, previously a null placeholder since no single Mathlib constant matches it) now has a real hand-written `def IsGCDOfList`, and textbook/build_definitions.py compiles any manifest's non-null lean_code entries into lib/Definitions.lean. goodman-1-6-24 (library) and the new goodman-1-6-ex-11-b/c, goodman-1-6-ex-12 (bench) all now use IsGCDOfList instead of repeating an inline existential. - textbook/README.md rewritten around the two-track design and records how this is meant to connect to lean-agent's eventual goal-directed proof search against the growing library (not yet built). - New tests for extract_exercises.py and build_definitions.py; full local suite still green (18/19, the one failure is the pre-existing, unrelated lean_agent import error in test_dataset.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x |
||
|
claude-bot
|
79c728567a |
Fix 10 failing goodman-1.6 statements based on Lean compiler diagnostics
All 21 statements in datasets/goodman-1.6.jsonl now elaborate cleanly
(verified locally via check_statements.py, 11/21 passing before this
change). Fixes applied, one per failing id:
- goodman-1-6-7: single-binder existential-unique over a pair (qr : ZxZ)
instead of the invalid two-binder 'exists unique' form.
- goodman-1-6-10: rewrote set-builder notation to the explicit
{v | exists a b, v = ...} form.
- goodman-1-6-11: replaced 'let d := ... in ...' with the required
semicolon form 'let d := ...; ...', and added the missing Nat-to-Int
casts on d.
- goodman-1-6-17: added explicit Nat-to-Int coercions on the three
divisibility relations.
- goodman-1-6-18: changed p from Z to N (Nat.Prime requires N), added
casts, and replaced the nonexistent Int.Coprime with IsCoprime.
- goodman-1-6-19: changed p from Z to N and added the matching casts.
- goodman-1-6-20: changed p from Z to N, added casts, and fixed the
quantifier structure so the prime-divides-product hypothesis is not
nested inside the per-element forall.
- goodman-1-6-23: replaced the Fin n literal comparison 'j = 0' (which
needs a NeZero instance) with 'j.val = 0'.
- goodman-1-6-24: replaced the invalid bare '(*)' operator section with
'(fun x y => x * y)'-equivalent '(. * .)', and replaced the
nonexistent GCD.gcd with an explicit existential statement of the gcd
property (divides every element; every common divisor divides it).
- goodman-1-6-26: replaced the nonexistent Int.Coprime with IsCoprime.
Also corrected the three parallel hallucinated Mathlib refs in
textbook/goodman-1.6.definitions.json (Int.Coprime, GCD.gcd,
Int.PairwiseCoprime -> IsCoprime, null with notes, and
List.Pairwise IsCoprime respectively), since the Lean compiler confirms
none of the original names exist.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
|
||
|
claude-bot
|
f98cb41f72 |
Run textbook pipeline pilot on section 1.6 (Divisibility in the Integers)
Generated with formalize.py against Qwen (qwen-max via DashScope), then hand-fixed before pushing, since these classes of bug pass elaboration but are still wrong and check_statements.py can't catch them without running Lean: - goodman-1-6-2 split into 4 separate theorems (1-6-2-a..d): the model's single merged statement had an operator-precedence bug (missing parens around an implicit right-associated AND after OR) that silently scoped 3 of the book's 4 independent claims inside the wrong disjunct. - goodman-1-6-1: dropped two vacuous "x + y : Nat is a Nat" clauses that confused Lean's type membership with set membership (element-of Nat on already-Nat-typed variables) and are automatically true by typing anyway. - goodman-1-6-9: swapped Mathlib's pointwise Set multiplication (x * I, which needs 'open scoped Pointwise' and is easy to get subtly wrong) for the equivalent elementwise statement. - goodman-1-6-13: replaced a misuse of min (a 2-argument function) applied to a Set with Mathlib's IsLeast predicate, matching what "least element" means in Int.gcd's characterization. - goodman-1-6-23: fixed Matrix index types (Fin n, not a bare Nat) and replaced an illegal literal '...' row with Matrix.vecMul; verified against the book's raw text that the n-by-n unimodular-matrix formulation is faithful, not a model invention. textbook/goodman-1.6.raw.jsonl keeps the unedited model output for reference. The remaining 13 statements are unedited and still need a real Lean elaboration pass (check_statements.py) plus a provability run (lean-bench) on a Mathlib-backed project -- neither ran here since this sandbox has no Lean. |
||
|
claude-bot
|
caedf2e6f4 |
Add textbook formalization pipeline (extract -> LLM formalize -> elaboration check -> library assembly)
- extract_section.py: slice a book section out of a pdftotext -layout dump and list its labeled Theorem/Proposition/Lemma/Corollary/Definition items. - formalize.py: LLM-formalize a section into Lean 4 candidate statements (preferring Mathlib vocabulary) plus a definitions manifest. - check_statements.py: elaboration-only sanity check against a real Mathlib-backed Lean project (must run outside the Cowork sandbox, which has no Lean toolchain and can't run macOS binaries anyway). - build_library.py: assemble solved lean-bench results into a reusable .lean file, reusing each winning proof verbatim. - llm.py: standalone OpenAI-compatible chat client + .env loader, kept independent of lean_agent so this pipeline works without it installed. - Source PDFs and raw/layout text dumps are gitignored (copyrighted material); only code and the final short Lean statement datasets are tracked. |