9e0457a0e1554415263637b85e3ccd3e85ca4504
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 claude-bot
|
9e0457a0e1 |
Drop the generated search index; derive it from the source instead
The index file was 12KB of which ~96% was a byte-for-byte copy of
database/goodman-1.6.jsonl. Its only original content was three derived
fields: `name` (literally id.replace('-','_')), `status` (a two-line
computation over proof_checked), and `identifiers` (a regex over the
statement). It was a cache for work that costs microseconds, and it
created a real hazard: nothing checked that it matched its source, so
editing a statement and forgetting to regenerate would leave
search_database.py serving a premise whose text no longer matched the
generated .lean file.
- search_database.py now reads database/goodman-1.6.jsonl directly via a
new load_database(), deriving name/identifiers/status on the fly and
importing those functions from build_database.py so there is one
definition of each rather than two that can drift.
- build_database.py drops --out-index and index generation, and its
proof/status derivation is factored into resolve_proof() shared with
retrieval.
- database/goodman-1.6.index.jsonl deleted.
Also fixes a retrieval bug the removal surfaced: the tokenizer only
matched ASCII identifiers, so a goal that is pure algebra
("a * (b + c) = a * b + a * c") tokenized to nothing and retrieved
nothing at all -- it now keeps type and operator symbols, and drops
single-letter variable names, which say nothing when matched. That goal
now correctly finds Proposition 1.6.1, and the identifier-rich goals
still rank their exact premise first by a wide margin (9.3 vs 4.4).
Tests restructured accordingly: the proof gate is now pinned directly on
resolve_proof in both directions, plus new coverage for load_database
deriving status from proof_checked and for the symbol tokenization.
49 tests, 48 green (the one failure remains the pre-existing unrelated
lean_agent import error in test_dataset.py).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
|
||
|
claude-bot
|
fbcb0c3383 |
Record and gate proofs, so "proven" means Lean verified it
The index had no proof field and every entry was an axiom, so the database asserted its premises rather than proving them. Adds the proof half, with the verification gate that makes the status trustworthy. - Database records now carry `proof` and `proof_checked`, and the search index exposes the proof, with status one of proved / proof_pending / asserted. build_database.py emits a real `theorem` only when a proof has been verified (by check_proofs.py, or by a lean-bench run via --results); a recorded but unverified proof stays an `axiom` and is reported as proof_pending, so the library never claims more than Lean has confirmed. A test pins that gating in both directions. - textbook/check_proofs.py verifies proofs by compiling `theorem ... := by <proof>`. Stricter than check_statements.py in three ways: it rejects sorry/admit/native_decide outright, and it reads `#print axioms` to reject any proof depending on more than Lean's own three axioms (propext, Classical.choice, Quot.sound) -- which in particular catches a proof leaning on one of this database's own book-asserted axioms, since that is not an independent proof. With --update it writes proof_checked back, promoting entries on the next build. - Ten candidate proofs recorded, all currently proof_checked=false pending a Lean run: goodman-1-6-1 (the eight algebraic identities are `ring`, plus a short argument for the nonzero conjunct), the four parts of Proposition 1.6.2 (Int.mul_eq_one_iff_eq_one_or_neg_one, associated_of_dvd_dvd with Int.associated_iff, dvd_trans, dvd_add with mul_left), 1-6-4 (Nat.exists_prime_and_dvd), 1-6-15 (Int.gcd_eq_one_iff_coprime, where IsCoprime unfolds to exactly the target existential), 1-6-17 (IsCoprime.mul_dvd via Nat.isCoprime_iff_coprime), 1-6-19 (Int.Prime.dvd_mul') and 1-6-20 (Prime.dvd_prod_iff). These are citations of existing Mathlib results, which is the realistic way to populate a textbook premise library -- the remaining 11 (1-6-6, -7, -9, -10, -11, -13, -18, -21, -23, -24, -26) are the ones needing actual work and carry no proof yet. Written without Lean available here, so expect a real failure rate on first compile -- run check_proofs.py --update and send back the FAILED ids with diagnostics, same loop as the statement fixes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x |
||
|
claude-bot
|
69c5b788b9 |
Generate the citable premise database, and goal-driven retrieval over it
The database was the missing half: extraction and assembly existed, but nothing turned the book's main-content statements into something an agent could actually cite mid-proof, and nothing searched it. Two new scripts. textbook/build_database.py takes the main-content statements and emits both halves of a usable premise library: - an importable Lean file where every entry is a named declaration, so a proof can cite goodman_1_6_20 by name; - a JSONL search index carrying each entry's name, statement, mentioned identifiers and status. An entry is emitted as an `axiom` (asserted from the textbook) until a real lean-bench run proves it; passing --results upgrades those entries to real theorems carrying the exact verified proof. This mirrors how the book itself works -- its exercises are posed assuming its body results, the same way using Mathlib does not mean re-proving Mathlib. The cost is explicit: anything proven through an axiom inherits it, so a garbled extraction asserted as an axiom makes downstream proofs worthless. Guards: --require-checked refuses entries that have not passed check_statements.py (all 21 section 1.6 entries have, and are now marked accordingly), and the verifier's existing #print axioms check surfaces what a finished proof rests on. textbook/search_database.py is the retrieval step the proof loop calls at each line: given the current goal, it ranks premises by identifier and symbol overlap, weighting rare identifiers far above common ones (IsGCDOfList matching means much more than Int matching), preferring a proven premise over an asserted one, and formats the top few for the prompt. Verified against three real goals: the exactly-applicable premise tops each ranking by a wide margin (11.4 vs 1.6 for the gcd goal, 5.1 vs 3.7 for the prime-divides-product goal). It is a ranking heuristic, not a decision procedure -- only Lean can say whether a premise applies. Replacing score_premise() is the whole job of swapping in embeddings. Also: renamed library_candidates/ to database/ to match what it is; generated lib/Goodman/Chapter01Section06.lean (21 premises, 0 proven, 21 asserted) and database/goodman-1.6.index.jsonl; noted a latent bug in the older build_library.py, which attaches each entry's non-import prelude per-theorem and so would emit a shared `def` twice in one file -- build_database.py hoists and dedupes, and has a test pinning that. 13 new tests, all green (33 total, the one failure remains the pre-existing unrelated lean_agent import error in test_dataset.py). Still not done, and worth being clear about: nothing has been proven yet. The bench targets have not been elaboration-checked, no lean-bench run has happened over either file, and the goal-directed loop that would consume this retrieval does not exist -- lean-bench's solve() is still single-shot. Separately, lean-bench cannot currently run at all in this checkout, because lean_bench/cli.py imports lean_agent at module load and lean-agent is not installed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x |
||
|
claude-bot
|
04d827b3b3 |
Split pipeline into real bench (exercises) vs library candidates (body)
Restructures the textbook pipeline around the distinction the body-derived dataset had been blurring: a bench should be actual unsolved problems, and a reusable premise library should only ever contain statements that have really been proven, the way Mathlib itself works. - library_candidates/goodman-1.6.jsonl (renamed from datasets/goodman-1.6.jsonl): the book's own Theorem/Proposition/Lemma/Corollary claims from the main text. Not bench targets -- premises waiting to be proven once via a real lean-bench run, then assembled into the reusable library by build_library.py (which already only keeps success:true entries with their checked proof, unchanged by this commit). - datasets/goodman-1.6-exercises.jsonl (new): the actual bench, sourced from the book's Exercises 1.6 block via two new scripts, textbook/extract_exercises.py (slices the Exercises block, lists its numbered items, drops a known false-positive class where a wrapped cross-reference like "...Proposition\n1.6.19." looks like a real exercise heading) and textbook/formalize_exercises.py (LLM step, given both the exercises and body text together since many exercises say "Prove Proposition N.N.N" instead of restating the claim). - Both formalize.py and formalize_exercises.py's validate() now enforce that every checklist item is covered by at least one output entry (included or explicitly excluded with a reason); formalize_exercises.py did not have this yet and a first real run silently dropped 7 of 15 checklist items with no trace before this was added. - All 12 model-produced exercise entries needed real fixes before commit, the same review pass applied to the body pilot: 3 exercises literally ask to prove an already-stated body claim (Exercise 1.6.1 -> Proposition 1.6.7, 1.6.2 -> Proposition 1.6.9, 1.6.7 -> Proposition 1.6.18, 1.6.9 -> Corollary 1.6.20, 1.6.15 -> Lemma 1.6.26) and now reuse that claim's own already-checked statement verbatim instead of an independently re-derived (and in one case strictly weaker) restatement; two entries (1.6.2, 1.6.11) had literal math prose like "I(m, n)" or "gcd(n1,...,nk)" leaked into the statement field, which is not valid Lean; several had untyped/ambiguous gcd or Prime usage fixed the same way the earlier body fixes were (Nat.Prime, IsCoprime, explicit casts, IsGreatest instead of max on a Set); and Exercise 1.6.12 part (a) turned out to be a discussion question whose unconditional-identity reading is actually false (6, 10, 15 are jointly coprime but not pairwise coprime), so it was moved to the excluded file with a reason instead of committed as a false theorem. - Definitions can now be real, compiled Lean, not only JSON metadata: Definition 1.6.22 (gcd of a list of integers, previously a null placeholder since no single Mathlib constant matches it) now has a real hand-written `def IsGCDOfList`, and textbook/build_definitions.py compiles any manifest's non-null lean_code entries into lib/Definitions.lean. goodman-1-6-24 (library) and the new goodman-1-6-ex-11-b/c, goodman-1-6-ex-12 (bench) all now use IsGCDOfList instead of repeating an inline existential. - textbook/README.md rewritten around the two-track design and records how this is meant to connect to lean-agent's eventual goal-directed proof search against the growing library (not yet built). - New tests for extract_exercises.py and build_definitions.py; full local suite still green (18/19, the one failure is the pre-existing, unrelated lean_agent import error in test_dataset.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x |
||
|
claude-bot
|
5aa048593c |
Note goodman-1.6 dataset now fully elaborates (test push)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x |
||
|
claude-bot
|
79c728567a |
Fix 10 failing goodman-1.6 statements based on Lean compiler diagnostics
All 21 statements in datasets/goodman-1.6.jsonl now elaborate cleanly
(verified locally via check_statements.py, 11/21 passing before this
change). Fixes applied, one per failing id:
- goodman-1-6-7: single-binder existential-unique over a pair (qr : ZxZ)
instead of the invalid two-binder 'exists unique' form.
- goodman-1-6-10: rewrote set-builder notation to the explicit
{v | exists a b, v = ...} form.
- goodman-1-6-11: replaced 'let d := ... in ...' with the required
semicolon form 'let d := ...; ...', and added the missing Nat-to-Int
casts on d.
- goodman-1-6-17: added explicit Nat-to-Int coercions on the three
divisibility relations.
- goodman-1-6-18: changed p from Z to N (Nat.Prime requires N), added
casts, and replaced the nonexistent Int.Coprime with IsCoprime.
- goodman-1-6-19: changed p from Z to N and added the matching casts.
- goodman-1-6-20: changed p from Z to N, added casts, and fixed the
quantifier structure so the prime-divides-product hypothesis is not
nested inside the per-element forall.
- goodman-1-6-23: replaced the Fin n literal comparison 'j = 0' (which
needs a NeZero instance) with 'j.val = 0'.
- goodman-1-6-24: replaced the invalid bare '(*)' operator section with
'(fun x y => x * y)'-equivalent '(. * .)', and replaced the
nonexistent GCD.gcd with an explicit existential statement of the gcd
property (divides every element; every common divisor divides it).
- goodman-1-6-26: replaced the nonexistent Int.Coprime with IsCoprime.
Also corrected the three parallel hallucinated Mathlib refs in
textbook/goodman-1.6.definitions.json (Int.Coprime, GCD.gcd,
Int.PairwiseCoprime -> IsCoprime, null with notes, and
List.Pairwise IsCoprime respectively), since the Lean compiler confirms
none of the original names exist.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
|
||
|
claude-bot
|
f98cb41f72 |
Run textbook pipeline pilot on section 1.6 (Divisibility in the Integers)
Generated with formalize.py against Qwen (qwen-max via DashScope), then hand-fixed before pushing, since these classes of bug pass elaboration but are still wrong and check_statements.py can't catch them without running Lean: - goodman-1-6-2 split into 4 separate theorems (1-6-2-a..d): the model's single merged statement had an operator-precedence bug (missing parens around an implicit right-associated AND after OR) that silently scoped 3 of the book's 4 independent claims inside the wrong disjunct. - goodman-1-6-1: dropped two vacuous "x + y : Nat is a Nat" clauses that confused Lean's type membership with set membership (element-of Nat on already-Nat-typed variables) and are automatically true by typing anyway. - goodman-1-6-9: swapped Mathlib's pointwise Set multiplication (x * I, which needs 'open scoped Pointwise' and is easy to get subtly wrong) for the equivalent elementwise statement. - goodman-1-6-13: replaced a misuse of min (a 2-argument function) applied to a Set with Mathlib's IsLeast predicate, matching what "least element" means in Int.gcd's characterization. - goodman-1-6-23: fixed Matrix index types (Fin n, not a bare Nat) and replaced an illegal literal '...' row with Matrix.vecMul; verified against the book's raw text that the n-by-n unimodular-matrix formulation is faithful, not a model invention. textbook/goodman-1.6.raw.jsonl keeps the unedited model output for reference. The remaining 13 statements are unedited and still need a real Lean elaboration pass (check_statements.py) plus a provability run (lean-bench) on a Mathlib-backed project -- neither ran here since this sandbox has no Lean. |
||
|
claude-bot
|
caedf2e6f4 |
Add textbook formalization pipeline (extract -> LLM formalize -> elaboration check -> library assembly)
- extract_section.py: slice a book section out of a pdftotext -layout dump and list its labeled Theorem/Proposition/Lemma/Corollary/Definition items. - formalize.py: LLM-formalize a section into Lean 4 candidate statements (preferring Mathlib vocabulary) plus a definitions manifest. - check_statements.py: elaboration-only sanity check against a real Mathlib-backed Lean project (must run outside the Cowork sandbox, which has no Lean toolchain and can't run macOS binaries anyway). - build_library.py: assemble solved lean-bench results into a reusable .lean file, reusing each winning proof verbatim. - llm.py: standalone OpenAI-compatible chat client + .env loader, kept independent of lean_agent so this pipeline works without it installed. - Source PDFs and raw/layout text dumps are gitignored (copyrighted material); only code and the final short Lean statement datasets are tracked. |
||
 codex-bot
|
99638be637 | Implement provider-neutral Lean reasoning and evaluation foundation |