fbcb0c3383e98a3204d457cd2ce8d3fae4e92c11
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 claude-bot
|
fbcb0c3383 |
Record and gate proofs, so "proven" means Lean verified it
The index had no proof field and every entry was an axiom, so the database asserted its premises rather than proving them. Adds the proof half, with the verification gate that makes the status trustworthy. - Database records now carry `proof` and `proof_checked`, and the search index exposes the proof, with status one of proved / proof_pending / asserted. build_database.py emits a real `theorem` only when a proof has been verified (by check_proofs.py, or by a lean-bench run via --results); a recorded but unverified proof stays an `axiom` and is reported as proof_pending, so the library never claims more than Lean has confirmed. A test pins that gating in both directions. - textbook/check_proofs.py verifies proofs by compiling `theorem ... := by <proof>`. Stricter than check_statements.py in three ways: it rejects sorry/admit/native_decide outright, and it reads `#print axioms` to reject any proof depending on more than Lean's own three axioms (propext, Classical.choice, Quot.sound) -- which in particular catches a proof leaning on one of this database's own book-asserted axioms, since that is not an independent proof. With --update it writes proof_checked back, promoting entries on the next build. - Ten candidate proofs recorded, all currently proof_checked=false pending a Lean run: goodman-1-6-1 (the eight algebraic identities are `ring`, plus a short argument for the nonzero conjunct), the four parts of Proposition 1.6.2 (Int.mul_eq_one_iff_eq_one_or_neg_one, associated_of_dvd_dvd with Int.associated_iff, dvd_trans, dvd_add with mul_left), 1-6-4 (Nat.exists_prime_and_dvd), 1-6-15 (Int.gcd_eq_one_iff_coprime, where IsCoprime unfolds to exactly the target existential), 1-6-17 (IsCoprime.mul_dvd via Nat.isCoprime_iff_coprime), 1-6-19 (Int.Prime.dvd_mul') and 1-6-20 (Prime.dvd_prod_iff). These are citations of existing Mathlib results, which is the realistic way to populate a textbook premise library -- the remaining 11 (1-6-6, -7, -9, -10, -11, -13, -18, -21, -23, -24, -26) are the ones needing actual work and carry no proof yet. Written without Lean available here, so expect a real failure rate on first compile -- run check_proofs.py --update and send back the FAILED ids with diagnostics, same loop as the statement fixes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x |
||
|
claude-bot
|
69c5b788b9 |
Generate the citable premise database, and goal-driven retrieval over it
The database was the missing half: extraction and assembly existed, but nothing turned the book's main-content statements into something an agent could actually cite mid-proof, and nothing searched it. Two new scripts. textbook/build_database.py takes the main-content statements and emits both halves of a usable premise library: - an importable Lean file where every entry is a named declaration, so a proof can cite goodman_1_6_20 by name; - a JSONL search index carrying each entry's name, statement, mentioned identifiers and status. An entry is emitted as an `axiom` (asserted from the textbook) until a real lean-bench run proves it; passing --results upgrades those entries to real theorems carrying the exact verified proof. This mirrors how the book itself works -- its exercises are posed assuming its body results, the same way using Mathlib does not mean re-proving Mathlib. The cost is explicit: anything proven through an axiom inherits it, so a garbled extraction asserted as an axiom makes downstream proofs worthless. Guards: --require-checked refuses entries that have not passed check_statements.py (all 21 section 1.6 entries have, and are now marked accordingly), and the verifier's existing #print axioms check surfaces what a finished proof rests on. textbook/search_database.py is the retrieval step the proof loop calls at each line: given the current goal, it ranks premises by identifier and symbol overlap, weighting rare identifiers far above common ones (IsGCDOfList matching means much more than Int matching), preferring a proven premise over an asserted one, and formats the top few for the prompt. Verified against three real goals: the exactly-applicable premise tops each ranking by a wide margin (11.4 vs 1.6 for the gcd goal, 5.1 vs 3.7 for the prime-divides-product goal). It is a ranking heuristic, not a decision procedure -- only Lean can say whether a premise applies. Replacing score_premise() is the whole job of swapping in embeddings. Also: renamed library_candidates/ to database/ to match what it is; generated lib/Goodman/Chapter01Section06.lean (21 premises, 0 proven, 21 asserted) and database/goodman-1.6.index.jsonl; noted a latent bug in the older build_library.py, which attaches each entry's non-import prelude per-theorem and so would emit a shared `def` twice in one file -- build_database.py hoists and dedupes, and has a test pinning that. 13 new tests, all green (33 total, the one failure remains the pre-existing unrelated lean_agent import error in test_dataset.py). Still not done, and worth being clear about: nothing has been proven yet. The bench targets have not been elaboration-checked, no lean-bench run has happened over either file, and the goal-directed loop that would consume this retrieval does not exist -- lean-bench's solve() is still single-shot. Separately, lean-bench cannot currently run at all in this checkout, because lean_bench/cli.py imports lean_agent at module load and lean-agent is not installed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x |
||
|
claude-bot
|
04d827b3b3 |
Split pipeline into real bench (exercises) vs library candidates (body)
Restructures the textbook pipeline around the distinction the body-derived dataset had been blurring: a bench should be actual unsolved problems, and a reusable premise library should only ever contain statements that have really been proven, the way Mathlib itself works. - library_candidates/goodman-1.6.jsonl (renamed from datasets/goodman-1.6.jsonl): the book's own Theorem/Proposition/Lemma/Corollary claims from the main text. Not bench targets -- premises waiting to be proven once via a real lean-bench run, then assembled into the reusable library by build_library.py (which already only keeps success:true entries with their checked proof, unchanged by this commit). - datasets/goodman-1.6-exercises.jsonl (new): the actual bench, sourced from the book's Exercises 1.6 block via two new scripts, textbook/extract_exercises.py (slices the Exercises block, lists its numbered items, drops a known false-positive class where a wrapped cross-reference like "...Proposition\n1.6.19." looks like a real exercise heading) and textbook/formalize_exercises.py (LLM step, given both the exercises and body text together since many exercises say "Prove Proposition N.N.N" instead of restating the claim). - Both formalize.py and formalize_exercises.py's validate() now enforce that every checklist item is covered by at least one output entry (included or explicitly excluded with a reason); formalize_exercises.py did not have this yet and a first real run silently dropped 7 of 15 checklist items with no trace before this was added. - All 12 model-produced exercise entries needed real fixes before commit, the same review pass applied to the body pilot: 3 exercises literally ask to prove an already-stated body claim (Exercise 1.6.1 -> Proposition 1.6.7, 1.6.2 -> Proposition 1.6.9, 1.6.7 -> Proposition 1.6.18, 1.6.9 -> Corollary 1.6.20, 1.6.15 -> Lemma 1.6.26) and now reuse that claim's own already-checked statement verbatim instead of an independently re-derived (and in one case strictly weaker) restatement; two entries (1.6.2, 1.6.11) had literal math prose like "I(m, n)" or "gcd(n1,...,nk)" leaked into the statement field, which is not valid Lean; several had untyped/ambiguous gcd or Prime usage fixed the same way the earlier body fixes were (Nat.Prime, IsCoprime, explicit casts, IsGreatest instead of max on a Set); and Exercise 1.6.12 part (a) turned out to be a discussion question whose unconditional-identity reading is actually false (6, 10, 15 are jointly coprime but not pairwise coprime), so it was moved to the excluded file with a reason instead of committed as a false theorem. - Definitions can now be real, compiled Lean, not only JSON metadata: Definition 1.6.22 (gcd of a list of integers, previously a null placeholder since no single Mathlib constant matches it) now has a real hand-written `def IsGCDOfList`, and textbook/build_definitions.py compiles any manifest's non-null lean_code entries into lib/Definitions.lean. goodman-1-6-24 (library) and the new goodman-1-6-ex-11-b/c, goodman-1-6-ex-12 (bench) all now use IsGCDOfList instead of repeating an inline existential. - textbook/README.md rewritten around the two-track design and records how this is meant to connect to lean-agent's eventual goal-directed proof search against the growing library (not yet built). - New tests for extract_exercises.py and build_definitions.py; full local suite still green (18/19, the one failure is the pre-existing, unrelated lean_agent import error in test_dataset.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x |
||
|
claude-bot
|
caedf2e6f4 |
Add textbook formalization pipeline (extract -> LLM formalize -> elaboration check -> library assembly)
- extract_section.py: slice a book section out of a pdftotext -layout dump and list its labeled Theorem/Proposition/Lemma/Corollary/Definition items. - formalize.py: LLM-formalize a section into Lean 4 candidate statements (preferring Mathlib vocabulary) plus a definitions manifest. - check_statements.py: elaboration-only sanity check against a real Mathlib-backed Lean project (must run outside the Cowork sandbox, which has no Lean toolchain and can't run macOS binaries anyway). - build_library.py: assemble solved lean-bench results into a reusable .lean file, reusing each winning proof verbatim. - llm.py: standalone OpenAI-compatible chat client + .env loader, kept independent of lean_agent so this pipeline works without it installed. - Source PDFs and raw/layout text dumps are gitignored (copyrighted material); only code and the final short Lean statement datasets are tracked. |
||
 codex-bot
|
99638be637 | Implement provider-neutral Lean reasoning and evaluation foundation |