Zayden-Jung/lean-bench
2
1
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity
9 Commits 8 Branches 0 Tags
9e0457a0e1554415263637b85e3ccd3e85ca4504
Commit Graph

6 Commits

Author SHA1 Message Date
claude-bot 9e0457a0e1 Drop the generated search index; derive it from the source instead 
The index file was 12KB of which ~96% was a byte-for-byte copy of
database/goodman-1.6.jsonl. Its only original content was three derived
fields: `name` (literally id.replace('-','_')), `status` (a two-line
computation over proof_checked), and `identifiers` (a regex over the
statement). It was a cache for work that costs microseconds, and it
created a real hazard: nothing checked that it matched its source, so
editing a statement and forgetting to regenerate would leave
search_database.py serving a premise whose text no longer matched the
generated .lean file.

- search_database.py now reads database/goodman-1.6.jsonl directly via a
  new load_database(), deriving name/identifiers/status on the fly and
  importing those functions from build_database.py so there is one
  definition of each rather than two that can drift.
- build_database.py drops --out-index and index generation, and its
  proof/status derivation is factored into resolve_proof() shared with
  retrieval.
- database/goodman-1.6.index.jsonl deleted.

Also fixes a retrieval bug the removal surfaced: the tokenizer only
matched ASCII identifiers, so a goal that is pure algebra
("a * (b + c) = a * b + a * c") tokenized to nothing and retrieved
nothing at all -- it now keeps type and operator symbols, and drops
single-letter variable names, which say nothing when matched. That goal
now correctly finds Proposition 1.6.1, and the identifier-rich goals
still rank their exact premise first by a wide margin (9.3 vs 4.4).

Tests restructured accordingly: the proof gate is now pinned directly on
resolve_proof in both directions, plus new coverage for load_database
deriving status from proof_checked and for the symbol tokenization.
49 tests, 48 green (the one failure remains the pre-existing unrelated
lean_agent import error in test_dataset.py).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
 2026-09-18 19:50:10 +00:00
claude-bot fbcb0c3383 Record and gate proofs, so "proven" means Lean verified it 
The index had no proof field and every entry was an axiom, so the
database asserted its premises rather than proving them. Adds the proof
half, with the verification gate that makes the status trustworthy.

- Database records now carry `proof` and `proof_checked`, and the search
  index exposes the proof, with status one of proved / proof_pending /
  asserted. build_database.py emits a real `theorem` only when a proof
  has been verified (by check_proofs.py, or by a lean-bench run via
  --results); a recorded but unverified proof stays an `axiom` and is
  reported as proof_pending, so the library never claims more than Lean
  has confirmed. A test pins that gating in both directions.

- textbook/check_proofs.py verifies proofs by compiling
  `theorem ... := by <proof>`. Stricter than check_statements.py in three
  ways: it rejects sorry/admit/native_decide outright, and it reads
  `#print axioms` to reject any proof depending on more than Lean's own
  three axioms (propext, Classical.choice, Quot.sound) -- which in
  particular catches a proof leaning on one of this database's own
  book-asserted axioms, since that is not an independent proof. With
  --update it writes proof_checked back, promoting entries on the next
  build.

- Ten candidate proofs recorded, all currently proof_checked=false
  pending a Lean run: goodman-1-6-1 (the eight algebraic identities are
  `ring`, plus a short argument for the nonzero conjunct), the four parts
  of Proposition 1.6.2 (Int.mul_eq_one_iff_eq_one_or_neg_one,
  associated_of_dvd_dvd with Int.associated_iff, dvd_trans, dvd_add with
  mul_left), 1-6-4 (Nat.exists_prime_and_dvd), 1-6-15
  (Int.gcd_eq_one_iff_coprime, where IsCoprime unfolds to exactly the
  target existential), 1-6-17 (IsCoprime.mul_dvd via
  Nat.isCoprime_iff_coprime), 1-6-19 (Int.Prime.dvd_mul') and 1-6-20
  (Prime.dvd_prod_iff). These are citations of existing Mathlib results,
  which is the realistic way to populate a textbook premise library --
  the remaining 11 (1-6-6, -7, -9, -10, -11, -13, -18, -21, -23, -24,
  -26) are the ones needing actual work and carry no proof yet.

Written without Lean available here, so expect a real failure rate on
first compile -- run check_proofs.py --update and send back the FAILED
ids with diagnostics, same loop as the statement fixes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
 2026-09-18 19:43:54 +00:00
claude-bot 69c5b788b9 Generate the citable premise database, and goal-driven retrieval over it 
The database was the missing half: extraction and assembly existed, but
nothing turned the book's main-content statements into something an agent
could actually cite mid-proof, and nothing searched it. Two new scripts.

textbook/build_database.py takes the main-content statements and emits
both halves of a usable premise library:
- an importable Lean file where every entry is a named declaration, so a
  proof can cite goodman_1_6_20 by name;
- a JSONL search index carrying each entry's name, statement, mentioned
  identifiers and status.

An entry is emitted as an `axiom` (asserted from the textbook) until a
real lean-bench run proves it; passing --results upgrades those entries
to real theorems carrying the exact verified proof. This mirrors how the
book itself works -- its exercises are posed assuming its body results,
the same way using Mathlib does not mean re-proving Mathlib. The cost is
explicit: anything proven through an axiom inherits it, so a garbled
extraction asserted as an axiom makes downstream proofs worthless.
Guards: --require-checked refuses entries that have not passed
check_statements.py (all 21 section 1.6 entries have, and are now marked
accordingly), and the verifier's existing #print axioms check surfaces
what a finished proof rests on.

textbook/search_database.py is the retrieval step the proof loop calls at
each line: given the current goal, it ranks premises by identifier and
symbol overlap, weighting rare identifiers far above common ones
(IsGCDOfList matching means much more than Int matching), preferring a
proven premise over an asserted one, and formats the top few for the
prompt. Verified against three real goals: the exactly-applicable premise
tops each ranking by a wide margin (11.4 vs 1.6 for the gcd goal, 5.1 vs
3.7 for the prime-divides-product goal). It is a ranking heuristic, not a
decision procedure -- only Lean can say whether a premise applies.
Replacing score_premise() is the whole job of swapping in embeddings.

Also: renamed library_candidates/ to database/ to match what it is;
generated lib/Goodman/Chapter01Section06.lean (21 premises, 0 proven, 21
asserted) and database/goodman-1.6.index.jsonl; noted a latent bug in the
older build_library.py, which attaches each entry's non-import prelude
per-theorem and so would emit a shared `def` twice in one file --
build_database.py hoists and dedupes, and has a test pinning that.
13 new tests, all green (33 total, the one failure remains the
pre-existing unrelated lean_agent import error in test_dataset.py).

Still not done, and worth being clear about: nothing has been proven yet.
The bench targets have not been elaboration-checked, no lean-bench run
has happened over either file, and the goal-directed loop that would
consume this retrieval does not exist -- lean-bench's solve() is still
single-shot. Separately, lean-bench cannot currently run at all in this
checkout, because lean_bench/cli.py imports lean_agent at module load and
lean-agent is not installed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
 2026-09-18 19:36:18 +00:00
claude-bot 04d827b3b3 Split pipeline into real bench (exercises) vs library candidates (body) 
Restructures the textbook pipeline around the distinction the body-derived
dataset had been blurring: a bench should be actual unsolved problems, and
a reusable premise library should only ever contain statements that have
really been proven, the way Mathlib itself works.

- library_candidates/goodman-1.6.jsonl (renamed from datasets/goodman-1.6.jsonl):
  the book's own Theorem/Proposition/Lemma/Corollary claims from the main
  text. Not bench targets -- premises waiting to be proven once via a real
  lean-bench run, then assembled into the reusable library by
  build_library.py (which already only keeps success:true entries with
  their checked proof, unchanged by this commit).
- datasets/goodman-1.6-exercises.jsonl (new): the actual bench, sourced
  from the book's Exercises 1.6 block via two new scripts,
  textbook/extract_exercises.py (slices the Exercises block, lists its
  numbered items, drops a known false-positive class where a wrapped
  cross-reference like "...Proposition\n1.6.19." looks like a real
  exercise heading) and textbook/formalize_exercises.py (LLM step, given
  both the exercises and body text together since many exercises say
  "Prove Proposition N.N.N" instead of restating the claim).
- Both formalize.py and formalize_exercises.py's validate() now enforce
  that every checklist item is covered by at least one output entry
  (included or explicitly excluded with a reason); formalize_exercises.py
  did not have this yet and a first real run silently dropped 7 of 15
  checklist items with no trace before this was added.
- All 12 model-produced exercise entries needed real fixes before commit,
  the same review pass applied to the body pilot: 3 exercises literally
  ask to prove an already-stated body claim (Exercise 1.6.1 -> Proposition
  1.6.7, 1.6.2 -> Proposition 1.6.9, 1.6.7 -> Proposition 1.6.18, 1.6.9 ->
  Corollary 1.6.20, 1.6.15 -> Lemma 1.6.26) and now reuse that claim's own
  already-checked statement verbatim instead of an independently
  re-derived (and in one case strictly weaker) restatement; two entries
  (1.6.2, 1.6.11) had literal math prose like "I(m, n)" or
  "gcd(n1,...,nk)" leaked into the statement field, which is not valid
  Lean; several had untyped/ambiguous gcd or Prime usage fixed the same
  way the earlier body fixes were (Nat.Prime, IsCoprime, explicit casts,
  IsGreatest instead of max on a Set); and Exercise 1.6.12 part (a) turned
  out to be a discussion question whose unconditional-identity reading is
  actually false (6, 10, 15 are jointly coprime but not pairwise coprime),
  so it was moved to the excluded file with a reason instead of committed
  as a false theorem.
- Definitions can now be real, compiled Lean, not only JSON metadata:
  Definition 1.6.22 (gcd of a list of integers, previously a null
  placeholder since no single Mathlib constant matches it) now has a real
  hand-written `def IsGCDOfList`, and textbook/build_definitions.py
  compiles any manifest's non-null lean_code entries into
  lib/Definitions.lean. goodman-1-6-24 (library) and the new
  goodman-1-6-ex-11-b/c, goodman-1-6-ex-12 (bench) all now use
  IsGCDOfList instead of repeating an inline existential.
- textbook/README.md rewritten around the two-track design and records
  how this is meant to connect to lean-agent's eventual goal-directed
  proof search against the growing library (not yet built).
- New tests for extract_exercises.py and build_definitions.py; full
  local suite still green (18/19, the one failure is the pre-existing,
  unrelated lean_agent import error in test_dataset.py).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
 2026-09-18 19:27:40 +00:00
claude-bot caedf2e6f4 Add textbook formalization pipeline (extract -> LLM formalize -> elaboration check -> library assembly) 
- extract_section.py: slice a book section out of a pdftotext -layout dump and
  list its labeled Theorem/Proposition/Lemma/Corollary/Definition items.
- formalize.py: LLM-formalize a section into Lean 4 candidate statements
  (preferring Mathlib vocabulary) plus a definitions manifest.
- check_statements.py: elaboration-only sanity check against a real
  Mathlib-backed Lean project (must run outside the Cowork sandbox, which has
  no Lean toolchain and can't run macOS binaries anyway).
- build_library.py: assemble solved lean-bench results into a reusable .lean
  file, reusing each winning proof verbatim.
- llm.py: standalone OpenAI-compatible chat client + .env loader, kept
  independent of lean_agent so this pipeline works without it installed.
- Source PDFs and raw/layout text dumps are gitignored (copyrighted material);
  only code and the final short Lean statement datasets are tracked.
 2026-09-17 17:04:19 +00:00
codex-bot 99638be637 Implement provider-neutral Lean reasoning and evaluation foundation 2026-09-15 19:15:05 +08:00