Drop the generated search index; derive it from the source instead

The index file was 12KB of which ~96% was a byte-for-byte copy of
database/goodman-1.6.jsonl. Its only original content was three derived
fields: `name` (literally id.replace('-','_')), `status` (a two-line
computation over proof_checked), and `identifiers` (a regex over the
statement). It was a cache for work that costs microseconds, and it
created a real hazard: nothing checked that it matched its source, so
editing a statement and forgetting to regenerate would leave
search_database.py serving a premise whose text no longer matched the
generated .lean file.

- search_database.py now reads database/goodman-1.6.jsonl directly via a
  new load_database(), deriving name/identifiers/status on the fly and
  importing those functions from build_database.py so there is one
  definition of each rather than two that can drift.
- build_database.py drops --out-index and index generation, and its
  proof/status derivation is factored into resolve_proof() shared with
  retrieval.
- database/goodman-1.6.index.jsonl deleted.

Also fixes a retrieval bug the removal surfaced: the tokenizer only
matched ASCII identifiers, so a goal that is pure algebra
("a * (b + c) = a * b + a * c") tokenized to nothing and retrieved
nothing at all -- it now keeps type and operator symbols, and drops
single-letter variable names, which say nothing when matched. That goal
now correctly finds Proposition 1.6.1, and the identifier-rich goals
still rank their exact premise first by a wide margin (9.3 vs 4.4).

Tests restructured accordingly: the proof gate is now pinned directly on
resolve_proof in both directions, plus new coverage for load_database
deriving status from proof_checked and for the symbol tokenization.
49 tests, 48 green (the one failure remains the pre-existing unrelated
lean_agent import error in test_dataset.py).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
This commit is contained in:
2026-09-18 19:50:10 +00:00
parent fbcb0c3383
commit 9e0457a0e1
6 changed files with 230 additions and 128 deletions
+12 -7
View File
@@ -6,8 +6,8 @@ distinct things `lean-bench`/`lean-agent` can use, not one:
- **The bench** (`datasets/goodman-1.6-exercises.jsonl`): real problems the reader
is meant to solve, sourced from the book's own **Exercises** at the end of each
section. These are the actual targets `lean-bench` scores an agent against.
- **The database** (`database/goodman-1.6.jsonl` -> `lib/Goodman/Chapter01Section06.lean`
+ `database/goodman-1.6.index.jsonl`): the book's stated Theorem/Proposition/
- **The database** (`database/goodman-1.6.jsonl` -> `lib/Goodman/Chapter01Section06.lean`):
the book's stated Theorem/Proposition/
Lemma/Corollary claims from the main text, turned by `build_database.py` into real
Lean declarations an agent can **cite by name** while proving a bench target --
the "third party math library" role Mathlib plays, except scoped to this book.
@@ -25,7 +25,13 @@ distinct things `lean-bench`/`lean-agent` can use, not one:
axiom here still needs a human to have read it against the book.
- **Retrieval** (`search_database.py`): given the current goal, ranks database
premises by identifier/symbol overlap with rare identifiers weighted higher, and
formats the top few for the LLM prompt. This is the step the proof loop calls at
formats the top few for the LLM prompt. It reads the database JSONL directly and
derives each premise's declaration name, identifiers and proof status itself
(sharing those functions with `build_database.py`), so the data exists in exactly
one place. There is deliberately no generated index: at this scale deriving them
costs microseconds, and a second copy that nothing checks for freshness could
silently serve statements that no longer match the `.lean` file. An index earns
its place once it holds something actually expensive, i.e. embeddings. This is the step the proof loop calls at
each line -- it narrows ~20 premises to the few worth showing, and cannot tell you
whether one applies; only Lean can. Swapping in embeddings later means replacing
`score_premise()` and nothing else.
@@ -195,17 +201,16 @@ the next build.
```sh
python3 textbook/build_database.py database/goodman-1.6.jsonl \
--out-lean lib/Goodman/Chapter01Section06.lean \
--out-index database/goodman-1.6.index.jsonl \
--require-checked
```
Add `--results results/goodman-1.6-library-001/results.jsonl` once a `lean-bench`
run over `database/goodman-1.6.jsonl` has proven some of them, to upgrade those
entries from `axiom` to real theorems. Then query it with the current goal:
entries from `axiom` to real theorems. Query it with the current goal:
```sh
python3 textbook/search_database.py database/goodman-1.6.index.jsonl \
"\u22a2 Nat.Prime p \u2192 (p:\u2124) \u2223 List.prod as \u2192 \u2203 a \u2208 as, (p:\u2124) \u2223 a"
python3 textbook/search_database.py database/goodman-1.6.jsonl \
"⊢ Nat.Prime p → (p:ℤ) ∣ List.prod as → ∃ a ∈ as, (p:ℤ) ∣ a"
```
(`build_library.py` predates this and remains for the narrower job of assembling