Drop the generated search index; derive it from the source instead
The index file was 12KB of which ~96% was a byte-for-byte copy of
database/goodman-1.6.jsonl. Its only original content was three derived
fields: `name` (literally id.replace('-','_')), `status` (a two-line
computation over proof_checked), and `identifiers` (a regex over the
statement). It was a cache for work that costs microseconds, and it
created a real hazard: nothing checked that it matched its source, so
editing a statement and forgetting to regenerate would leave
search_database.py serving a premise whose text no longer matched the
generated .lean file.
- search_database.py now reads database/goodman-1.6.jsonl directly via a
new load_database(), deriving name/identifiers/status on the fly and
importing those functions from build_database.py so there is one
definition of each rather than two that can drift.
- build_database.py drops --out-index and index generation, and its
proof/status derivation is factored into resolve_proof() shared with
retrieval.
- database/goodman-1.6.index.jsonl deleted.
Also fixes a retrieval bug the removal surfaced: the tokenizer only
matched ASCII identifiers, so a goal that is pure algebra
("a * (b + c) = a * b + a * c") tokenized to nothing and retrieved
nothing at all -- it now keeps type and operator symbols, and drops
single-letter variable names, which say nothing when matched. That goal
now correctly finds Proposition 1.6.1, and the identifier-rich goals
still rank their exact premise first by a wide margin (9.3 vs 4.4).
Tests restructured accordingly: the proof gate is now pinned directly on
resolve_proof in both directions, plus new coverage for load_database
deriving status from proof_checked and for the symbol tokenization.
49 tests, 48 green (the one failure remains the pre-existing unrelated
lean_agent import error in test_dataset.py).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
This commit is contained in:
+12
-7
@@ -6,8 +6,8 @@ distinct things `lean-bench`/`lean-agent` can use, not one:
|
||||
- **The bench** (`datasets/goodman-1.6-exercises.jsonl`): real problems the reader
|
||||
is meant to solve, sourced from the book's own **Exercises** at the end of each
|
||||
section. These are the actual targets `lean-bench` scores an agent against.
|
||||
- **The database** (`database/goodman-1.6.jsonl` -> `lib/Goodman/Chapter01Section06.lean`
|
||||
+ `database/goodman-1.6.index.jsonl`): the book's stated Theorem/Proposition/
|
||||
- **The database** (`database/goodman-1.6.jsonl` -> `lib/Goodman/Chapter01Section06.lean`):
|
||||
the book's stated Theorem/Proposition/
|
||||
Lemma/Corollary claims from the main text, turned by `build_database.py` into real
|
||||
Lean declarations an agent can **cite by name** while proving a bench target --
|
||||
the "third party math library" role Mathlib plays, except scoped to this book.
|
||||
@@ -25,7 +25,13 @@ distinct things `lean-bench`/`lean-agent` can use, not one:
|
||||
axiom here still needs a human to have read it against the book.
|
||||
- **Retrieval** (`search_database.py`): given the current goal, ranks database
|
||||
premises by identifier/symbol overlap with rare identifiers weighted higher, and
|
||||
formats the top few for the LLM prompt. This is the step the proof loop calls at
|
||||
formats the top few for the LLM prompt. It reads the database JSONL directly and
|
||||
derives each premise's declaration name, identifiers and proof status itself
|
||||
(sharing those functions with `build_database.py`), so the data exists in exactly
|
||||
one place. There is deliberately no generated index: at this scale deriving them
|
||||
costs microseconds, and a second copy that nothing checks for freshness could
|
||||
silently serve statements that no longer match the `.lean` file. An index earns
|
||||
its place once it holds something actually expensive, i.e. embeddings. This is the step the proof loop calls at
|
||||
each line -- it narrows ~20 premises to the few worth showing, and cannot tell you
|
||||
whether one applies; only Lean can. Swapping in embeddings later means replacing
|
||||
`score_premise()` and nothing else.
|
||||
@@ -195,17 +201,16 @@ the next build.
|
||||
```sh
|
||||
python3 textbook/build_database.py database/goodman-1.6.jsonl \
|
||||
--out-lean lib/Goodman/Chapter01Section06.lean \
|
||||
--out-index database/goodman-1.6.index.jsonl \
|
||||
--require-checked
|
||||
```
|
||||
|
||||
Add `--results results/goodman-1.6-library-001/results.jsonl` once a `lean-bench`
|
||||
run over `database/goodman-1.6.jsonl` has proven some of them, to upgrade those
|
||||
entries from `axiom` to real theorems. Then query it with the current goal:
|
||||
entries from `axiom` to real theorems. Query it with the current goal:
|
||||
|
||||
```sh
|
||||
python3 textbook/search_database.py database/goodman-1.6.index.jsonl \
|
||||
"\u22a2 Nat.Prime p \u2192 (p:\u2124) \u2223 List.prod as \u2192 \u2203 a \u2208 as, (p:\u2124) \u2223 a"
|
||||
python3 textbook/search_database.py database/goodman-1.6.jsonl \
|
||||
"⊢ Nat.Prime p → (p:ℤ) ∣ List.prod as → ∃ a ∈ as, (p:ℤ) ∣ a"
|
||||
```
|
||||
|
||||
(`build_library.py` predates this and remains for the narrower job of assembling
|
||||
|
||||
Reference in New Issue
Block a user