The index file was 12KB of which ~96% was a byte-for-byte copy of
database/goodman-1.6.jsonl. Its only original content was three derived
fields: `name` (literally id.replace('-','_')), `status` (a two-line
computation over proof_checked), and `identifiers` (a regex over the
statement). It was a cache for work that costs microseconds, and it
created a real hazard: nothing checked that it matched its source, so
editing a statement and forgetting to regenerate would leave
search_database.py serving a premise whose text no longer matched the
generated .lean file.
- search_database.py now reads database/goodman-1.6.jsonl directly via a
new load_database(), deriving name/identifiers/status on the fly and
importing those functions from build_database.py so there is one
definition of each rather than two that can drift.
- build_database.py drops --out-index and index generation, and its
proof/status derivation is factored into resolve_proof() shared with
retrieval.
- database/goodman-1.6.index.jsonl deleted.
Also fixes a retrieval bug the removal surfaced: the tokenizer only
matched ASCII identifiers, so a goal that is pure algebra
("a * (b + c) = a * b + a * c") tokenized to nothing and retrieved
nothing at all -- it now keeps type and operator symbols, and drops
single-letter variable names, which say nothing when matched. That goal
now correctly finds Proposition 1.6.1, and the identifier-rich goals
still rank their exact premise first by a wide margin (9.3 vs 4.4).
Tests restructured accordingly: the proof gate is now pinned directly on
resolve_proof in both directions, plus new coverage for load_database
deriving status from proof_checked and for the symbol tokenization.
49 tests, 48 green (the one failure remains the pre-existing unrelated
lean_agent import error in test_dataset.py).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
The database was the missing half: extraction and assembly existed, but
nothing turned the book's main-content statements into something an agent
could actually cite mid-proof, and nothing searched it. Two new scripts.
textbook/build_database.py takes the main-content statements and emits
both halves of a usable premise library:
- an importable Lean file where every entry is a named declaration, so a
proof can cite goodman_1_6_20 by name;
- a JSONL search index carrying each entry's name, statement, mentioned
identifiers and status.
An entry is emitted as an `axiom` (asserted from the textbook) until a
real lean-bench run proves it; passing --results upgrades those entries
to real theorems carrying the exact verified proof. This mirrors how the
book itself works -- its exercises are posed assuming its body results,
the same way using Mathlib does not mean re-proving Mathlib. The cost is
explicit: anything proven through an axiom inherits it, so a garbled
extraction asserted as an axiom makes downstream proofs worthless.
Guards: --require-checked refuses entries that have not passed
check_statements.py (all 21 section 1.6 entries have, and are now marked
accordingly), and the verifier's existing #print axioms check surfaces
what a finished proof rests on.
textbook/search_database.py is the retrieval step the proof loop calls at
each line: given the current goal, it ranks premises by identifier and
symbol overlap, weighting rare identifiers far above common ones
(IsGCDOfList matching means much more than Int matching), preferring a
proven premise over an asserted one, and formats the top few for the
prompt. Verified against three real goals: the exactly-applicable premise
tops each ranking by a wide margin (11.4 vs 1.6 for the gcd goal, 5.1 vs
3.7 for the prime-divides-product goal). It is a ranking heuristic, not a
decision procedure -- only Lean can say whether a premise applies.
Replacing score_premise() is the whole job of swapping in embeddings.
Also: renamed library_candidates/ to database/ to match what it is;
generated lib/Goodman/Chapter01Section06.lean (21 premises, 0 proven, 21
asserted) and database/goodman-1.6.index.jsonl; noted a latent bug in the
older build_library.py, which attaches each entry's non-import prelude
per-theorem and so would emit a shared `def` twice in one file --
build_database.py hoists and dedupes, and has a test pinning that.
13 new tests, all green (33 total, the one failure remains the
pre-existing unrelated lean_agent import error in test_dataset.py).
Still not done, and worth being clear about: nothing has been proven yet.
The bench targets have not been elaboration-checked, no lean-bench run
has happened over either file, and the goal-directed loop that would
consume this retrieval does not exist -- lean-bench's solve() is still
single-shot. Separately, lean-bench cannot currently run at all in this
checkout, because lean_bench/cli.py imports lean_agent at module load and
lean-agent is not installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
claude-bot