Generate the citable premise database, and goal-driven retrieval over it

The database was the missing half: extraction and assembly existed, but
nothing turned the book's main-content statements into something an agent
could actually cite mid-proof, and nothing searched it. Two new scripts.

textbook/build_database.py takes the main-content statements and emits
both halves of a usable premise library:
- an importable Lean file where every entry is a named declaration, so a
  proof can cite goodman_1_6_20 by name;
- a JSONL search index carrying each entry's name, statement, mentioned
  identifiers and status.

An entry is emitted as an `axiom` (asserted from the textbook) until a
real lean-bench run proves it; passing --results upgrades those entries
to real theorems carrying the exact verified proof. This mirrors how the
book itself works -- its exercises are posed assuming its body results,
the same way using Mathlib does not mean re-proving Mathlib. The cost is
explicit: anything proven through an axiom inherits it, so a garbled
extraction asserted as an axiom makes downstream proofs worthless.
Guards: --require-checked refuses entries that have not passed
check_statements.py (all 21 section 1.6 entries have, and are now marked
accordingly), and the verifier's existing #print axioms check surfaces
what a finished proof rests on.

textbook/search_database.py is the retrieval step the proof loop calls at
each line: given the current goal, it ranks premises by identifier and
symbol overlap, weighting rare identifiers far above common ones
(IsGCDOfList matching means much more than Int matching), preferring a
proven premise over an asserted one, and formats the top few for the
prompt. Verified against three real goals: the exactly-applicable premise
tops each ranking by a wide margin (11.4 vs 1.6 for the gcd goal, 5.1 vs
3.7 for the prime-divides-product goal). It is a ranking heuristic, not a
decision procedure -- only Lean can say whether a premise applies.
Replacing score_premise() is the whole job of swapping in embeddings.

Also: renamed library_candidates/ to database/ to match what it is;
generated lib/Goodman/Chapter01Section06.lean (21 premises, 0 proven, 21
asserted) and database/goodman-1.6.index.jsonl; noted a latent bug in the
older build_library.py, which attaches each entry's non-import prelude
per-theorem and so would emit a shared `def` twice in one file --
build_database.py hoists and dedupes, and has a test pinning that.
13 new tests, all green (33 total, the one failure remains the
pre-existing unrelated lean_agent import error in test_dataset.py).

Still not done, and worth being clear about: nothing has been proven yet.
The bench targets have not been elaboration-checked, no lean-bench run
has happened over either file, and the goal-directed loop that would
consume this retrieval does not exist -- lean-bench's solve() is still
single-shot. Separately, lean-bench cannot currently run at all in this
checkout, because lean_bench/cli.py imports lean_agent at module load and
lean-agent is not installed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x

textbook/README.md

@@ -6,23 +6,35 @@ distinct things `lean-bench`/`lean-agent` can use, not one:
 - **The bench** (`datasets/goodman-1.6-exercises.jsonl`): real problems the reader
   is meant to solve, sourced from the book's own **Exercises** at the end of each
   section. These are the actual targets `lean-bench` scores an agent against.
-- **The library candidates** (`library_candidates/goodman-1.6.jsonl`): the book's
-  stated Theorem/Proposition/Lemma/Corollary claims from the section's main text.
-  These are NOT bench targets -- they are premises waiting to be proven once and
-  then reused, the way Mathlib itself is a library of already-proven facts an agent
-  can cite while proving something new. An entry only belongs in the trustworthy,
-  reusable library after `lean-bench` has actually proven it for real (see step 5);
-  `build_library.py` already enforces this -- it only assembles entries whose
-  `results.jsonl` says `success: true`, with the exact proof that was checked, never
-  anything from this file directly.
-- **Definitions** (`textbook/*.definitions.json` + `lib/Definitions.lean`): a
-  concept a bench/library statement depends on. When Mathlib already has it
-  (`mathlib_ref` set), statements just use that name directly. When it doesn't
-  (`mathlib_ref: null`), the manifest also carries real, hand-written Lean code
-  (`lean_code`), compiled by `build_definitions.py` into an actual importable
-  `.lean` file -- Lean can only ever verify a proof about a concept that exists as
-  real Lean code somewhere, a JSON description of a definition is not enough for it
-  to typecheck anything that mentions it.
+- **The database** (`database/goodman-1.6.jsonl` -> `lib/Goodman/Chapter01Section06.lean`
+  + `database/goodman-1.6.index.jsonl`): the book's stated Theorem/Proposition/
+  Lemma/Corollary claims from the main text, turned by `build_database.py` into real
+  Lean declarations an agent can **cite by name** while proving a bench target --
+  the "third party math library" role Mathlib plays, except scoped to this book.
+  You do not re-prove Mathlib's theorems to use them, and the same applies here: the
+  book proves its own body results, and its exercises are posed assuming them. So an
+  entry is emitted as an `axiom` (asserted from the textbook) until a real
+  `lean-bench` run proves it, at which point `--results` upgrades it to a
+  `theorem` carrying the exact proof that was verified.
+
+  The cost of that shortcut is real and worth stating plainly: anything proven
+  *through* an axiom inherits it, so a garbled extraction asserted as an axiom can
+  make downstream "proofs" worthless. Two guards: `--require-checked` refuses any
+  entry that hasn't passed `check_statements.py`, and the verifier's
+  `#print axioms` check surfaces what a finished proof actually rests on. Every
+  axiom here still needs a human to have read it against the book.
+- **Retrieval** (`search_database.py`): given the current goal, ranks database
+  premises by identifier/symbol overlap with rare identifiers weighted higher, and
+  formats the top few for the LLM prompt. This is the step the proof loop calls at
+  each line -- it narrows ~20 premises to the few worth showing, and cannot tell you
+  whether one applies; only Lean can. Swapping in embeddings later means replacing
+  `score_premise()` and nothing else.
+- **Definitions** (`textbook/*.definitions.json` + `lib/Definitions.lean`): a concept
+  a statement depends on. When Mathlib has it (`mathlib_ref` set), statements use
+  that name directly. When it doesn't (`mathlib_ref: null`), the manifest carries
+  real hand-written Lean (`lean_code`), compiled by `build_definitions.py` into an
+  importable `.lean` file -- Lean can only verify a proof about a concept that
+  exists as real Lean code, a JSON description of a definition is not enough.
 
 This pipeline does not itself prove anything or run Lean; see "What this pipeline
 does not do" below and `lean-agent`'s own README for how proving actually happens
@@ -88,7 +100,7 @@ export DASHSCOPE_API_KEY=...   # or put it in lean-bench/.env (gitignored)
 python3 textbook/formalize.py \
   textbook/section_1_6.txt textbook/section_1_6.index.json \
   --model qwen-max --base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
-  --out-dataset library_candidates/goodman-1.6.jsonl \
+  --out-dataset database/goodman-1.6.jsonl \
   --out-definitions textbook/goodman-1.6.definitions.json \
   --out-excluded textbook/goodman-1.6.excluded.jsonl
 
@@ -121,7 +133,7 @@ mis-included as an unconditional theorem).
 ### 4. Check candidate statements type-check on your machine
 
 ```sh
-python3 textbook/check_statements.py library_candidates/goodman-1.6.jsonl \
+python3 textbook/check_statements.py database/goodman-1.6.jsonl \
   --lean-project /path/to/your/mathlib-project \
   --lean-command '["lake","env","lean"]' \
   --report textbook/goodman-1.6.elab-report.jsonl
@@ -157,19 +169,28 @@ task, matching how `check_statements.py` elaborates one task at a time), so
 `lib/Definitions.lean` is the single canonical copy and the inlined ones must stay
 in sync with it by construction (generated from the same manifest).
 
-### 7. Assemble proven library candidates into a reusable library file
+### 7. Generate the citable premise database
 
 ```sh
-python3 textbook/build_library.py library_candidates/goodman-1.6.jsonl \
-  results/goodman-1.6-001-library/results.jsonl --out lib/Chapter01Section06.lean
+python3 textbook/build_database.py database/goodman-1.6.jsonl \
+  --out-lean lib/Goodman/Chapter01Section06.lean \
+  --out-index database/goodman-1.6.index.jsonl \
+  --require-checked
 ```
 
-(This step needs its own `lean-bench` run over `library_candidates/`, proving those
-statements the same way step 5 proves the exercises -- not yet done for section
-1.6.) Only entries that were actually proven in that run go in, with the exact
-proof that was checked, verbatim. Later chapters `import` this file to build on
-already-proven results -- it's one flat `.lean` file, not a real Lake
-package/module boundary.
+Add `--results results/goodman-1.6-library-001/results.jsonl` once a `lean-bench`
+run over `database/goodman-1.6.jsonl` has proven some of them, to upgrade those
+entries from `axiom` to real theorems. Then query it with the current goal:
+
+```sh
+python3 textbook/search_database.py database/goodman-1.6.index.jsonl \
+  "\u22a2 Nat.Prime p \u2192 (p:\u2124) \u2223 List.prod as \u2192 \u2203 a \u2208 as, (p:\u2124) \u2223 a"
+```
+
+(`build_library.py` predates this and remains for the narrower job of assembling
+*only* proven entries with no axioms at all; note it attaches each entry's
+non-import prelude per-theorem, which would emit a shared `def` twice in one file --
+`build_database.py` hoists and dedupes them instead.)
 
 ## How this is meant to connect to lean-agent