Generate the citable premise database, and goal-driven retrieval over it

The database was the missing half: extraction and assembly existed, but
nothing turned the book's main-content statements into something an agent
could actually cite mid-proof, and nothing searched it. Two new scripts.

textbook/build_database.py takes the main-content statements and emits
both halves of a usable premise library:
- an importable Lean file where every entry is a named declaration, so a
  proof can cite goodman_1_6_20 by name;
- a JSONL search index carrying each entry's name, statement, mentioned
  identifiers and status.

An entry is emitted as an `axiom` (asserted from the textbook) until a
real lean-bench run proves it; passing --results upgrades those entries
to real theorems carrying the exact verified proof. This mirrors how the
book itself works -- its exercises are posed assuming its body results,
the same way using Mathlib does not mean re-proving Mathlib. The cost is
explicit: anything proven through an axiom inherits it, so a garbled
extraction asserted as an axiom makes downstream proofs worthless.
Guards: --require-checked refuses entries that have not passed
check_statements.py (all 21 section 1.6 entries have, and are now marked
accordingly), and the verifier's existing #print axioms check surfaces
what a finished proof rests on.

textbook/search_database.py is the retrieval step the proof loop calls at
each line: given the current goal, it ranks premises by identifier and
symbol overlap, weighting rare identifiers far above common ones
(IsGCDOfList matching means much more than Int matching), preferring a
proven premise over an asserted one, and formats the top few for the
prompt. Verified against three real goals: the exactly-applicable premise
tops each ranking by a wide margin (11.4 vs 1.6 for the gcd goal, 5.1 vs
3.7 for the prime-divides-product goal). It is a ranking heuristic, not a
decision procedure -- only Lean can say whether a premise applies.
Replacing score_premise() is the whole job of swapping in embeddings.

Also: renamed library_candidates/ to database/ to match what it is;
generated lib/Goodman/Chapter01Section06.lean (21 premises, 0 proven, 21
asserted) and database/goodman-1.6.index.jsonl; noted a latent bug in the
older build_library.py, which attaches each entry's non-import prelude
per-theorem and so would emit a shared `def` twice in one file --
build_database.py hoists and dedupes, and has a test pinning that.
13 new tests, all green (33 total, the one failure remains the
pre-existing unrelated lean_agent import error in test_dataset.py).

Still not done, and worth being clear about: nothing has been proven yet.
The bench targets have not been elaboration-checked, no lean-bench run
has happened over either file, and the goal-directed loop that would
consume this retrieval does not exist -- lean-bench's solve() is still
single-shot. Separately, lean-bench cannot currently run at all in this
checkout, because lean_bench/cli.py imports lean_agent at module load and
lean-agent is not installed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x

tests/test_search_database.py

@@ -0,0 +1,58 @@
+import unittest
+
+from textbook.search_database import format_for_prompt, search, tokenize
+
+INDEX = [
+    {"name": "goodman_1_6_20", "id": "goodman-1-6-20", "source_label": "Corollary 1.6.20",
+     "kind": "corollary", "status": "asserted",
+     "statement": "\u2200 (p : \u2115) (as : List \u2124), Nat.Prime p \u2192 (p:\u2124) \u2223 List.prod as \u2192 \u2203 a \u2208 as, (p:\u2124) \u2223 a",
+     "natural_language": "", "identifiers": ["Nat.Prime", "List.prod"], "requires": ["Nat.Prime"]},
+    {"name": "goodman_1_6_24", "id": "goodman-1-6-24", "source_label": "Proposition 1.6.24",
+     "kind": "proposition", "status": "asserted",
+     "statement": "\u2200 (as : List \u2124), \u2203 d, IsGCDOfList d as",
+     "natural_language": "", "identifiers": ["IsGCDOfList"], "requires": []},
+    {"name": "goodman_1_6_1", "id": "goodman-1-6-1", "source_label": "Proposition 1.6.1",
+     "kind": "proposition", "status": "asserted",
+     "statement": "\u2200 (a : \u2124), a + 0 = a", "natural_language": "",
+     "identifiers": [], "requires": []},
+]
+
+
+class SearchDatabaseTests(unittest.TestCase):
+    def test_tokenize_keeps_qualified_names_and_drops_stopwords(self):
+        tokens = tokenize('let x := Nat.Prime p')
+        self.assertIn('Nat.Prime', tokens)
+        self.assertNotIn('let', tokens)
+
+    def test_exact_premise_ranks_first(self):
+        hits = search(INDEX, '\u22a2 IsGCDOfList d as', limit=3)
+        self.assertEqual(hits[0]['name'], 'goodman_1_6_24')
+
+    def test_rare_identifier_outweighs_common_one(self):
+        # Both premises mention List; only one mentions Nat.Prime, so the goal about
+        # primes must prefer it rather than tying on the common token.
+        hits = search(INDEX, '\u22a2 Nat.Prime p \u2192 (p:\u2124) \u2223 List.prod as', limit=3)
+        self.assertEqual(hits[0]['name'], 'goodman_1_6_20')
+
+    def test_irrelevant_premises_are_dropped_not_ranked_last(self):
+        hits = search(INDEX, '\u22a2 IsGCDOfList d as', limit=5)
+        self.assertNotIn('goodman_1_6_1', [h['name'] for h in hits])
+
+    def test_proven_premise_is_preferred_over_an_asserted_one(self):
+        index = [dict(INDEX[1]), dict(INDEX[1], name='proved_twin', id='proved-twin',
+                                       status='proved', source_label='Proposition 1.6.24b')]
+        hits = search(index, '\u22a2 IsGCDOfList d as', limit=2)
+        self.assertEqual(hits[0]['name'], 'proved_twin')
+
+    def test_limit_is_respected_and_empty_result_is_explained(self):
+        self.assertEqual(len(search(INDEX, '\u22a2 List.prod as', limit=1)), 1)
+        self.assertIn('no database premise', format_for_prompt([]))
+
+    def test_format_for_prompt_marks_axioms_distinctly(self):
+        text = format_for_prompt(search(INDEX, '\u22a2 IsGCDOfList d as', limit=1))
+        self.assertIn('asserted from the book (axiom)', text)
+        self.assertIn('goodman_1_6_24', text)
+
+
+if __name__ == '__main__':
+    unittest.main()