Generate the citable premise database, and goal-driven retrieval over it

The database was the missing half: extraction and assembly existed, but
nothing turned the book's main-content statements into something an agent
could actually cite mid-proof, and nothing searched it. Two new scripts.

textbook/build_database.py takes the main-content statements and emits
both halves of a usable premise library:
- an importable Lean file where every entry is a named declaration, so a
  proof can cite goodman_1_6_20 by name;
- a JSONL search index carrying each entry's name, statement, mentioned
  identifiers and status.

An entry is emitted as an `axiom` (asserted from the textbook) until a
real lean-bench run proves it; passing --results upgrades those entries
to real theorems carrying the exact verified proof. This mirrors how the
book itself works -- its exercises are posed assuming its body results,
the same way using Mathlib does not mean re-proving Mathlib. The cost is
explicit: anything proven through an axiom inherits it, so a garbled
extraction asserted as an axiom makes downstream proofs worthless.
Guards: --require-checked refuses entries that have not passed
check_statements.py (all 21 section 1.6 entries have, and are now marked
accordingly), and the verifier's existing #print axioms check surfaces
what a finished proof rests on.

textbook/search_database.py is the retrieval step the proof loop calls at
each line: given the current goal, it ranks premises by identifier and
symbol overlap, weighting rare identifiers far above common ones
(IsGCDOfList matching means much more than Int matching), preferring a
proven premise over an asserted one, and formats the top few for the
prompt. Verified against three real goals: the exactly-applicable premise
tops each ranking by a wide margin (11.4 vs 1.6 for the gcd goal, 5.1 vs
3.7 for the prime-divides-product goal). It is a ranking heuristic, not a
decision procedure -- only Lean can say whether a premise applies.
Replacing score_premise() is the whole job of swapping in embeddings.

Also: renamed library_candidates/ to database/ to match what it is;
generated lib/Goodman/Chapter01Section06.lean (21 premises, 0 proven, 21
asserted) and database/goodman-1.6.index.jsonl; noted a latent bug in the
older build_library.py, which attaches each entry's non-import prelude
per-theorem and so would emit a shared `def` twice in one file --
build_database.py hoists and dedupes, and has a test pinning that.
13 new tests, all green (33 total, the one failure remains the
pre-existing unrelated lean_agent import error in test_dataset.py).

Still not done, and worth being clear about: nothing has been proven yet.
The bench targets have not been elaboration-checked, no lean-bench run
has happened over either file, and the goal-directed loop that would
consume this retrieval does not exist -- lean-bench's solve() is still
single-shot. Separately, lean-bench cannot currently run at all in this
checkout, because lean_bench/cli.py imports lean_agent at module load and
lean-agent is not installed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x

textbook/search_database.py

@@ -0,0 +1,99 @@
+"""Rank premises from the database against the current proof goals.
+
+This is the retrieval step of the intended lean-agent loop: rather than asking the
+LLM to prove a target in one shot, the agent keeps a list of open goals (the proof
+state, as vscode-lean4 displays it), and at each step searches the premise database
+for entries that might apply to the goal in front of it. The ranked results are what
+gets shown to the LLM alongside the goal, so the next line of Lean it writes can
+cite a real premise by name instead of inventing a lemma.
+
+Scoring here is deliberately simple and dependency-free: overlap of identifiers and
+symbols between the goal and each premise, weighted so that a rare identifier
+matters more than a common one (`IsGCDOfList` appearing in both is a much stronger
+signal than `Int`), plus a small bonus for an already-proven premise over one merely
+asserted as an axiom from the book. It is a ranking heuristic, not a decision
+procedure: it narrows 20-odd premises to the few worth showing, and cannot tell you
+whether a premise actually applies -- only Lean can. Swapping in embeddings later
+means replacing score_premise() and nothing else.
+"""
+import argparse
+import json
+import math
+import re
+from collections import Counter
+from pathlib import Path
+
+TOKEN_RE = re.compile(r'[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)+|[A-Za-z_][A-Za-z0-9_]*|[∣∀∃↔→∧∨¬≤≥≠∈]')
+STOPWORDS = {'let', 'in', 'fun', 'if', 'then', 'else', 'true', 'false', 'Prop', 'Type', 'this'}
+
+
+def load_index(path):
+    return [json.loads(line) for line in path.read_text(encoding='utf-8').splitlines() if line.strip()]
+
+
+def tokenize(text):
+    return [t for t in TOKEN_RE.findall(text or '') if t not in STOPWORDS and len(t) > 1]
+
+
+def document_frequencies(index):
+    """How many premises each token appears in, for inverse-document-frequency weighting."""
+    frequencies = Counter()
+    for record in index:
+        for token in set(tokenize(record['statement']) + record.get('identifiers', [])
+                          + record.get('requires', [])):
+            frequencies[token] += 1
+    return frequencies
+
+
+def score_premise(goal_tokens, record, frequencies, total):
+    premise_tokens = set(tokenize(record['statement']) + record.get('identifiers', [])
+                          + record.get('requires', []))
+    score = 0.0
+    for token in set(goal_tokens) & premise_tokens:
+        # Rare tokens are far more informative than ones every premise mentions.
+        score += math.log(1 + total / (1 + frequencies.get(token, 0)))
+    if record.get('status') == 'proved':
+        score *= 1.1  # prefer a real theorem over an axiom asserted from the book
+    return score
+
+
+def search(index, goal, limit=5):
+    """Premises ranked against one goal, best first. Zero-score entries are dropped."""
+    frequencies = document_frequencies(index)
+    total = max(len(index), 1)
+    goal_tokens = tokenize(goal)
+    scored = []
+    for record in index:
+        score = score_premise(goal_tokens, record, frequencies, total)
+        if score > 0:
+            scored.append((score, record))
+    scored.sort(key=lambda pair: (-pair[0], pair[1]['name']))
+    return [{**record, "score": round(score, 3)} for score, record in scored[:limit]]
+
+
+def format_for_prompt(hits):
+    """The premise list as it would be shown to the LLM next to the current goal."""
+    if not hits:
+        return '(no database premise looks relevant to this goal)'
+    lines = []
+    for hit in hits:
+        status = 'proven' if hit['status'] == 'proved' else 'asserted from the book (axiom)'
+        lines.append(f"{hit['name']}  [{hit['source_label']}, {status}, score {hit['score']}]\n"
+                     f"  {hit['statement']}")
+    return '\n'.join(lines)
+
+
+def main():
+    parser = argparse.ArgumentParser(description='Rank database premises against a proof goal')
+    parser.add_argument('index', type=Path, help='search index from build_database.py')
+    parser.add_argument('goal', help='the current goal, as Lean prints it')
+    parser.add_argument('--limit', type=int, default=5)
+    parser.add_argument('--json', action='store_true', help='emit raw JSON instead of prompt text')
+    args = parser.parse_args()
+
+    hits = search(load_index(args.index), args.goal, args.limit)
+    print(json.dumps(hits, ensure_ascii=False, indent=2) if args.json else format_for_prompt(hits))
+
+
+if __name__ == '__main__':
+    main()