Drop the generated search index; derive it from the source instead

The index file was 12KB of which ~96% was a byte-for-byte copy of
database/goodman-1.6.jsonl. Its only original content was three derived
fields: `name` (literally id.replace('-','_')), `status` (a two-line
computation over proof_checked), and `identifiers` (a regex over the
statement). It was a cache for work that costs microseconds, and it
created a real hazard: nothing checked that it matched its source, so
editing a statement and forgetting to regenerate would leave
search_database.py serving a premise whose text no longer matched the
generated .lean file.

- search_database.py now reads database/goodman-1.6.jsonl directly via a
  new load_database(), deriving name/identifiers/status on the fly and
  importing those functions from build_database.py so there is one
  definition of each rather than two that can drift.
- build_database.py drops --out-index and index generation, and its
  proof/status derivation is factored into resolve_proof() shared with
  retrieval.
- database/goodman-1.6.index.jsonl deleted.

Also fixes a retrieval bug the removal surfaced: the tokenizer only
matched ASCII identifiers, so a goal that is pure algebra
("a * (b + c) = a * b + a * c") tokenized to nothing and retrieved
nothing at all -- it now keeps type and operator symbols, and drops
single-letter variable names, which say nothing when matched. That goal
now correctly finds Proposition 1.6.1, and the identifier-rich goals
still rank their exact premise first by a wide margin (9.3 vs 4.4).

Tests restructured accordingly: the proof gate is now pinned directly on
resolve_proof in both directions, plus new coverage for load_database
deriving status from proof_checked and for the symbol tokenization.
49 tests, 48 green (the one failure remains the pre-existing unrelated
lean_agent import error in test_dataset.py).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
This commit is contained in:
2026-09-18 19:50:10 +00:00
parent fbcb0c3383
commit 9e0457a0e1
6 changed files with 230 additions and 128 deletions
+12 -7
View File
@@ -6,8 +6,8 @@ distinct things `lean-bench`/`lean-agent` can use, not one:
- **The bench** (`datasets/goodman-1.6-exercises.jsonl`): real problems the reader
is meant to solve, sourced from the book's own **Exercises** at the end of each
section. These are the actual targets `lean-bench` scores an agent against.
- **The database** (`database/goodman-1.6.jsonl` -> `lib/Goodman/Chapter01Section06.lean`
+ `database/goodman-1.6.index.jsonl`): the book's stated Theorem/Proposition/
- **The database** (`database/goodman-1.6.jsonl` -> `lib/Goodman/Chapter01Section06.lean`):
the book's stated Theorem/Proposition/
Lemma/Corollary claims from the main text, turned by `build_database.py` into real
Lean declarations an agent can **cite by name** while proving a bench target --
the "third party math library" role Mathlib plays, except scoped to this book.
@@ -25,7 +25,13 @@ distinct things `lean-bench`/`lean-agent` can use, not one:
axiom here still needs a human to have read it against the book.
- **Retrieval** (`search_database.py`): given the current goal, ranks database
premises by identifier/symbol overlap with rare identifiers weighted higher, and
formats the top few for the LLM prompt. This is the step the proof loop calls at
formats the top few for the LLM prompt. It reads the database JSONL directly and
derives each premise's declaration name, identifiers and proof status itself
(sharing those functions with `build_database.py`), so the data exists in exactly
one place. There is deliberately no generated index: at this scale deriving them
costs microseconds, and a second copy that nothing checks for freshness could
silently serve statements that no longer match the `.lean` file. An index earns
its place once it holds something actually expensive, i.e. embeddings. This is the step the proof loop calls at
each line -- it narrows ~20 premises to the few worth showing, and cannot tell you
whether one applies; only Lean can. Swapping in embeddings later means replacing
`score_premise()` and nothing else.
@@ -195,17 +201,16 @@ the next build.
```sh
python3 textbook/build_database.py database/goodman-1.6.jsonl \
--out-lean lib/Goodman/Chapter01Section06.lean \
--out-index database/goodman-1.6.index.jsonl \
--require-checked
```
Add `--results results/goodman-1.6-library-001/results.jsonl` once a `lean-bench`
run over `database/goodman-1.6.jsonl` has proven some of them, to upgrade those
entries from `axiom` to real theorems. Then query it with the current goal:
entries from `axiom` to real theorems. Query it with the current goal:
```sh
python3 textbook/search_database.py database/goodman-1.6.index.jsonl \
"\u22a2 Nat.Prime p \u2192 (p:\u2124) \u2223 List.prod as \u2192 \u2203 a \u2208 as, (p:\u2124) \u2223 a"
python3 textbook/search_database.py database/goodman-1.6.jsonl \
"⊢ Nat.Prime p → (p:ℤ) ∣ List.prod as → ∃ a ∈ as, (p:ℤ) ∣ a"
```
(`build_library.py` predates this and remains for the narrower job of assembling
+43 -51
View File
@@ -5,23 +5,21 @@ Theorem/Proposition/Lemma/Corollary claims, turned into real Lean declarations t
lean-agent can cite by name while proving something else (the bench targets, which
come from the exercises instead -- see formalize_exercises.py).
Two outputs:
Output: an importable Lean file where every database entry is a named declaration,
so a proof can cite goodman_1_6_20 by name. An entry is emitted as a real
`theorem ... := by <proof>` once its proof has been verified, and otherwise as an
`axiom` -- asserted from the textbook, which is what the book itself does for its
own body results before an exercise asks you to use them. This is deliberate and
has a real cost: anything proven *through* an axiom inherits it, which is exactly
why the verifier's `#print axioms` check matters. A statement that only elaborates
is not the same as a statement that is true, and a garbled extraction asserted as
an axiom can make downstream "proofs" worthless -- so every axiom here needs to have
been read by a human against the book, and `--require-checked` refuses to emit any
entry not marked as having passed check_statements.py.
1. An importable Lean file where every database entry is a named declaration. An
entry is emitted as a real `theorem ... := by <proof>` when a lean-bench run has
actually proven it (status "proved"), and otherwise as an `axiom` -- asserted from
the textbook, which is what the book itself does for its own body results before
an exercise asks you to use them. This is deliberate and has a real cost: anything
proven *through* an axiom inherits it, which is exactly why the verifier's
`#print axioms` check matters. A statement that only elaborates is not the same as
a statement that is true, and a garbled extraction asserted as an axiom can make
downstream "proofs" worthless -- so every axiom here needs to have been read by a
human against the book, and `--require-checked` refuses to emit any entry not
marked as having passed check_statements.py.
2. A search index (JSONL) carrying each entry's declaration name, statement, the
Mathlib identifiers it mentions, and its status -- what search_database.py ranks
against the current goals at each step of a proof.
There is deliberately no generated search index: search_database.py reads this same
source file and derives the declaration name, identifiers and status itself, so
there is only ever one copy of the data and nothing to keep in sync.
Pass a lean-bench run's results.jsonl with --results to upgrade the entries it
proved from axiom to a real theorem carrying the exact proof that run verified.
@@ -53,8 +51,26 @@ def split_prelude(prelude):
return imports, '\n'.join(other).strip()
def resolve_proof(entry, result=None):
"""(proof, status) for one entry.
A proof only counts once Lean has verified it: check_proofs.py sets
proof_checked on the entry, or a lean-bench run reports success. An unverified
proof leaves the entry asserted, reported as proof_pending so it is visible
without being trusted."""
run_proof = result['attempts'][-1]['proof'] if (
result and result.get('success') and result.get('attempts')) else None
own_proof = entry.get('proof') if entry.get('proof_checked') else None
proof = run_proof or own_proof
if proof:
return proof, 'proved'
if entry.get('proof'):
return None, 'proof_pending'
return None, 'asserted'
def statement_identifiers(statement):
"""Identifiers mentioned in a statement, for the search index."""
"""Identifiers mentioned in a statement, for premise retrieval."""
found = []
for match in IDENT_RE.findall(statement or ''):
if match in STOPWORDS or match in found or len(match) < 2:
@@ -89,7 +105,7 @@ def build(entries, results=None, require_checked=False):
lines.append(block)
lines.append('')
index, emitted, skipped = [], [], []
emitted, skipped = [], []
for entry in entries:
entry_id = entry['id']
if require_checked and not entry.get('elaborates'):
@@ -97,17 +113,9 @@ def build(entries, results=None, require_checked=False):
continue
name = decl_name(entry_id)
result = results.get(entry_id)
# A proof can come from a lean-bench run, or be recorded on the entry itself
# (hand-written, or authored by an agent outside a bench run). Either way it
# only counts once check_proofs.py has actually compiled it: an unverified
# proof stays an axiom, so the library never claims more than Lean confirmed.
run_proof = result['attempts'][-1]['proof'] if (
result and result.get('success') and result.get('attempts')) else None
own_proof = entry.get('proof') if entry.get('proof_checked') else None
proof = run_proof or own_proof
proved = bool(proof)
proof_pending = bool(not proved and entry.get('proof'))
proof, status = resolve_proof(entry, results.get(entry_id))
proved = status == 'proved'
proof_pending = status == 'proof_pending'
provenance = entry.get('source_label', entry_id)
doc = f'/-- {provenance} (Goodman, Algebra: Abstract and Concrete).'
@@ -129,21 +137,9 @@ def build(entries, results=None, require_checked=False):
lines.append(f'axiom {name} : {entry["statement"]}')
lines.append('')
index.append({
"name": name,
"id": entry_id,
"source_label": entry.get('source_label'),
"kind": entry.get('kind'),
"status": "proved" if proved else ("proof_pending" if proof_pending else "asserted"),
"proof": proof if proved else entry.get('proof'),
"statement": entry['statement'],
"natural_language": entry.get('natural_language', '').split('[fix note:')[0].strip(),
"identifiers": statement_identifiers(entry['statement']),
"requires": entry.get('requires', []),
})
emitted.append(entry_id)
return '\n'.join(lines), index, emitted, skipped
return '\n'.join(lines), emitted, skipped
def main():
@@ -151,27 +147,23 @@ def main():
parser.add_argument('database', type=Path, help='JSONL of main-content statements, e.g. database/goodman-1.6.jsonl')
parser.add_argument('--results', type=Path, help="a lean-bench run's results.jsonl; proven entries become real theorems")
parser.add_argument('--out-lean', type=Path, required=True, help='importable Lean file of citable premises')
parser.add_argument('--out-index', type=Path, required=True, help='JSONL search index for search_database.py')
parser.add_argument('--require-checked', action='store_true',
help='refuse entries not marked "elaborates": true (i.e. not yet verified by check_statements.py)')
args = parser.parse_args()
entries = load_jsonl(args.database)
results = {r['id']: r for r in load_jsonl(args.results)} if args.results else None
text, index, emitted, skipped = build(entries, results, args.require_checked)
text, emitted, skipped = build(entries, results, args.require_checked)
args.out_lean.parent.mkdir(parents=True, exist_ok=True)
args.out_lean.write_text(text + '\n', encoding='utf-8')
args.out_index.parent.mkdir(parents=True, exist_ok=True)
with args.out_index.open('w', encoding='utf-8') as stream:
for record in index:
stream.write(json.dumps(record, ensure_ascii=False) + '\n')
proved = sum(1 for r in index if r['status'] == 'proved')
pending = sum(1 for r in index if r['status'] == 'proof_pending')
by_id = {e['id']: e for e in entries}
statuses = [resolve_proof(by_id[i], (results or {}).get(i))[1] for i in emitted]
proved = statuses.count('proved')
pending = statuses.count('proof_pending')
print(f'{len(emitted)} premises -> {args.out_lean} ({proved} proven, '
f'{len(emitted) - proved} asserted as axioms, of which {pending} have an unverified proof)')
print(f'search index -> {args.out_index}')
if skipped:
print(f'{len(skipped)} skipped for not being elaboration-checked: {", ".join(skipped)}')
+58 -12
View File
@@ -7,6 +7,13 @@ for entries that might apply to the goal in front of it. The ranked results are
gets shown to the LLM alongside the goal, so the next line of Lean it writes can
cite a real premise by name instead of inventing a lemma.
It reads the database JSONL directly and derives each premise's declaration name,
identifiers and proof status on the fly, sharing those functions with
build_database.py. There is no precomputed index: at this scale deriving them costs
microseconds, and a second generated copy of the data would only create something
that can silently go stale against the source. A precomputed index earns its place
when it holds something genuinely expensive, i.e. embeddings.
Scoring here is deliberately simple and dependency-free: overlap of identifiers and
symbols between the goal and each premise, weighted so that a rare identifier
matters more than a common one (`IsGCDOfList` appearing in both is a much stronger
@@ -23,22 +30,61 @@ import re
from collections import Counter
from pathlib import Path
TOKEN_RE = re.compile(r'[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)+|[A-Za-z_][A-Za-z0-9_]*|[∣∀∃↔→∧∨¬≤≥≠∈]')
# Qualified names (Nat.Prime) first so they aren't split, then bare identifiers, then
# the type and operator symbols that carry signal. Arithmetic operators are included
# deliberately: a goal that is pure algebra ("a * (b + c) = a * b + a * c") has no
# identifiers at all, and without these it would tokenize to nothing and retrieve
# nothing. They appear in nearly every premise, so inverse-document-frequency
# weighting already discounts them to almost zero -- they only decide ties.
TOKEN_RE = re.compile(
r'[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)+'
r'|[A-Za-z_][A-Za-z0-9_]*'
r'|[∣∀∃↔→∧∨¬≤≥≠∈ℤℕℚℝ×∑∏]'
r'|[+*=/-]')
STOPWORDS = {'let', 'in', 'fun', 'if', 'then', 'else', 'true', 'false', 'Prop', 'Type', 'this'}
def load_index(path):
return [json.loads(line) for line in path.read_text(encoding='utf-8').splitlines() if line.strip()]
try:
from .build_database import decl_name, resolve_proof, statement_identifiers
except ImportError:
from build_database import decl_name, resolve_proof, statement_identifiers
def load_database(path, results=None):
"""Database entries in the shape search() ranks, derived from the source file."""
entries = [json.loads(line) for line in path.read_text(encoding='utf-8').splitlines() if line.strip()]
records = []
for entry in entries:
proof, status = resolve_proof(entry, (results or {}).get(entry['id']))
records.append({
"name": decl_name(entry['id']),
"id": entry['id'],
"source_label": entry.get('source_label'),
"kind": entry.get('kind'),
"status": status,
"proof": proof or entry.get('proof'),
"statement": entry['statement'],
"natural_language": entry.get('natural_language', '').split('[fix note:')[0].strip(),
"identifiers": statement_identifiers(entry['statement']),
"requires": entry.get('requires', []),
})
return records
SYMBOLS = set('∣∀∃↔→∧∨¬≤≥≠∈ℤℕℚℝ×∑∏+*=/-')
def tokenize(text):
return [t for t in TOKEN_RE.findall(text or '') if t not in STOPWORDS and len(t) > 1]
"""Identifiers and meaningful symbols. Single-letter variable names are dropped
(matching `a` against `a` says nothing), but single-character symbols are kept."""
return [t for t in TOKEN_RE.findall(text or '')
if t not in STOPWORDS and (len(t) > 1 or t in SYMBOLS)]
def document_frequencies(index):
def document_frequencies(records):
"""How many premises each token appears in, for inverse-document-frequency weighting."""
frequencies = Counter()
for record in index:
for record in records:
for token in set(tokenize(record['statement']) + record.get('identifiers', [])
+ record.get('requires', [])):
frequencies[token] += 1
@@ -57,13 +103,13 @@ def score_premise(goal_tokens, record, frequencies, total):
return score
def search(index, goal, limit=5):
def search(records, goal, limit=5):
"""Premises ranked against one goal, best first. Zero-score entries are dropped."""
frequencies = document_frequencies(index)
total = max(len(index), 1)
frequencies = document_frequencies(records)
total = max(len(records), 1)
goal_tokens = tokenize(goal)
scored = []
for record in index:
for record in records:
score = score_premise(goal_tokens, record, frequencies, total)
if score > 0:
scored.append((score, record))
@@ -85,13 +131,13 @@ def format_for_prompt(hits):
def main():
parser = argparse.ArgumentParser(description='Rank database premises against a proof goal')
parser.add_argument('index', type=Path, help='search index from build_database.py')
parser.add_argument('database', type=Path, help='database JSONL, e.g. database/goodman-1.6.jsonl')
parser.add_argument('goal', help='the current goal, as Lean prints it')
parser.add_argument('--limit', type=int, default=5)
parser.add_argument('--json', action='store_true', help='emit raw JSON instead of prompt text')
args = parser.parse_args()
hits = search(load_index(args.index), args.goal, args.limit)
hits = search(load_database(args.database), args.goal, args.limit)
print(json.dumps(hits, ensure_ascii=False, indent=2) if args.json else format_for_prompt(hits))