Drop the generated search index; derive it from the source instead
The index file was 12KB of which ~96% was a byte-for-byte copy of
database/goodman-1.6.jsonl. Its only original content was three derived
fields: `name` (literally id.replace('-','_')), `status` (a two-line
computation over proof_checked), and `identifiers` (a regex over the
statement). It was a cache for work that costs microseconds, and it
created a real hazard: nothing checked that it matched its source, so
editing a statement and forgetting to regenerate would leave
search_database.py serving a premise whose text no longer matched the
generated .lean file.
- search_database.py now reads database/goodman-1.6.jsonl directly via a
new load_database(), deriving name/identifiers/status on the fly and
importing those functions from build_database.py so there is one
definition of each rather than two that can drift.
- build_database.py drops --out-index and index generation, and its
proof/status derivation is factored into resolve_proof() shared with
retrieval.
- database/goodman-1.6.index.jsonl deleted.
Also fixes a retrieval bug the removal surfaced: the tokenizer only
matched ASCII identifiers, so a goal that is pure algebra
("a * (b + c) = a * b + a * c") tokenized to nothing and retrieved
nothing at all -- it now keeps type and operator symbols, and drops
single-letter variable names, which say nothing when matched. That goal
now correctly finds Proposition 1.6.1, and the identifier-rich goals
still rank their exact premise first by a wide margin (9.3 vs 4.4).
Tests restructured accordingly: the proof gate is now pinned directly on
resolve_proof in both directions, plus new coverage for load_database
deriving status from proof_checked and for the symbol tokenization.
49 tests, 48 green (the one failure remains the pre-existing unrelated
lean_agent import error in test_dataset.py).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
This commit is contained in:
+12
-7
@@ -6,8 +6,8 @@ distinct things `lean-bench`/`lean-agent` can use, not one:
|
||||
- **The bench** (`datasets/goodman-1.6-exercises.jsonl`): real problems the reader
|
||||
is meant to solve, sourced from the book's own **Exercises** at the end of each
|
||||
section. These are the actual targets `lean-bench` scores an agent against.
|
||||
- **The database** (`database/goodman-1.6.jsonl` -> `lib/Goodman/Chapter01Section06.lean`
|
||||
+ `database/goodman-1.6.index.jsonl`): the book's stated Theorem/Proposition/
|
||||
- **The database** (`database/goodman-1.6.jsonl` -> `lib/Goodman/Chapter01Section06.lean`):
|
||||
the book's stated Theorem/Proposition/
|
||||
Lemma/Corollary claims from the main text, turned by `build_database.py` into real
|
||||
Lean declarations an agent can **cite by name** while proving a bench target --
|
||||
the "third party math library" role Mathlib plays, except scoped to this book.
|
||||
@@ -25,7 +25,13 @@ distinct things `lean-bench`/`lean-agent` can use, not one:
|
||||
axiom here still needs a human to have read it against the book.
|
||||
- **Retrieval** (`search_database.py`): given the current goal, ranks database
|
||||
premises by identifier/symbol overlap with rare identifiers weighted higher, and
|
||||
formats the top few for the LLM prompt. This is the step the proof loop calls at
|
||||
formats the top few for the LLM prompt. It reads the database JSONL directly and
|
||||
derives each premise's declaration name, identifiers and proof status itself
|
||||
(sharing those functions with `build_database.py`), so the data exists in exactly
|
||||
one place. There is deliberately no generated index: at this scale deriving them
|
||||
costs microseconds, and a second copy that nothing checks for freshness could
|
||||
silently serve statements that no longer match the `.lean` file. An index earns
|
||||
its place once it holds something actually expensive, i.e. embeddings. This is the step the proof loop calls at
|
||||
each line -- it narrows ~20 premises to the few worth showing, and cannot tell you
|
||||
whether one applies; only Lean can. Swapping in embeddings later means replacing
|
||||
`score_premise()` and nothing else.
|
||||
@@ -195,17 +201,16 @@ the next build.
|
||||
```sh
|
||||
python3 textbook/build_database.py database/goodman-1.6.jsonl \
|
||||
--out-lean lib/Goodman/Chapter01Section06.lean \
|
||||
--out-index database/goodman-1.6.index.jsonl \
|
||||
--require-checked
|
||||
```
|
||||
|
||||
Add `--results results/goodman-1.6-library-001/results.jsonl` once a `lean-bench`
|
||||
run over `database/goodman-1.6.jsonl` has proven some of them, to upgrade those
|
||||
entries from `axiom` to real theorems. Then query it with the current goal:
|
||||
entries from `axiom` to real theorems. Query it with the current goal:
|
||||
|
||||
```sh
|
||||
python3 textbook/search_database.py database/goodman-1.6.index.jsonl \
|
||||
"\u22a2 Nat.Prime p \u2192 (p:\u2124) \u2223 List.prod as \u2192 \u2203 a \u2208 as, (p:\u2124) \u2223 a"
|
||||
python3 textbook/search_database.py database/goodman-1.6.jsonl \
|
||||
"⊢ Nat.Prime p → (p:ℤ) ∣ List.prod as → ∃ a ∈ as, (p:ℤ) ∣ a"
|
||||
```
|
||||
|
||||
(`build_library.py` predates this and remains for the narrower job of assembling
|
||||
|
||||
+43
-51
@@ -5,23 +5,21 @@ Theorem/Proposition/Lemma/Corollary claims, turned into real Lean declarations t
|
||||
lean-agent can cite by name while proving something else (the bench targets, which
|
||||
come from the exercises instead -- see formalize_exercises.py).
|
||||
|
||||
Two outputs:
|
||||
Output: an importable Lean file where every database entry is a named declaration,
|
||||
so a proof can cite goodman_1_6_20 by name. An entry is emitted as a real
|
||||
`theorem ... := by <proof>` once its proof has been verified, and otherwise as an
|
||||
`axiom` -- asserted from the textbook, which is what the book itself does for its
|
||||
own body results before an exercise asks you to use them. This is deliberate and
|
||||
has a real cost: anything proven *through* an axiom inherits it, which is exactly
|
||||
why the verifier's `#print axioms` check matters. A statement that only elaborates
|
||||
is not the same as a statement that is true, and a garbled extraction asserted as
|
||||
an axiom can make downstream "proofs" worthless -- so every axiom here needs to have
|
||||
been read by a human against the book, and `--require-checked` refuses to emit any
|
||||
entry not marked as having passed check_statements.py.
|
||||
|
||||
1. An importable Lean file where every database entry is a named declaration. An
|
||||
entry is emitted as a real `theorem ... := by <proof>` when a lean-bench run has
|
||||
actually proven it (status "proved"), and otherwise as an `axiom` -- asserted from
|
||||
the textbook, which is what the book itself does for its own body results before
|
||||
an exercise asks you to use them. This is deliberate and has a real cost: anything
|
||||
proven *through* an axiom inherits it, which is exactly why the verifier's
|
||||
`#print axioms` check matters. A statement that only elaborates is not the same as
|
||||
a statement that is true, and a garbled extraction asserted as an axiom can make
|
||||
downstream "proofs" worthless -- so every axiom here needs to have been read by a
|
||||
human against the book, and `--require-checked` refuses to emit any entry not
|
||||
marked as having passed check_statements.py.
|
||||
|
||||
2. A search index (JSONL) carrying each entry's declaration name, statement, the
|
||||
Mathlib identifiers it mentions, and its status -- what search_database.py ranks
|
||||
against the current goals at each step of a proof.
|
||||
There is deliberately no generated search index: search_database.py reads this same
|
||||
source file and derives the declaration name, identifiers and status itself, so
|
||||
there is only ever one copy of the data and nothing to keep in sync.
|
||||
|
||||
Pass a lean-bench run's results.jsonl with --results to upgrade the entries it
|
||||
proved from axiom to a real theorem carrying the exact proof that run verified.
|
||||
@@ -53,8 +51,26 @@ def split_prelude(prelude):
|
||||
return imports, '\n'.join(other).strip()
|
||||
|
||||
|
||||
def resolve_proof(entry, result=None):
|
||||
"""(proof, status) for one entry.
|
||||
|
||||
A proof only counts once Lean has verified it: check_proofs.py sets
|
||||
proof_checked on the entry, or a lean-bench run reports success. An unverified
|
||||
proof leaves the entry asserted, reported as proof_pending so it is visible
|
||||
without being trusted."""
|
||||
run_proof = result['attempts'][-1]['proof'] if (
|
||||
result and result.get('success') and result.get('attempts')) else None
|
||||
own_proof = entry.get('proof') if entry.get('proof_checked') else None
|
||||
proof = run_proof or own_proof
|
||||
if proof:
|
||||
return proof, 'proved'
|
||||
if entry.get('proof'):
|
||||
return None, 'proof_pending'
|
||||
return None, 'asserted'
|
||||
|
||||
|
||||
def statement_identifiers(statement):
|
||||
"""Identifiers mentioned in a statement, for the search index."""
|
||||
"""Identifiers mentioned in a statement, for premise retrieval."""
|
||||
found = []
|
||||
for match in IDENT_RE.findall(statement or ''):
|
||||
if match in STOPWORDS or match in found or len(match) < 2:
|
||||
@@ -89,7 +105,7 @@ def build(entries, results=None, require_checked=False):
|
||||
lines.append(block)
|
||||
lines.append('')
|
||||
|
||||
index, emitted, skipped = [], [], []
|
||||
emitted, skipped = [], []
|
||||
for entry in entries:
|
||||
entry_id = entry['id']
|
||||
if require_checked and not entry.get('elaborates'):
|
||||
@@ -97,17 +113,9 @@ def build(entries, results=None, require_checked=False):
|
||||
continue
|
||||
|
||||
name = decl_name(entry_id)
|
||||
result = results.get(entry_id)
|
||||
# A proof can come from a lean-bench run, or be recorded on the entry itself
|
||||
# (hand-written, or authored by an agent outside a bench run). Either way it
|
||||
# only counts once check_proofs.py has actually compiled it: an unverified
|
||||
# proof stays an axiom, so the library never claims more than Lean confirmed.
|
||||
run_proof = result['attempts'][-1]['proof'] if (
|
||||
result and result.get('success') and result.get('attempts')) else None
|
||||
own_proof = entry.get('proof') if entry.get('proof_checked') else None
|
||||
proof = run_proof or own_proof
|
||||
proved = bool(proof)
|
||||
proof_pending = bool(not proved and entry.get('proof'))
|
||||
proof, status = resolve_proof(entry, results.get(entry_id))
|
||||
proved = status == 'proved'
|
||||
proof_pending = status == 'proof_pending'
|
||||
provenance = entry.get('source_label', entry_id)
|
||||
|
||||
doc = f'/-- {provenance} (Goodman, Algebra: Abstract and Concrete).'
|
||||
@@ -129,21 +137,9 @@ def build(entries, results=None, require_checked=False):
|
||||
lines.append(f'axiom {name} : {entry["statement"]}')
|
||||
lines.append('')
|
||||
|
||||
index.append({
|
||||
"name": name,
|
||||
"id": entry_id,
|
||||
"source_label": entry.get('source_label'),
|
||||
"kind": entry.get('kind'),
|
||||
"status": "proved" if proved else ("proof_pending" if proof_pending else "asserted"),
|
||||
"proof": proof if proved else entry.get('proof'),
|
||||
"statement": entry['statement'],
|
||||
"natural_language": entry.get('natural_language', '').split('[fix note:')[0].strip(),
|
||||
"identifiers": statement_identifiers(entry['statement']),
|
||||
"requires": entry.get('requires', []),
|
||||
})
|
||||
emitted.append(entry_id)
|
||||
|
||||
return '\n'.join(lines), index, emitted, skipped
|
||||
return '\n'.join(lines), emitted, skipped
|
||||
|
||||
|
||||
def main():
|
||||
@@ -151,27 +147,23 @@ def main():
|
||||
parser.add_argument('database', type=Path, help='JSONL of main-content statements, e.g. database/goodman-1.6.jsonl')
|
||||
parser.add_argument('--results', type=Path, help="a lean-bench run's results.jsonl; proven entries become real theorems")
|
||||
parser.add_argument('--out-lean', type=Path, required=True, help='importable Lean file of citable premises')
|
||||
parser.add_argument('--out-index', type=Path, required=True, help='JSONL search index for search_database.py')
|
||||
parser.add_argument('--require-checked', action='store_true',
|
||||
help='refuse entries not marked "elaborates": true (i.e. not yet verified by check_statements.py)')
|
||||
args = parser.parse_args()
|
||||
|
||||
entries = load_jsonl(args.database)
|
||||
results = {r['id']: r for r in load_jsonl(args.results)} if args.results else None
|
||||
text, index, emitted, skipped = build(entries, results, args.require_checked)
|
||||
text, emitted, skipped = build(entries, results, args.require_checked)
|
||||
|
||||
args.out_lean.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out_lean.write_text(text + '\n', encoding='utf-8')
|
||||
args.out_index.parent.mkdir(parents=True, exist_ok=True)
|
||||
with args.out_index.open('w', encoding='utf-8') as stream:
|
||||
for record in index:
|
||||
stream.write(json.dumps(record, ensure_ascii=False) + '\n')
|
||||
|
||||
proved = sum(1 for r in index if r['status'] == 'proved')
|
||||
pending = sum(1 for r in index if r['status'] == 'proof_pending')
|
||||
by_id = {e['id']: e for e in entries}
|
||||
statuses = [resolve_proof(by_id[i], (results or {}).get(i))[1] for i in emitted]
|
||||
proved = statuses.count('proved')
|
||||
pending = statuses.count('proof_pending')
|
||||
print(f'{len(emitted)} premises -> {args.out_lean} ({proved} proven, '
|
||||
f'{len(emitted) - proved} asserted as axioms, of which {pending} have an unverified proof)')
|
||||
print(f'search index -> {args.out_index}')
|
||||
if skipped:
|
||||
print(f'{len(skipped)} skipped for not being elaboration-checked: {", ".join(skipped)}')
|
||||
|
||||
|
||||
+58
-12
@@ -7,6 +7,13 @@ for entries that might apply to the goal in front of it. The ranked results are
|
||||
gets shown to the LLM alongside the goal, so the next line of Lean it writes can
|
||||
cite a real premise by name instead of inventing a lemma.
|
||||
|
||||
It reads the database JSONL directly and derives each premise's declaration name,
|
||||
identifiers and proof status on the fly, sharing those functions with
|
||||
build_database.py. There is no precomputed index: at this scale deriving them costs
|
||||
microseconds, and a second generated copy of the data would only create something
|
||||
that can silently go stale against the source. A precomputed index earns its place
|
||||
when it holds something genuinely expensive, i.e. embeddings.
|
||||
|
||||
Scoring here is deliberately simple and dependency-free: overlap of identifiers and
|
||||
symbols between the goal and each premise, weighted so that a rare identifier
|
||||
matters more than a common one (`IsGCDOfList` appearing in both is a much stronger
|
||||
@@ -23,22 +30,61 @@ import re
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
TOKEN_RE = re.compile(r'[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)+|[A-Za-z_][A-Za-z0-9_]*|[∣∀∃↔→∧∨¬≤≥≠∈]')
|
||||
# Qualified names (Nat.Prime) first so they aren't split, then bare identifiers, then
|
||||
# the type and operator symbols that carry signal. Arithmetic operators are included
|
||||
# deliberately: a goal that is pure algebra ("a * (b + c) = a * b + a * c") has no
|
||||
# identifiers at all, and without these it would tokenize to nothing and retrieve
|
||||
# nothing. They appear in nearly every premise, so inverse-document-frequency
|
||||
# weighting already discounts them to almost zero -- they only decide ties.
|
||||
TOKEN_RE = re.compile(
|
||||
r'[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)+'
|
||||
r'|[A-Za-z_][A-Za-z0-9_]*'
|
||||
r'|[∣∀∃↔→∧∨¬≤≥≠∈ℤℕℚℝ×∑∏]'
|
||||
r'|[+*=/-]')
|
||||
STOPWORDS = {'let', 'in', 'fun', 'if', 'then', 'else', 'true', 'false', 'Prop', 'Type', 'this'}
|
||||
|
||||
|
||||
def load_index(path):
|
||||
return [json.loads(line) for line in path.read_text(encoding='utf-8').splitlines() if line.strip()]
|
||||
try:
|
||||
from .build_database import decl_name, resolve_proof, statement_identifiers
|
||||
except ImportError:
|
||||
from build_database import decl_name, resolve_proof, statement_identifiers
|
||||
|
||||
|
||||
def load_database(path, results=None):
|
||||
"""Database entries in the shape search() ranks, derived from the source file."""
|
||||
entries = [json.loads(line) for line in path.read_text(encoding='utf-8').splitlines() if line.strip()]
|
||||
records = []
|
||||
for entry in entries:
|
||||
proof, status = resolve_proof(entry, (results or {}).get(entry['id']))
|
||||
records.append({
|
||||
"name": decl_name(entry['id']),
|
||||
"id": entry['id'],
|
||||
"source_label": entry.get('source_label'),
|
||||
"kind": entry.get('kind'),
|
||||
"status": status,
|
||||
"proof": proof or entry.get('proof'),
|
||||
"statement": entry['statement'],
|
||||
"natural_language": entry.get('natural_language', '').split('[fix note:')[0].strip(),
|
||||
"identifiers": statement_identifiers(entry['statement']),
|
||||
"requires": entry.get('requires', []),
|
||||
})
|
||||
return records
|
||||
|
||||
|
||||
SYMBOLS = set('∣∀∃↔→∧∨¬≤≥≠∈ℤℕℚℝ×∑∏+*=/-')
|
||||
|
||||
|
||||
def tokenize(text):
|
||||
return [t for t in TOKEN_RE.findall(text or '') if t not in STOPWORDS and len(t) > 1]
|
||||
"""Identifiers and meaningful symbols. Single-letter variable names are dropped
|
||||
(matching `a` against `a` says nothing), but single-character symbols are kept."""
|
||||
return [t for t in TOKEN_RE.findall(text or '')
|
||||
if t not in STOPWORDS and (len(t) > 1 or t in SYMBOLS)]
|
||||
|
||||
|
||||
def document_frequencies(index):
|
||||
def document_frequencies(records):
|
||||
"""How many premises each token appears in, for inverse-document-frequency weighting."""
|
||||
frequencies = Counter()
|
||||
for record in index:
|
||||
for record in records:
|
||||
for token in set(tokenize(record['statement']) + record.get('identifiers', [])
|
||||
+ record.get('requires', [])):
|
||||
frequencies[token] += 1
|
||||
@@ -57,13 +103,13 @@ def score_premise(goal_tokens, record, frequencies, total):
|
||||
return score
|
||||
|
||||
|
||||
def search(index, goal, limit=5):
|
||||
def search(records, goal, limit=5):
|
||||
"""Premises ranked against one goal, best first. Zero-score entries are dropped."""
|
||||
frequencies = document_frequencies(index)
|
||||
total = max(len(index), 1)
|
||||
frequencies = document_frequencies(records)
|
||||
total = max(len(records), 1)
|
||||
goal_tokens = tokenize(goal)
|
||||
scored = []
|
||||
for record in index:
|
||||
for record in records:
|
||||
score = score_premise(goal_tokens, record, frequencies, total)
|
||||
if score > 0:
|
||||
scored.append((score, record))
|
||||
@@ -85,13 +131,13 @@ def format_for_prompt(hits):
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Rank database premises against a proof goal')
|
||||
parser.add_argument('index', type=Path, help='search index from build_database.py')
|
||||
parser.add_argument('database', type=Path, help='database JSONL, e.g. database/goodman-1.6.jsonl')
|
||||
parser.add_argument('goal', help='the current goal, as Lean prints it')
|
||||
parser.add_argument('--limit', type=int, default=5)
|
||||
parser.add_argument('--json', action='store_true', help='emit raw JSON instead of prompt text')
|
||||
args = parser.parse_args()
|
||||
|
||||
hits = search(load_index(args.index), args.goal, args.limit)
|
||||
hits = search(load_database(args.database), args.goal, args.limit)
|
||||
print(json.dumps(hits, ensure_ascii=False, indent=2) if args.json else format_for_prompt(hits))
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user