Drop the generated search index; derive it from the source instead

The index file was 12KB of which ~96% was a byte-for-byte copy of
database/goodman-1.6.jsonl. Its only original content was three derived
fields: `name` (literally id.replace('-','_')), `status` (a two-line
computation over proof_checked), and `identifiers` (a regex over the
statement). It was a cache for work that costs microseconds, and it
created a real hazard: nothing checked that it matched its source, so
editing a statement and forgetting to regenerate would leave
search_database.py serving a premise whose text no longer matched the
generated .lean file.

- search_database.py now reads database/goodman-1.6.jsonl directly via a
  new load_database(), deriving name/identifiers/status on the fly and
  importing those functions from build_database.py so there is one
  definition of each rather than two that can drift.
- build_database.py drops --out-index and index generation, and its
  proof/status derivation is factored into resolve_proof() shared with
  retrieval.
- database/goodman-1.6.index.jsonl deleted.

Also fixes a retrieval bug the removal surfaced: the tokenizer only
matched ASCII identifiers, so a goal that is pure algebra
("a * (b + c) = a * b + a * c") tokenized to nothing and retrieved
nothing at all -- it now keeps type and operator symbols, and drops
single-letter variable names, which say nothing when matched. That goal
now correctly finds Proposition 1.6.1, and the identifier-rich goals
still rank their exact premise first by a wide margin (9.3 vs 4.4).

Tests restructured accordingly: the proof gate is now pinned directly on
resolve_proof in both directions, plus new coverage for load_database
deriving status from proof_checked and for the symbol tokenization.
49 tests, 48 green (the one failure remains the pre-existing unrelated
lean_agent import error in test_dataset.py).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
This commit is contained in:
2026-09-18 19:50:10 +00:00
parent fbcb0c3383
commit 9e0457a0e1
6 changed files with 230 additions and 128 deletions
+43 -51
View File
@@ -5,23 +5,21 @@ Theorem/Proposition/Lemma/Corollary claims, turned into real Lean declarations t
lean-agent can cite by name while proving something else (the bench targets, which
come from the exercises instead -- see formalize_exercises.py).
Two outputs:
Output: an importable Lean file where every database entry is a named declaration,
so a proof can cite goodman_1_6_20 by name. An entry is emitted as a real
`theorem ... := by <proof>` once its proof has been verified, and otherwise as an
`axiom` -- asserted from the textbook, which is what the book itself does for its
own body results before an exercise asks you to use them. This is deliberate and
has a real cost: anything proven *through* an axiom inherits it, which is exactly
why the verifier's `#print axioms` check matters. A statement that only elaborates
is not the same as a statement that is true, and a garbled extraction asserted as
an axiom can make downstream "proofs" worthless -- so every axiom here needs to have
been read by a human against the book, and `--require-checked` refuses to emit any
entry not marked as having passed check_statements.py.
1. An importable Lean file where every database entry is a named declaration. An
entry is emitted as a real `theorem ... := by <proof>` when a lean-bench run has
actually proven it (status "proved"), and otherwise as an `axiom` -- asserted from
the textbook, which is what the book itself does for its own body results before
an exercise asks you to use them. This is deliberate and has a real cost: anything
proven *through* an axiom inherits it, which is exactly why the verifier's
`#print axioms` check matters. A statement that only elaborates is not the same as
a statement that is true, and a garbled extraction asserted as an axiom can make
downstream "proofs" worthless -- so every axiom here needs to have been read by a
human against the book, and `--require-checked` refuses to emit any entry not
marked as having passed check_statements.py.
2. A search index (JSONL) carrying each entry's declaration name, statement, the
Mathlib identifiers it mentions, and its status -- what search_database.py ranks
against the current goals at each step of a proof.
There is deliberately no generated search index: search_database.py reads this same
source file and derives the declaration name, identifiers and status itself, so
there is only ever one copy of the data and nothing to keep in sync.
Pass a lean-bench run's results.jsonl with --results to upgrade the entries it
proved from axiom to a real theorem carrying the exact proof that run verified.
@@ -53,8 +51,26 @@ def split_prelude(prelude):
return imports, '\n'.join(other).strip()
def resolve_proof(entry, result=None):
"""(proof, status) for one entry.
A proof only counts once Lean has verified it: check_proofs.py sets
proof_checked on the entry, or a lean-bench run reports success. An unverified
proof leaves the entry asserted, reported as proof_pending so it is visible
without being trusted."""
run_proof = result['attempts'][-1]['proof'] if (
result and result.get('success') and result.get('attempts')) else None
own_proof = entry.get('proof') if entry.get('proof_checked') else None
proof = run_proof or own_proof
if proof:
return proof, 'proved'
if entry.get('proof'):
return None, 'proof_pending'
return None, 'asserted'
def statement_identifiers(statement):
"""Identifiers mentioned in a statement, for the search index."""
"""Identifiers mentioned in a statement, for premise retrieval."""
found = []
for match in IDENT_RE.findall(statement or ''):
if match in STOPWORDS or match in found or len(match) < 2:
@@ -89,7 +105,7 @@ def build(entries, results=None, require_checked=False):
lines.append(block)
lines.append('')
index, emitted, skipped = [], [], []
emitted, skipped = [], []
for entry in entries:
entry_id = entry['id']
if require_checked and not entry.get('elaborates'):
@@ -97,17 +113,9 @@ def build(entries, results=None, require_checked=False):
continue
name = decl_name(entry_id)
result = results.get(entry_id)
# A proof can come from a lean-bench run, or be recorded on the entry itself
# (hand-written, or authored by an agent outside a bench run). Either way it
# only counts once check_proofs.py has actually compiled it: an unverified
# proof stays an axiom, so the library never claims more than Lean confirmed.
run_proof = result['attempts'][-1]['proof'] if (
result and result.get('success') and result.get('attempts')) else None
own_proof = entry.get('proof') if entry.get('proof_checked') else None
proof = run_proof or own_proof
proved = bool(proof)
proof_pending = bool(not proved and entry.get('proof'))
proof, status = resolve_proof(entry, results.get(entry_id))
proved = status == 'proved'
proof_pending = status == 'proof_pending'
provenance = entry.get('source_label', entry_id)
doc = f'/-- {provenance} (Goodman, Algebra: Abstract and Concrete).'
@@ -129,21 +137,9 @@ def build(entries, results=None, require_checked=False):
lines.append(f'axiom {name} : {entry["statement"]}')
lines.append('')
index.append({
"name": name,
"id": entry_id,
"source_label": entry.get('source_label'),
"kind": entry.get('kind'),
"status": "proved" if proved else ("proof_pending" if proof_pending else "asserted"),
"proof": proof if proved else entry.get('proof'),
"statement": entry['statement'],
"natural_language": entry.get('natural_language', '').split('[fix note:')[0].strip(),
"identifiers": statement_identifiers(entry['statement']),
"requires": entry.get('requires', []),
})
emitted.append(entry_id)
return '\n'.join(lines), index, emitted, skipped
return '\n'.join(lines), emitted, skipped
def main():
@@ -151,27 +147,23 @@ def main():
parser.add_argument('database', type=Path, help='JSONL of main-content statements, e.g. database/goodman-1.6.jsonl')
parser.add_argument('--results', type=Path, help="a lean-bench run's results.jsonl; proven entries become real theorems")
parser.add_argument('--out-lean', type=Path, required=True, help='importable Lean file of citable premises')
parser.add_argument('--out-index', type=Path, required=True, help='JSONL search index for search_database.py')
parser.add_argument('--require-checked', action='store_true',
help='refuse entries not marked "elaborates": true (i.e. not yet verified by check_statements.py)')
args = parser.parse_args()
entries = load_jsonl(args.database)
results = {r['id']: r for r in load_jsonl(args.results)} if args.results else None
text, index, emitted, skipped = build(entries, results, args.require_checked)
text, emitted, skipped = build(entries, results, args.require_checked)
args.out_lean.parent.mkdir(parents=True, exist_ok=True)
args.out_lean.write_text(text + '\n', encoding='utf-8')
args.out_index.parent.mkdir(parents=True, exist_ok=True)
with args.out_index.open('w', encoding='utf-8') as stream:
for record in index:
stream.write(json.dumps(record, ensure_ascii=False) + '\n')
proved = sum(1 for r in index if r['status'] == 'proved')
pending = sum(1 for r in index if r['status'] == 'proof_pending')
by_id = {e['id']: e for e in entries}
statuses = [resolve_proof(by_id[i], (results or {}).get(i))[1] for i in emitted]
proved = statuses.count('proved')
pending = statuses.count('proof_pending')
print(f'{len(emitted)} premises -> {args.out_lean} ({proved} proven, '
f'{len(emitted) - proved} asserted as axioms, of which {pending} have an unverified proof)')
print(f'search index -> {args.out_index}')
if skipped:
print(f'{len(skipped)} skipped for not being elaboration-checked: {", ".join(skipped)}')