Drop the generated search index; derive it from the source instead
The index file was 12KB of which ~96% was a byte-for-byte copy of
database/goodman-1.6.jsonl. Its only original content was three derived
fields: `name` (literally id.replace('-','_')), `status` (a two-line
computation over proof_checked), and `identifiers` (a regex over the
statement). It was a cache for work that costs microseconds, and it
created a real hazard: nothing checked that it matched its source, so
editing a statement and forgetting to regenerate would leave
search_database.py serving a premise whose text no longer matched the
generated .lean file.
- search_database.py now reads database/goodman-1.6.jsonl directly via a
new load_database(), deriving name/identifiers/status on the fly and
importing those functions from build_database.py so there is one
definition of each rather than two that can drift.
- build_database.py drops --out-index and index generation, and its
proof/status derivation is factored into resolve_proof() shared with
retrieval.
- database/goodman-1.6.index.jsonl deleted.
Also fixes a retrieval bug the removal surfaced: the tokenizer only
matched ASCII identifiers, so a goal that is pure algebra
("a * (b + c) = a * b + a * c") tokenized to nothing and retrieved
nothing at all -- it now keeps type and operator symbols, and drops
single-letter variable names, which say nothing when matched. That goal
now correctly finds Proposition 1.6.1, and the identifier-rich goals
still rank their exact premise first by a wide margin (9.3 vs 4.4).
Tests restructured accordingly: the proof gate is now pinned directly on
resolve_proof in both directions, plus new coverage for load_database
deriving status from proof_checked and for the symbol tokenization.
49 tests, 48 green (the one failure remains the pre-existing unrelated
lean_agent import error in test_dataset.py).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
This commit is contained in:
+43
-51
@@ -5,23 +5,21 @@ Theorem/Proposition/Lemma/Corollary claims, turned into real Lean declarations t
|
||||
lean-agent can cite by name while proving something else (the bench targets, which
|
||||
come from the exercises instead -- see formalize_exercises.py).
|
||||
|
||||
Two outputs:
|
||||
Output: an importable Lean file where every database entry is a named declaration,
|
||||
so a proof can cite goodman_1_6_20 by name. An entry is emitted as a real
|
||||
`theorem ... := by <proof>` once its proof has been verified, and otherwise as an
|
||||
`axiom` -- asserted from the textbook, which is what the book itself does for its
|
||||
own body results before an exercise asks you to use them. This is deliberate and
|
||||
has a real cost: anything proven *through* an axiom inherits it, which is exactly
|
||||
why the verifier's `#print axioms` check matters. A statement that only elaborates
|
||||
is not the same as a statement that is true, and a garbled extraction asserted as
|
||||
an axiom can make downstream "proofs" worthless -- so every axiom here needs to have
|
||||
been read by a human against the book, and `--require-checked` refuses to emit any
|
||||
entry not marked as having passed check_statements.py.
|
||||
|
||||
1. An importable Lean file where every database entry is a named declaration. An
|
||||
entry is emitted as a real `theorem ... := by <proof>` when a lean-bench run has
|
||||
actually proven it (status "proved"), and otherwise as an `axiom` -- asserted from
|
||||
the textbook, which is what the book itself does for its own body results before
|
||||
an exercise asks you to use them. This is deliberate and has a real cost: anything
|
||||
proven *through* an axiom inherits it, which is exactly why the verifier's
|
||||
`#print axioms` check matters. A statement that only elaborates is not the same as
|
||||
a statement that is true, and a garbled extraction asserted as an axiom can make
|
||||
downstream "proofs" worthless -- so every axiom here needs to have been read by a
|
||||
human against the book, and `--require-checked` refuses to emit any entry not
|
||||
marked as having passed check_statements.py.
|
||||
|
||||
2. A search index (JSONL) carrying each entry's declaration name, statement, the
|
||||
Mathlib identifiers it mentions, and its status -- what search_database.py ranks
|
||||
against the current goals at each step of a proof.
|
||||
There is deliberately no generated search index: search_database.py reads this same
|
||||
source file and derives the declaration name, identifiers and status itself, so
|
||||
there is only ever one copy of the data and nothing to keep in sync.
|
||||
|
||||
Pass a lean-bench run's results.jsonl with --results to upgrade the entries it
|
||||
proved from axiom to a real theorem carrying the exact proof that run verified.
|
||||
@@ -53,8 +51,26 @@ def split_prelude(prelude):
|
||||
return imports, '\n'.join(other).strip()
|
||||
|
||||
|
||||
def resolve_proof(entry, result=None):
|
||||
"""(proof, status) for one entry.
|
||||
|
||||
A proof only counts once Lean has verified it: check_proofs.py sets
|
||||
proof_checked on the entry, or a lean-bench run reports success. An unverified
|
||||
proof leaves the entry asserted, reported as proof_pending so it is visible
|
||||
without being trusted."""
|
||||
run_proof = result['attempts'][-1]['proof'] if (
|
||||
result and result.get('success') and result.get('attempts')) else None
|
||||
own_proof = entry.get('proof') if entry.get('proof_checked') else None
|
||||
proof = run_proof or own_proof
|
||||
if proof:
|
||||
return proof, 'proved'
|
||||
if entry.get('proof'):
|
||||
return None, 'proof_pending'
|
||||
return None, 'asserted'
|
||||
|
||||
|
||||
def statement_identifiers(statement):
|
||||
"""Identifiers mentioned in a statement, for the search index."""
|
||||
"""Identifiers mentioned in a statement, for premise retrieval."""
|
||||
found = []
|
||||
for match in IDENT_RE.findall(statement or ''):
|
||||
if match in STOPWORDS or match in found or len(match) < 2:
|
||||
@@ -89,7 +105,7 @@ def build(entries, results=None, require_checked=False):
|
||||
lines.append(block)
|
||||
lines.append('')
|
||||
|
||||
index, emitted, skipped = [], [], []
|
||||
emitted, skipped = [], []
|
||||
for entry in entries:
|
||||
entry_id = entry['id']
|
||||
if require_checked and not entry.get('elaborates'):
|
||||
@@ -97,17 +113,9 @@ def build(entries, results=None, require_checked=False):
|
||||
continue
|
||||
|
||||
name = decl_name(entry_id)
|
||||
result = results.get(entry_id)
|
||||
# A proof can come from a lean-bench run, or be recorded on the entry itself
|
||||
# (hand-written, or authored by an agent outside a bench run). Either way it
|
||||
# only counts once check_proofs.py has actually compiled it: an unverified
|
||||
# proof stays an axiom, so the library never claims more than Lean confirmed.
|
||||
run_proof = result['attempts'][-1]['proof'] if (
|
||||
result and result.get('success') and result.get('attempts')) else None
|
||||
own_proof = entry.get('proof') if entry.get('proof_checked') else None
|
||||
proof = run_proof or own_proof
|
||||
proved = bool(proof)
|
||||
proof_pending = bool(not proved and entry.get('proof'))
|
||||
proof, status = resolve_proof(entry, results.get(entry_id))
|
||||
proved = status == 'proved'
|
||||
proof_pending = status == 'proof_pending'
|
||||
provenance = entry.get('source_label', entry_id)
|
||||
|
||||
doc = f'/-- {provenance} (Goodman, Algebra: Abstract and Concrete).'
|
||||
@@ -129,21 +137,9 @@ def build(entries, results=None, require_checked=False):
|
||||
lines.append(f'axiom {name} : {entry["statement"]}')
|
||||
lines.append('')
|
||||
|
||||
index.append({
|
||||
"name": name,
|
||||
"id": entry_id,
|
||||
"source_label": entry.get('source_label'),
|
||||
"kind": entry.get('kind'),
|
||||
"status": "proved" if proved else ("proof_pending" if proof_pending else "asserted"),
|
||||
"proof": proof if proved else entry.get('proof'),
|
||||
"statement": entry['statement'],
|
||||
"natural_language": entry.get('natural_language', '').split('[fix note:')[0].strip(),
|
||||
"identifiers": statement_identifiers(entry['statement']),
|
||||
"requires": entry.get('requires', []),
|
||||
})
|
||||
emitted.append(entry_id)
|
||||
|
||||
return '\n'.join(lines), index, emitted, skipped
|
||||
return '\n'.join(lines), emitted, skipped
|
||||
|
||||
|
||||
def main():
|
||||
@@ -151,27 +147,23 @@ def main():
|
||||
parser.add_argument('database', type=Path, help='JSONL of main-content statements, e.g. database/goodman-1.6.jsonl')
|
||||
parser.add_argument('--results', type=Path, help="a lean-bench run's results.jsonl; proven entries become real theorems")
|
||||
parser.add_argument('--out-lean', type=Path, required=True, help='importable Lean file of citable premises')
|
||||
parser.add_argument('--out-index', type=Path, required=True, help='JSONL search index for search_database.py')
|
||||
parser.add_argument('--require-checked', action='store_true',
|
||||
help='refuse entries not marked "elaborates": true (i.e. not yet verified by check_statements.py)')
|
||||
args = parser.parse_args()
|
||||
|
||||
entries = load_jsonl(args.database)
|
||||
results = {r['id']: r for r in load_jsonl(args.results)} if args.results else None
|
||||
text, index, emitted, skipped = build(entries, results, args.require_checked)
|
||||
text, emitted, skipped = build(entries, results, args.require_checked)
|
||||
|
||||
args.out_lean.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out_lean.write_text(text + '\n', encoding='utf-8')
|
||||
args.out_index.parent.mkdir(parents=True, exist_ok=True)
|
||||
with args.out_index.open('w', encoding='utf-8') as stream:
|
||||
for record in index:
|
||||
stream.write(json.dumps(record, ensure_ascii=False) + '\n')
|
||||
|
||||
proved = sum(1 for r in index if r['status'] == 'proved')
|
||||
pending = sum(1 for r in index if r['status'] == 'proof_pending')
|
||||
by_id = {e['id']: e for e in entries}
|
||||
statuses = [resolve_proof(by_id[i], (results or {}).get(i))[1] for i in emitted]
|
||||
proved = statuses.count('proved')
|
||||
pending = statuses.count('proof_pending')
|
||||
print(f'{len(emitted)} premises -> {args.out_lean} ({proved} proven, '
|
||||
f'{len(emitted) - proved} asserted as axioms, of which {pending} have an unverified proof)')
|
||||
print(f'search index -> {args.out_index}')
|
||||
if skipped:
|
||||
print(f'{len(skipped)} skipped for not being elaboration-checked: {", ".join(skipped)}')
|
||||
|
||||
|
||||
Reference in New Issue
Block a user