claude-bot
69c5b788b9
The database was the missing half: extraction and assembly existed, but nothing turned the book's main-content statements into something an agent could actually cite mid-proof, and nothing searched it. Two new scripts. textbook/build_database.py takes the main-content statements and emits both halves of a usable premise library: - an importable Lean file where every entry is a named declaration, so a proof can cite goodman_1_6_20 by name; - a JSONL search index carrying each entry's name, statement, mentioned identifiers and status. An entry is emitted as an `axiom` (asserted from the textbook) until a real lean-bench run proves it; passing --results upgrades those entries to real theorems carrying the exact verified proof. This mirrors how the book itself works -- its exercises are posed assuming its body results, the same way using Mathlib does not mean re-proving Mathlib. The cost is explicit: anything proven through an axiom inherits it, so a garbled extraction asserted as an axiom makes downstream proofs worthless. Guards: --require-checked refuses entries that have not passed check_statements.py (all 21 section 1.6 entries have, and are now marked accordingly), and the verifier's existing #print axioms check surfaces what a finished proof rests on. textbook/search_database.py is the retrieval step the proof loop calls at each line: given the current goal, it ranks premises by identifier and symbol overlap, weighting rare identifiers far above common ones (IsGCDOfList matching means much more than Int matching), preferring a proven premise over an asserted one, and formats the top few for the prompt. Verified against three real goals: the exactly-applicable premise tops each ranking by a wide margin (11.4 vs 1.6 for the gcd goal, 5.1 vs 3.7 for the prime-divides-product goal). It is a ranking heuristic, not a decision procedure -- only Lean can say whether a premise applies. Replacing score_premise() is the whole job of swapping in embeddings. Also: renamed library_candidates/ to database/ to match what it is; generated lib/Goodman/Chapter01Section06.lean (21 premises, 0 proven, 21 asserted) and database/goodman-1.6.index.jsonl; noted a latent bug in the older build_library.py, which attaches each entry's non-import prelude per-theorem and so would emit a shared `def` twice in one file -- build_database.py hoists and dedupes, and has a test pinning that. 13 new tests, all green (33 total, the one failure remains the pre-existing unrelated lean_agent import error in test_dataset.py). Still not done, and worth being clear about: nothing has been proven yet. The bench targets have not been elaboration-checked, no lean-bench run has happened over either file, and the goal-directed loop that would consume this retrieval does not exist -- lean-bench's solve() is still single-shot. Separately, lean-bench cannot currently run at all in this checkout, because lean_bench/cli.py imports lean_agent at module load and lean-agent is not installed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
70 lines
3.5 KiB
Python
70 lines
3.5 KiB
Python
import unittest
|
|
|
|
from textbook.build_database import build, decl_name, split_prelude, statement_identifiers
|
|
|
|
ENTRIES = [
|
|
{"id": "goodman-1-6-1", "source_label": "Proposition 1.6.1", "kind": "proposition",
|
|
"statement": "\u2200 (a : \u2124), a + 0 = a", "prelude": "import Mathlib",
|
|
"natural_language": "Zero is an additive identity. [fix note: irrelevant]",
|
|
"requires": ["Int.add_zero"], "elaborates": True},
|
|
{"id": "goodman-1-6-24", "source_label": "Proposition 1.6.24", "kind": "proposition",
|
|
"statement": "\u2200 (as : List \u2124), IsGCDOfList 1 as", "prelude": "import Mathlib\ndef IsGCDOfList (d : \u2115) (as : List \u2124) : Prop := True",
|
|
"natural_language": "", "requires": [], "elaborates": True},
|
|
{"id": "goodman-1-6-99", "source_label": "Proposition 1.6.99", "kind": "proposition",
|
|
"statement": "\u2200 (as : List \u2124), IsGCDOfList 2 as", "prelude": "import Mathlib\ndef IsGCDOfList (d : \u2115) (as : List \u2124) : Prop := True",
|
|
"natural_language": "", "requires": [], "elaborates": True},
|
|
]
|
|
|
|
|
|
class BuildDatabaseTests(unittest.TestCase):
|
|
def test_decl_name_is_a_legal_lean_identifier(self):
|
|
self.assertEqual(decl_name('goodman-1-6-2-a'), 'goodman_1_6_2_a')
|
|
|
|
def test_split_prelude_separates_imports_from_defs(self):
|
|
imports, other = split_prelude('import Mathlib\ndef Foo := True')
|
|
self.assertEqual(imports, ['import Mathlib'])
|
|
self.assertEqual(other, 'def Foo := True')
|
|
|
|
def test_unproven_entries_become_axioms(self):
|
|
text, index, emitted, skipped = build(ENTRIES)
|
|
self.assertIn('axiom goodman_1_6_1 :', text)
|
|
self.assertEqual(len(emitted), 3)
|
|
self.assertEqual(skipped, [])
|
|
self.assertTrue(all(r['status'] == 'asserted' for r in index))
|
|
|
|
def test_proven_entries_become_real_theorems_with_their_proof(self):
|
|
results = {"goodman-1-6-1": {"id": "goodman-1-6-1", "success": True,
|
|
"attempts": [{"proof": "simp"}]}}
|
|
text, index, _emitted, _skipped = build(ENTRIES, results)
|
|
self.assertIn('theorem goodman_1_6_1 :', text)
|
|
self.assertNotIn('axiom goodman_1_6_1 :', text)
|
|
self.assertIn(' simp', text)
|
|
statuses = {r['id']: r['status'] for r in index}
|
|
self.assertEqual(statuses['goodman-1-6-1'], 'proved')
|
|
self.assertEqual(statuses['goodman-1-6-24'], 'asserted')
|
|
|
|
def test_shared_def_in_prelude_is_emitted_once_not_per_entry(self):
|
|
# Two entries share the same `def`; declaring it twice in one Lean file is an
|
|
# "already declared" error, so it must be hoisted and deduplicated.
|
|
text, _index, _emitted, _skipped = build(ENTRIES)
|
|
self.assertEqual(text.count('def IsGCDOfList'), 1)
|
|
self.assertEqual(text.count('import Mathlib'), 1)
|
|
|
|
def test_require_checked_skips_unverified_entries(self):
|
|
entries = [dict(ENTRIES[0]), dict(ENTRIES[1], elaborates=False)]
|
|
_text, index, emitted, skipped = build(entries, require_checked=True)
|
|
self.assertEqual(emitted, ['goodman-1-6-1'])
|
|
self.assertEqual(skipped, ['goodman-1-6-24'])
|
|
self.assertEqual(len(index), 1)
|
|
|
|
def test_index_carries_identifiers_and_strips_fix_notes(self):
|
|
_text, index, _emitted, _skipped = build(ENTRIES)
|
|
first = index[0]
|
|
self.assertIn('Int.add_zero', first['requires'])
|
|
self.assertNotIn('fix note', first['natural_language'])
|
|
self.assertNotIn('let', statement_identifiers('let x := 1'))
|
|
|
|
|
|
if __name__ == '__main__':
|
|
unittest.main()
|