claude-bot
04d827b3b3
Restructures the textbook pipeline around the distinction the body-derived dataset had been blurring: a bench should be actual unsolved problems, and a reusable premise library should only ever contain statements that have really been proven, the way Mathlib itself works. - library_candidates/goodman-1.6.jsonl (renamed from datasets/goodman-1.6.jsonl): the book's own Theorem/Proposition/Lemma/Corollary claims from the main text. Not bench targets -- premises waiting to be proven once via a real lean-bench run, then assembled into the reusable library by build_library.py (which already only keeps success:true entries with their checked proof, unchanged by this commit). - datasets/goodman-1.6-exercises.jsonl (new): the actual bench, sourced from the book's Exercises 1.6 block via two new scripts, textbook/extract_exercises.py (slices the Exercises block, lists its numbered items, drops a known false-positive class where a wrapped cross-reference like "...Proposition\n1.6.19." looks like a real exercise heading) and textbook/formalize_exercises.py (LLM step, given both the exercises and body text together since many exercises say "Prove Proposition N.N.N" instead of restating the claim). - Both formalize.py and formalize_exercises.py's validate() now enforce that every checklist item is covered by at least one output entry (included or explicitly excluded with a reason); formalize_exercises.py did not have this yet and a first real run silently dropped 7 of 15 checklist items with no trace before this was added. - All 12 model-produced exercise entries needed real fixes before commit, the same review pass applied to the body pilot: 3 exercises literally ask to prove an already-stated body claim (Exercise 1.6.1 -> Proposition 1.6.7, 1.6.2 -> Proposition 1.6.9, 1.6.7 -> Proposition 1.6.18, 1.6.9 -> Corollary 1.6.20, 1.6.15 -> Lemma 1.6.26) and now reuse that claim's own already-checked statement verbatim instead of an independently re-derived (and in one case strictly weaker) restatement; two entries (1.6.2, 1.6.11) had literal math prose like "I(m, n)" or "gcd(n1,...,nk)" leaked into the statement field, which is not valid Lean; several had untyped/ambiguous gcd or Prime usage fixed the same way the earlier body fixes were (Nat.Prime, IsCoprime, explicit casts, IsGreatest instead of max on a Set); and Exercise 1.6.12 part (a) turned out to be a discussion question whose unconditional-identity reading is actually false (6, 10, 15 are jointly coprime but not pairwise coprime), so it was moved to the excluded file with a reason instead of committed as a false theorem. - Definitions can now be real, compiled Lean, not only JSON metadata: Definition 1.6.22 (gcd of a list of integers, previously a null placeholder since no single Mathlib constant matches it) now has a real hand-written `def IsGCDOfList`, and textbook/build_definitions.py compiles any manifest's non-null lean_code entries into lib/Definitions.lean. goodman-1-6-24 (library) and the new goodman-1-6-ex-11-b/c, goodman-1-6-ex-12 (bench) all now use IsGCDOfList instead of repeating an inline existential. - textbook/README.md rewritten around the two-track design and records how this is meant to connect to lean-agent's eventual goal-directed proof search against the growing library (not yet built). - New tests for extract_exercises.py and build_definitions.py; full local suite still green (18/19, the one failure is the pre-existing, unrelated lean_agent import error in test_dataset.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
50 lines
1.9 KiB
Python
50 lines
1.9 KiB
Python
import unittest
|
|
from textbook.extract_exercises import extract_exercises, numbered_items
|
|
|
|
FIXTURE = """\
|
|
1.6. Divisibility in the Integers
|
|
body text
|
|
Proof. Exercise 1.6.15. n
|
|
|
|
Exercises 1.6
|
|
|
|
1.6.1. Complete the following sketch.
|
|
(a) part a text
|
|
1.6.2. Prove Proposition 1.6.9.
|
|
1.6.3. Suppose that a natural number p > 1 has some property. This is the
|
|
converse of Proposition
|
|
1.6.19.
|
|
|
|
1.7. Modular Arithmetic
|
|
next section text
|
|
"""
|
|
|
|
|
|
class ExtractExercisesTests(unittest.TestCase):
|
|
def test_extract_exercises_slices_between_headings(self):
|
|
block = extract_exercises(FIXTURE, 'Exercises 1.6', '1.7. Modular Arithmetic')
|
|
self.assertIn('1.6.1. Complete', block)
|
|
self.assertNotIn('next section text', block)
|
|
|
|
def test_extract_exercises_missing_heading_raises(self):
|
|
with self.assertRaises(ValueError):
|
|
extract_exercises(FIXTURE, 'Exercises 1.6', 'Exercises 9.9')
|
|
|
|
def test_numbered_items_orders_and_dedupes(self):
|
|
block = extract_exercises(FIXTURE, 'Exercises 1.6', '1.7. Modular Arithmetic')
|
|
items, _dropped = numbered_items(block, '1.6.')
|
|
self.assertEqual([i['number'] for i in items], ['1.6.1', '1.6.2', '1.6.3'])
|
|
|
|
def test_numbered_items_drops_wrapped_cross_reference(self):
|
|
# '1.6.19.' alone on its own line, right after a big jump from 1.6.3, with no
|
|
# text on the same line, is a wrapped cross-reference ("...Proposition\n1.6.19."),
|
|
# not a real exercise 16 numbers ahead of the last real one.
|
|
block = extract_exercises(FIXTURE, 'Exercises 1.6', '1.7. Modular Arithmetic')
|
|
items, dropped = numbered_items(block, '1.6.', max_gap=3)
|
|
self.assertNotIn('1.6.19', [i['number'] for i in items])
|
|
self.assertEqual([d['number'] for d in dropped], ['1.6.19'])
|
|
|
|
|
|
if __name__ == '__main__':
|
|
unittest.main()
|