Split pipeline into real bench (exercises) vs library candidates (body)
Restructures the textbook pipeline around the distinction the body-derived dataset had been blurring: a bench should be actual unsolved problems, and a reusable premise library should only ever contain statements that have really been proven, the way Mathlib itself works. - library_candidates/goodman-1.6.jsonl (renamed from datasets/goodman-1.6.jsonl): the book's own Theorem/Proposition/Lemma/Corollary claims from the main text. Not bench targets -- premises waiting to be proven once via a real lean-bench run, then assembled into the reusable library by build_library.py (which already only keeps success:true entries with their checked proof, unchanged by this commit). - datasets/goodman-1.6-exercises.jsonl (new): the actual bench, sourced from the book's Exercises 1.6 block via two new scripts, textbook/extract_exercises.py (slices the Exercises block, lists its numbered items, drops a known false-positive class where a wrapped cross-reference like "...Proposition\n1.6.19." looks like a real exercise heading) and textbook/formalize_exercises.py (LLM step, given both the exercises and body text together since many exercises say "Prove Proposition N.N.N" instead of restating the claim). - Both formalize.py and formalize_exercises.py's validate() now enforce that every checklist item is covered by at least one output entry (included or explicitly excluded with a reason); formalize_exercises.py did not have this yet and a first real run silently dropped 7 of 15 checklist items with no trace before this was added. - All 12 model-produced exercise entries needed real fixes before commit, the same review pass applied to the body pilot: 3 exercises literally ask to prove an already-stated body claim (Exercise 1.6.1 -> Proposition 1.6.7, 1.6.2 -> Proposition 1.6.9, 1.6.7 -> Proposition 1.6.18, 1.6.9 -> Corollary 1.6.20, 1.6.15 -> Lemma 1.6.26) and now reuse that claim's own already-checked statement verbatim instead of an independently re-derived (and in one case strictly weaker) restatement; two entries (1.6.2, 1.6.11) had literal math prose like "I(m, n)" or "gcd(n1,...,nk)" leaked into the statement field, which is not valid Lean; several had untyped/ambiguous gcd or Prime usage fixed the same way the earlier body fixes were (Nat.Prime, IsCoprime, explicit casts, IsGreatest instead of max on a Set); and Exercise 1.6.12 part (a) turned out to be a discussion question whose unconditional-identity reading is actually false (6, 10, 15 are jointly coprime but not pairwise coprime), so it was moved to the excluded file with a reason instead of committed as a false theorem. - Definitions can now be real, compiled Lean, not only JSON metadata: Definition 1.6.22 (gcd of a list of integers, previously a null placeholder since no single Mathlib constant matches it) now has a real hand-written `def IsGCDOfList`, and textbook/build_definitions.py compiles any manifest's non-null lean_code entries into lib/Definitions.lean. goodman-1-6-24 (library) and the new goodman-1-6-ex-11-b/c, goodman-1-6-ex-12 (bench) all now use IsGCDOfList instead of repeating an inline existential. - textbook/README.md rewritten around the two-track design and records how this is meant to connect to lean-agent's eventual goal-directed proof search against the growing library (not yet built). - New tests for extract_exercises.py and build_definitions.py; full local suite still green (18/19, the one failure is the pre-existing, unrelated lean_agent import error in test_dataset.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vbLFFJiHSZqGZHEkjvG5x
This commit is contained in:
@@ -0,0 +1,101 @@
|
||||
"""Slice the 'Exercises N.N' block out of a pdftotext -layout dump and list its
|
||||
numbered problems, as input for formalize_exercises.py.
|
||||
|
||||
Mirrors extract_section.py but for the exercise list at the end of a section, which
|
||||
uses bare 'N.N.N.' numbering at the start of a line (no Theorem/Proposition keyword)
|
||||
instead of labeled Theorem/Proposition/Lemma/Corollary/Definition entries. Exercise
|
||||
numbers are their own independent sequence (Exercise 1.6.5 has nothing to do with
|
||||
Proposition/Theorem/Lemma/Corollary/Definition 1.6.5 in the body) -- keep their ids
|
||||
namespaced separately downstream so the two never collide.
|
||||
|
||||
A known false-positive source: a sentence like "...converse of Proposition\n1.6.19."
|
||||
can wrap so the bare number lands alone at the start of a line, which looks
|
||||
identical to a real exercise heading. numbered_items() drops any item whose number
|
||||
jumps ahead of the running sequence by more than --max-gap AND has no text on its
|
||||
own line (a real item almost always has problem text right after the number; a
|
||||
wrapped cross-reference is just the number and a period). Dropped candidates are
|
||||
still reported, never silently discarded.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ITEM_RE = re.compile(r'^(\d+\.\d+\.\d+)\.\s*(.*)$')
|
||||
|
||||
|
||||
def find_heading_line(lines, heading):
|
||||
for i, line in enumerate(lines):
|
||||
if line.strip() == heading:
|
||||
return i
|
||||
raise ValueError(f'Heading not found: {heading!r}')
|
||||
|
||||
|
||||
def extract_exercises(full_text, start_heading, end_heading):
|
||||
lines = full_text.split('\n')
|
||||
start = find_heading_line(lines, start_heading)
|
||||
end = find_heading_line(lines, end_heading) if end_heading else len(lines)
|
||||
if end <= start:
|
||||
raise ValueError('end_heading must come after start_heading')
|
||||
return '\n'.join(lines[start:end])
|
||||
|
||||
|
||||
def numbered_items(exercises_text, section_prefix, max_gap=3):
|
||||
"""Unique numbered items in exercises_text whose number starts with
|
||||
section_prefix (e.g. '1.6.'), keeping only the first occurrence of each number
|
||||
(a running page header can repeat text near a page break). Returned in book
|
||||
order (numeric, not lexical). See module docstring for the wrapped-cross-
|
||||
reference false-positive filter; dropped candidates are returned separately so
|
||||
callers can report them instead of silently losing items."""
|
||||
seen = {}
|
||||
for line in exercises_text.split('\n'):
|
||||
line = line.strip()
|
||||
m = ITEM_RE.match(line)
|
||||
if not m:
|
||||
continue
|
||||
number, rest = m.group(1), m.group(2)
|
||||
if not number.startswith(section_prefix) or number in seen:
|
||||
continue
|
||||
seen[number] = {"number": number, "first_line": line[:120], "rest_empty": not rest.strip()}
|
||||
|
||||
def sort_key(item):
|
||||
return tuple(int(p) for p in item['number'].split('.'))
|
||||
|
||||
ordered = sorted(seen.values(), key=sort_key)
|
||||
kept, dropped = [], []
|
||||
last_n = 0
|
||||
for item in ordered:
|
||||
n = int(item['number'].split('.')[-1])
|
||||
if item['rest_empty'] and last_n and (n - last_n) > max_gap:
|
||||
dropped.append(item)
|
||||
continue
|
||||
kept.append({"number": item['number'], "first_line": item['first_line']})
|
||||
last_n = n
|
||||
return kept, dropped
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Slice one "Exercises N.N" block out of a pdftotext -layout dump')
|
||||
parser.add_argument('layout_txt', type=Path)
|
||||
parser.add_argument('--start-heading', required=True, help="e.g. 'Exercises 1.6'")
|
||||
parser.add_argument('--end-heading', help="e.g. '1.7. Modular Arithmetic'; omit for end of file")
|
||||
parser.add_argument('--section-prefix', required=True, help="e.g. '1.6.'")
|
||||
parser.add_argument('--max-gap', type=int, default=3, help='see module docstring')
|
||||
parser.add_argument('--out-text', type=Path, required=True)
|
||||
parser.add_argument('--out-index', type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
full_text = args.layout_txt.read_text(encoding='utf-8')
|
||||
exercises_text = extract_exercises(full_text, args.start_heading, args.end_heading)
|
||||
args.out_text.write_text(exercises_text, encoding='utf-8')
|
||||
index, dropped = numbered_items(exercises_text, args.section_prefix, args.max_gap)
|
||||
args.out_index.write_text(json.dumps(index, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
|
||||
print(f'{len(index)} numbered exercises -> {args.out_index}')
|
||||
for item in dropped:
|
||||
print(f" dropped as likely a wrapped cross-reference, not a real exercise: {item['number']}"
|
||||
f" ({item['first_line']!r})", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user