Rebuild lean-bench: pages in, verified indexed Lean out
ci / test (push) Failing after 12s
ci / lean-project (push) Successful in 5s
docker / image (push) Failing after 5s

Replaces the previous tree entirely. The pipeline is four separable pieces, each
testable on its own:

1. leanbench/llm.py      provider-neutral OpenAI-compatible connector, stdlib only,
                        with retries, JSON coercion and a scripted client so the
                        rest of the code is testable without a network.
2. leanbench/detect.py   per-page detection of formalizable mathematics. Case 1 is
                        mathematics asserted as fact (definition/theorem/lemma/...,
                        deliberately not sub-classified); case 2 is a problem left
                        open to the reader. Hits come back as spans -- start page
                        and index, end page and index -- so several per page is
                        normal. A hit cut off by the page end makes the detector
                        re-read that page together with the next, repeatedly, so an
                        item spread over three pages is found whole.
                        Offsets come from verbatim quotes resolved by
                        leanbench/anchors.py (exact -> whitespace-insensitive ->
                        fuzzy), because models quote well and count badly.
3. leanbench/formalize.py + leanbench/verify.py
                        each span becomes one Lean 4 declaration, judged by the real
                        compiler: full elaboration for case 1, statement legality
                        only for case 2, which has no proof to check. Lean's own
                        diagnostics feed back for a repair attempt. "verified" (Lean
                        accepted it) and "proved" (accepted with no sorry) are kept
                        strictly apart.
4. leanbench/db.py       indexed SQLite plus JSONL export. Every item is indexed by
                        the types of its inputs, or by its own type when it is not a
                        proof, and records the source book (sha256) and the page it
                        starts on -- so `drop --source H --pages A-B` followed by a
                        re-ingest redoes exactly those pages and nothing else.

Also: lean/ Lake project pinned to Mathlib, docker/ image carrying Lean 4 and a
prebuilt Mathlib (the Mathlib revision and the toolchain are resolved against each
other at build time rather than both hardcoded), and .gitea/workflows for tests,
image build and push to the Gitea registry, and a manual "submit a book" run.

58 tests; they need neither network nor Lean.
This commit is contained in:
Zayden-Jung
2026-09-19 07:05:52 +00:00
commit d3555d9a52
37 changed files with 4251 additions and 0 deletions
+46
View File
@@ -0,0 +1,46 @@
from __future__ import annotations
from leanbench.document import Document, Page, PageWindow, normalise_page_text
def test_pages_are_numbered_from_one(two_page_book: Document) -> None:
assert len(two_page_book) == 2
assert [p.number for p in two_page_book.pages] == [1, 2]
assert two_page_book.page(1).text.startswith("Chapter 1")
def test_window_locate_maps_back_to_the_right_page() -> None:
window = PageWindow([Page(7, "abcdef"), Page(8, "ghij")])
assert window.text == "abcdef\nghij"
assert window.locate(0) == (7, 0)
assert window.locate(5) == (7, 5)
assert window.locate(6) == (7, 6) # end of page 7
assert window.locate(7) == (8, 0) # first char of page 8
assert window.locate(10) == (8, 3)
assert window.locate(999) == (8, 4) # clamped
def test_window_globalise_is_the_inverse() -> None:
window = PageWindow([Page(1, "hello"), Page(2, "world")])
for page, offset in [(1, 0), (1, 4), (2, 0), (2, 4)]:
assert window.locate(window.globalise(page, offset)) == (page, offset)
def test_document_slice_reconstructs_a_cross_page_span(two_page_book: Document) -> None:
page_one = two_page_book.page(1).text
start = page_one.index("Theorem 1.6.3")
end = two_page_book.page(2).text.index("has a greatest")
text = two_page_book.slice(1, start, 2, end)
assert text.startswith("Theorem 1.6.3")
assert text.rstrip().endswith("divisors of a and b")
def test_normalisation_is_stable_and_gentle() -> None:
text = "first line \nsecond"
assert normalise_page_text(text) == "first line\nsecond"
assert normalise_page_text(normalise_page_text(text)) == normalise_page_text(text)
def test_page_checksum_changes_with_content() -> None:
assert Page(1, "a").checksum != Page(1, "b").checksum
assert Page(1, "a").checksum == Page(9, "a").checksum