Rebuild lean-bench: pages in, verified indexed Lean out
ci / test (push) Successful in 1m22s
ci / lean-project (push) Successful in 6s
docker / image (push) Failing after 3h0m9s

Replaces the previous tree entirely. The pipeline is four separable pieces, each
testable on its own:

1. leanbench/llm.py      provider-neutral OpenAI-compatible connector, stdlib only,
                        with retries, JSON coercion and a scripted client so the
                        rest of the code is testable without a network.
2. leanbench/detect.py   per-page detection of formalizable mathematics. Case 1 is
                        mathematics asserted as fact (definition/theorem/lemma/...,
                        deliberately not sub-classified); case 2 is a problem left
                        open to the reader. Hits come back as spans -- start page
                        and index, end page and index -- so several per page is
                        normal. A hit cut off by the page end makes the detector
                        re-read that page together with the next, repeatedly, so an
                        item spread over three pages is found whole.
                        Offsets come from verbatim quotes resolved by
                        leanbench/anchors.py (exact -> whitespace-insensitive ->
                        fuzzy), because models quote well and count badly.
3. leanbench/formalize.py + leanbench/verify.py
                        each span becomes one Lean 4 declaration, judged by the real
                        compiler: full elaboration for case 1, statement legality
                        only for case 2, which has no proof to check. Lean's own
                        diagnostics feed back for a repair attempt. "verified" (Lean
                        accepted it) and "proved" (accepted with no sorry) are kept
                        strictly apart.
4. leanbench/db.py       indexed SQLite plus JSONL export. Every item is indexed by
                        the types of its inputs, or by its own type when it is not a
                        proof, and records the source book (sha256) and the page it
                        starts on -- so `drop --source H --pages A-B` followed by a
                        re-ingest redoes exactly those pages and nothing else.

Infrastructure:

- lean/            Lake project pinned to Mathlib.
- docker/          image carrying Lean 4 and a prebuilt Mathlib. The Mathlib
                   revision and the toolchain are resolved against each other at
                   build time rather than both hardcoded, so the build does not rot.
                   docker/smoke.py runs the whole pipeline inside the image against
                   the real compiler, with the LLM scripted.
- .gitea/workflows ci (lint + tests in a venv, since the runner image's Python is
                   Debian-managed and refuses installs into itself), docker (build,
                   verify, optionally push), ingest (submit a book by hand).

The docker workflow is not a throwaway build: it leaves the image, the layer cache,
an idle `lean-bench` container and the `lean-bench-data` / `lean-bench-books`
volumes in place on the runner, and never prunes, so the environment is there to be
used afterwards. Pushing to the registry is optional and cannot fail the build --
Gitea's container registry does not accept the built-in Actions token, so that needs
a REGISTRY_TOKEN secret with write:package.

58 tests; they need neither network nor Lean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ci5T3gWDhhmqDec3e1XtYZ
This commit is contained in:
2026-09-19 07:16:37 +00:00
commit b9e37952e6
38 changed files with 4548 additions and 0 deletions
+54
View File
@@ -0,0 +1,54 @@
from __future__ import annotations
from leanbench.anchors import find, resolve_span
TEXT = (
"Theorem 1.6.2. Let a, b and c be integers. If a divides b and a divides c,\n"
"then a divides b + c.\n\nProof. Write b = a k."
)
def test_exact_match() -> None:
match = find(TEXT, "Theorem 1.6.2.")
assert match is not None and match.method == "exact"
assert TEXT[match.start : match.end] == "Theorem 1.6.2."
def test_whitespace_insensitive_match_across_a_line_break() -> None:
match = find(TEXT, "a divides c, then a divides b + c.")
assert match is not None
assert match.method == "whitespace"
assert TEXT[match.start : match.end].replace("\n", " ") == "a divides c, then a divides b + c."
def test_fuzzy_match_survives_a_typo() -> None:
match = find(TEXT, "Let a, b and c be integres. If a divides b")
assert match is not None
assert match.method == "fuzzy"
assert "integers" in TEXT[match.start : match.end]
def test_missing_anchor_is_reported_not_guessed() -> None:
assert find(TEXT, "nothing like this appears anywhere in the page") is None
def test_resolve_span_covers_start_to_end() -> None:
span = resolve_span(TEXT, "Theorem 1.6.2.", "a divides b + c.")
assert span is not None and span.resolved
assert TEXT[span.start : span.end].startswith("Theorem 1.6.2.")
assert TEXT[span.start : span.end].endswith("a divides b + c.")
def test_resolve_span_falls_back_when_the_end_quote_is_wrong() -> None:
span = resolve_span(TEXT, "Theorem 1.6.2.", "this ending was invented", fallback_length=40)
assert span is not None
assert "truncated" in span.method
assert span.end - span.start == 40
def test_resolve_span_searches_forward_from_the_cursor() -> None:
text = "alpha beta alpha gamma"
first = resolve_span(text, "alpha", "beta")
assert first is not None and first.start == 0
second = resolve_span(text, "alpha", "gamma", search_from=first.end)
assert second is not None and second.start == text.rindex("alpha")