Rebuild lean-bench: pages in, verified indexed Lean out

Replaces the previous tree entirely. The pipeline is four separable pieces, each testable on its own: 1. leanbench/llm.py provider-neutral OpenAI-compatible connector, stdlib only, with retries, JSON coercion and a scripted client so the rest of the code is testable without a network. 2. leanbench/detect.py per-page detection of formalizable mathematics. Case 1 is mathematics asserted as fact (definition/theorem/lemma/..., deliberately not sub-classified); case 2 is a problem left open to the reader. Hits come back as spans -- start page and index, end page and index -- so several per page is normal. A hit cut off by the page end makes the detector re-read that page together with the next, repeatedly, so an item spread over three pages is found whole. Offsets come from verbatim quotes resolved by leanbench/anchors.py (exact -> whitespace-insensitive -> fuzzy), because models quote well and count badly. 3. leanbench/formalize.py + leanbench/verify.py each span becomes one Lean 4 declaration, judged by the real compiler: full elaboration for case 1, statement legality only for case 2, which has no proof to check. Lean's own diagnostics feed back for a repair attempt. "verified" (Lean accepted it) and "proved" (accepted with no sorry) are kept strictly apart. 4. leanbench/db.py indexed SQLite plus JSONL export. Every item is indexed by the types of its inputs, or by its own type when it is not a proof, and records the source book (sha256) and the page it starts on -- so `drop --source H --pages A-B` followed by a re-ingest redoes exactly those pages and nothing else. Infrastructure: - lean/ Lake project pinned to Mathlib. - docker/ image carrying Lean 4 and a prebuilt Mathlib. The Mathlib revision and the toolchain are resolved against each other at build time rather than both hardcoded, so the build does not rot. docker/smoke.py runs the whole pipeline inside the image against the real compiler, with the LLM scripted. - .gitea/workflows ci (lint + tests in a venv, since the runner image's Python is Debian-managed and refuses installs into itself), docker (build, verify, optionally push), ingest (submit a book by hand). The docker workflow is not a throwaway build: it leaves the image, the layer cache, an idle `lean-bench` container and the `lean-bench-data` / `lean-bench-books` volumes in place on the runner, and never prunes, so the environment is there to be used afterwards. Pushing to the registry is optional and cannot fail the build -- Gitea's container registry does not accept the built-in Actions token, so that needs a REGISTRY_TOKEN secret with write:package. 58 tests; they need neither network nor Lean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ci5T3gWDhhmqDec3e1XtYZ
2026-09-19 07:16:37 +00:00
commit b9e37952e6
38 changed files with 4548 additions and 0 deletions
@@ -0,0 +1,134 @@
+"""End-to-end smoke test for the image, with the real Lean compiler attached.
+
+The LLM is scripted (the image has no API key and CI should not need one), but
+everything else is real: pages are extracted, spans are resolved to page/index,
+Lean elaborates the generated code, and the items land in an indexed database.
+
+Run it against a built image without mounting anything:
+
+    docker run --rm -i --entrypoint python3 lean-bench:latest - < docker/smoke.py
+"""
+from __future__ import annotations
+
+import json
+import os
+import sys
+import tempfile
+
+from leanbench.config import Settings
+from leanbench.db import Database
+from leanbench.llm import ScriptedClient
+from leanbench.pipeline import build_checker, ingest
+
+BOOK_PAGE_ONE = "Theorem 1. For every natural number n, n + 0 = n.\n"
+BOOK_PAGE_TWO = "Exercise 2. Show that 2 + 2 = 4.\n"
+
+
+def detection(items: list[dict]) -> str:
+    return json.dumps({"items": items})
+
+
+def main() -> int:
+    workdir = tempfile.mkdtemp(prefix="leanbench-smoke-")
+    book = os.path.join(workdir, "book.txt")
+    with open(book, "w", encoding="utf-8") as stream:
+        stream.write(BOOK_PAGE_ONE + "\f" + BOOK_PAGE_TWO)
+
+    client = ScriptedClient(
+        [
+            detection(
+                [
+                    {
+                        "kind": "statement",
+                        "label": "Theorem 1",
+                        "summary": "adding zero changes nothing",
+                        "start_quote": "Theorem 1. For every natural number n",
+                        "end_quote": "n + 0 = n.",
+                        "continues_past_end": False,
+                    }
+                ]
+            ),
+            detection(
+                [
+                    {
+                        "kind": "problem",
+                        "label": "Exercise 2",
+                        "summary": "two plus two",
+                        "start_quote": "Exercise 2. Show that",
+                        "end_quote": "2 + 2 = 4.",
+                        "continues_past_end": False,
+                    }
+                ]
+            ),
+            json.dumps(
+                {
+                    "skip": False,
+                    "imports": ["Mathlib"],
+                    "name": "add_zero_nat",
+                    "lean_code": "theorem add_zero_nat (n : Nat) : n + 0 = n := by simp",
+                    "has_proof": True,
+                    "informal": "n + 0 = n",
+                    "index_types": ["Nat"],
+                    "confidence": 1.0,
+                }
+            ),
+            json.dumps(
+                {
+                    "skip": False,
+                    "imports": ["Mathlib"],
+                    "name": "two_add_two",
+                    "lean_code": "theorem two_add_two : (2 : Nat) + 2 = 4 := by sorry",
+                    "has_proof": False,
+                    "informal": "2 + 2 = 4",
+                    "index_types": ["Nat"],
+                    "confidence": 1.0,
+                }
+            ),
+        ]
+    )
+
+    settings = Settings.from_env(dotenv=None)
+    settings.database = os.path.join(workdir, "smoke.sqlite")
+    settings.repair_attempts = 0
+
+    checker = build_checker(settings)
+    if not getattr(checker, "available", lambda: False)():
+        print("FAIL: this image has no working Lean; verification would be skipped")
+        return 1
+
+    database = Database(settings.database)
+    result = ingest(
+        book,
+        settings=settings,
+        client=client,
+        database=database,
+        checker=checker,
+        title="smoke",
+    )
+    summary = result.summary()
+    print(json.dumps(summary, indent=2))
+
+    failures = []
+    if summary["detected"] != 2:
+        failures.append(f"expected 2 detections, got {summary['detected']}")
+    if summary["verified"] != 2:
+        failures.append(f"Lean should have accepted both, got {summary['verified']}")
+    if summary["proved"] != 1:
+        failures.append(f"exactly one item carries a real proof, got {summary['proved']}")
+    if not database.search(types=["Nat"]):
+        failures.append("the type index is empty")
+    problems = database.search(kind="problem")
+    if len(problems) != 1 or problems[0]["verify_mode"] != "statement":
+        failures.append("the open problem should be checked as a statement only")
+    database.close()
+
+    for line in failures:
+        print("FAIL:", line)
+    if failures:
+        return 1
+    print("pipeline + lean end to end: ok")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())