claude-bot
b9e37952e6
Replaces the previous tree entirely. The pipeline is four separable pieces, each
testable on its own:
1. leanbench/llm.py provider-neutral OpenAI-compatible connector, stdlib only,
with retries, JSON coercion and a scripted client so the
rest of the code is testable without a network.
2. leanbench/detect.py per-page detection of formalizable mathematics. Case 1 is
mathematics asserted as fact (definition/theorem/lemma/...,
deliberately not sub-classified); case 2 is a problem left
open to the reader. Hits come back as spans -- start page
and index, end page and index -- so several per page is
normal. A hit cut off by the page end makes the detector
re-read that page together with the next, repeatedly, so an
item spread over three pages is found whole.
Offsets come from verbatim quotes resolved by
leanbench/anchors.py (exact -> whitespace-insensitive ->
fuzzy), because models quote well and count badly.
3. leanbench/formalize.py + leanbench/verify.py
each span becomes one Lean 4 declaration, judged by the real
compiler: full elaboration for case 1, statement legality
only for case 2, which has no proof to check. Lean's own
diagnostics feed back for a repair attempt. "verified" (Lean
accepted it) and "proved" (accepted with no sorry) are kept
strictly apart.
4. leanbench/db.py indexed SQLite plus JSONL export. Every item is indexed by
the types of its inputs, or by its own type when it is not a
proof, and records the source book (sha256) and the page it
starts on -- so `drop --source H --pages A-B` followed by a
re-ingest redoes exactly those pages and nothing else.
Infrastructure:
- lean/ Lake project pinned to Mathlib.
- docker/ image carrying Lean 4 and a prebuilt Mathlib. The Mathlib
revision and the toolchain are resolved against each other at
build time rather than both hardcoded, so the build does not rot.
docker/smoke.py runs the whole pipeline inside the image against
the real compiler, with the LLM scripted.
- .gitea/workflows ci (lint + tests in a venv, since the runner image's Python is
Debian-managed and refuses installs into itself), docker (build,
verify, optionally push), ingest (submit a book by hand).
The docker workflow is not a throwaway build: it leaves the image, the layer cache,
an idle `lean-bench` container and the `lean-bench-data` / `lean-bench-books`
volumes in place on the runner, and never prunes, so the environment is there to be
used afterwards. Pushing to the registry is optional and cannot fail the build --
Gitea's container registry does not accept the built-in Actions token, so that needs
a REGISTRY_TOKEN secret with write:package.
58 tests; they need neither network nor Lean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ci5T3gWDhhmqDec3e1XtYZ
80 lines
2.2 KiB
Python
80 lines
2.2 KiB
Python
from __future__ import annotations
|
||
|
||
import json
|
||
from pathlib import Path
|
||
|
||
import pytest
|
||
|
||
from leanbench.document import Document
|
||
from leanbench.verify import MODE_FULL, VerifyResult
|
||
|
||
PAGE_ONE = """Chapter 1. Divisibility
|
||
|
||
1.6 Divisibility in the Integers
|
||
|
||
Definition 1.6.1. Let a and b be integers. We say a divides b, written a | b,
|
||
if there exists an integer k such that b = a k.
|
||
|
||
Theorem 1.6.2. Let a, b and c be integers. If a divides b and a divides c,
|
||
then a divides b + c.
|
||
|
||
Proof. Write b = a k and c = a m. Then b + c = a (k + m). QED
|
||
|
||
Theorem 1.6.3. Let a and b be nonzero integers. Then the set of common
|
||
"""
|
||
|
||
PAGE_TWO = """divisors of a and b has a greatest element, the greatest common divisor.
|
||
|
||
Exercise 1.6.4. Show that if a divides b and b divides a, then a = b or a = -b.
|
||
|
||
Exercise 1.6.5. Prove that the sum of two odd integers is even.
|
||
"""
|
||
|
||
|
||
@pytest.fixture()
|
||
def two_page_book(tmp_path: Path) -> Document:
|
||
path = tmp_path / "book.txt"
|
||
path.write_text(PAGE_ONE + "\f" + PAGE_TWO, encoding="utf-8")
|
||
return Document.load(path, title="Test Book")
|
||
|
||
|
||
class FakeChecker:
|
||
"""A Lean stand-in: fails anything containing ``BAD``, otherwise accepts."""
|
||
|
||
def __init__(self) -> None:
|
||
self.calls: list[str] = []
|
||
|
||
def available(self) -> bool:
|
||
return True
|
||
|
||
def check(self, code: str, *, mode: str = MODE_FULL, imports=None) -> VerifyResult:
|
||
self.calls.append(code)
|
||
if "BAD" in code:
|
||
return VerifyResult(False, mode, False, "LeanBenchCheck.lean:3:0: error: bad")
|
||
return VerifyResult(True, mode, "sorry" in code, "")
|
||
|
||
|
||
@pytest.fixture()
|
||
def fake_checker() -> FakeChecker:
|
||
return FakeChecker()
|
||
|
||
|
||
def detection_reply(items: list[dict]) -> str:
|
||
return json.dumps({"items": items})
|
||
|
||
|
||
def formalize_reply(**overrides) -> str:
|
||
payload = {
|
||
"skip": False,
|
||
"reason": "",
|
||
"imports": ["Mathlib"],
|
||
"name": "test_decl",
|
||
"lean_code": "theorem test_decl (a b : Int) (h : a ∣ b) : a ∣ b := by\n sorry",
|
||
"has_proof": False,
|
||
"informal": "a divides b",
|
||
"index_types": ["Int"],
|
||
"confidence": 0.8,
|
||
}
|
||
payload.update(overrides)
|
||
return json.dumps(payload)
|