Rebuild lean-bench: pages in, verified indexed Lean out
Replaces the previous tree entirely. The pipeline is four separable pieces, each
testable on its own:
1. leanbench/llm.py provider-neutral OpenAI-compatible connector, stdlib only,
with retries, JSON coercion and a scripted client so the
rest of the code is testable without a network.
2. leanbench/detect.py per-page detection of formalizable mathematics. Case 1 is
mathematics asserted as fact (definition/theorem/lemma/...,
deliberately not sub-classified); case 2 is a problem left
open to the reader. Hits come back as spans -- start page
and index, end page and index -- so several per page is
normal. A hit cut off by the page end makes the detector
re-read that page together with the next, repeatedly, so an
item spread over three pages is found whole.
Offsets come from verbatim quotes resolved by
leanbench/anchors.py (exact -> whitespace-insensitive ->
fuzzy), because models quote well and count badly.
3. leanbench/formalize.py + leanbench/verify.py
each span becomes one Lean 4 declaration, judged by the real
compiler: full elaboration for case 1, statement legality
only for case 2, which has no proof to check. Lean's own
diagnostics feed back for a repair attempt. "verified" (Lean
accepted it) and "proved" (accepted with no sorry) are kept
strictly apart.
4. leanbench/db.py indexed SQLite plus JSONL export. Every item is indexed by
the types of its inputs, or by its own type when it is not a
proof, and records the source book (sha256) and the page it
starts on -- so `drop --source H --pages A-B` followed by a
re-ingest redoes exactly those pages and nothing else.
Also: lean/ Lake project pinned to Mathlib, docker/ image carrying Lean 4 and a
prebuilt Mathlib (the Mathlib revision and the toolchain are resolved against each
other at build time rather than both hardcoded), and .gitea/workflows for tests,
image build and push to the Gitea registry, and a manual "submit a book" run.
58 tests; they need neither network nor Lean.
This commit is contained in:
@@ -0,0 +1,18 @@
|
||||
/-
|
||||
The verification environment.
|
||||
|
||||
Candidate declarations are never added to this file: each one is elaborated in its
|
||||
own scratch file against this project's imports, so a broken candidate can never
|
||||
break the environment the next candidate is checked in.
|
||||
|
||||
Anything you want every candidate to see -- a shared convention, a local notation --
|
||||
belongs here.
|
||||
-/
|
||||
import Mathlib
|
||||
|
||||
namespace LeanBench
|
||||
|
||||
/-- Sanity check: if this fails, the toolchain and Mathlib do not agree. -/
|
||||
theorem environment_ok : (2 : ℕ) + 2 = 4 := by norm_num
|
||||
|
||||
end LeanBench
|
||||
@@ -0,0 +1,12 @@
|
||||
name = "leanbench"
|
||||
defaultTargets = ["LeanBench"]
|
||||
|
||||
# The revision is rewritten by docker/resolve-mathlib.sh at image build time, so the
|
||||
# Mathlib revision and the Lean toolchain always agree. Edit MATHLIB_REV, not this.
|
||||
[[require]]
|
||||
name = "mathlib"
|
||||
git = "https://github.com/leanprover-community/mathlib4.git"
|
||||
rev = "v4.33.1"
|
||||
|
||||
[[lean_lib]]
|
||||
name = "LeanBench"
|
||||
@@ -0,0 +1 @@
|
||||
leanprover/lean4:v4.33.1
|
||||
Reference in New Issue
Block a user