Rebuild lean-bench: pages in, verified indexed Lean out

Replaces the previous tree entirely. The pipeline is four separable pieces, each testable on its own: 1. leanbench/llm.py provider-neutral OpenAI-compatible connector, stdlib only, with retries, JSON coercion and a scripted client so the rest of the code is testable without a network. 2. leanbench/detect.py per-page detection of formalizable mathematics. Case 1 is mathematics asserted as fact (definition/theorem/lemma/..., deliberately not sub-classified); case 2 is a problem left open to the reader. Hits come back as spans -- start page and index, end page and index -- so several per page is normal. A hit cut off by the page end makes the detector re-read that page together with the next, repeatedly, so an item spread over three pages is found whole. Offsets come from verbatim quotes resolved by leanbench/anchors.py (exact -> whitespace-insensitive -> fuzzy), because models quote well and count badly. 3. leanbench/formalize.py + leanbench/verify.py each span becomes one Lean 4 declaration, judged by the real compiler: full elaboration for case 1, statement legality only for case 2, which has no proof to check. Lean's own diagnostics feed back for a repair attempt. "verified" (Lean accepted it) and "proved" (accepted with no sorry) are kept strictly apart. 4. leanbench/db.py indexed SQLite plus JSONL export. Every item is indexed by the types of its inputs, or by its own type when it is not a proof, and records the source book (sha256) and the page it starts on -- so `drop --source H --pages A-B` followed by a re-ingest redoes exactly those pages and nothing else. Infrastructure: - lean/ Lake project pinned to Mathlib. - docker/ image carrying Lean 4 and a prebuilt Mathlib. The Mathlib revision and the toolchain are resolved against each other at build time rather than both hardcoded, so the build does not rot. docker/smoke.py runs the whole pipeline inside the image against the real compiler, with the LLM scripted. - .gitea/workflows ci (lint + tests in a venv, since the runner image's Python is Debian-managed and refuses installs into itself), docker (build, verify, optionally push), ingest (submit a book by hand). The docker workflow is not a throwaway build: it leaves the image, the layer cache, an idle `lean-bench` container and the `lean-bench-data` / `lean-bench-books` volumes in place on the runner, and never prunes, so the environment is there to be used afterwards. Pushing to the registry is optional and cannot fail the build -- Gitea's container registry does not accept the built-in Actions token, so that needs a REGISTRY_TOKEN secret with write:package. 58 tests; they need neither network nor Lean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ci5T3gWDhhmqDec3e1XtYZ
2026-09-19 07:16:37 +00:00
commit b9e37952e6
38 changed files with 4548 additions and 0 deletions
@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+# Pin Mathlib and the Lean toolchain to each other.
+#
+# Mathlib tags do not exist for every Lean point release, so hardcoding both is how
+# image builds break six weeks later. Instead: choose a Mathlib revision, then take
+# *its* lean-toolchain as the truth and write both into the Lean project.
+#
+#   resolve-mathlib.sh <lean-project-dir> [revision]
+#
+# revision defaults to "auto": the newest Mathlib tag whose toolchain matches the
+# toolchain already in the project, else the newest tag, else master.
+set -euo pipefail
+
+PROJECT="${1:?usage: resolve-mathlib.sh <lean-project-dir> [revision]}"
+REQUESTED="${2:-auto}"
+REPO="https://github.com/leanprover-community/mathlib4.git"
+RAW="https://raw.githubusercontent.com/leanprover-community/mathlib4"
+TOOLCHAIN_FILE="$PROJECT/lean-toolchain"
+LAKEFILE="$PROJECT/lakefile.toml"
+
+toolchain_of() {  # toolchain_of <rev> -> prints e.g. leanprover/lean4:v4.33.0
+  curl -fsSL --retry 3 --max-time 30 "$RAW/$1/lean-toolchain" 2>/dev/null | tr -d '[:space:]'
+}
+
+candidates() {
+  git ls-remote --tags --refs "$REPO" 'v*' 2>/dev/null \
+    | awk '{print $2}' | sed 's#refs/tags/##' \
+    | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' \
+    | sort -V -r
+}
+
+wanted="$(tr -d '[:space:]' < "$TOOLCHAIN_FILE" 2>/dev/null || true)"
+rev=""
+
+if [ "$REQUESTED" != "auto" ]; then
+  rev="$REQUESTED"
+  echo "resolve-mathlib: using requested revision $rev"
+else
+  echo "resolve-mathlib: looking for a Mathlib tag built on ${wanted:-<unset>}"
+  all="$(candidates || true)"
+  if [ -n "$wanted" ]; then
+    # only the tags sharing a major.minor with what we asked for are plausible
+    series="$(printf '%s' "$wanted" | sed -E 's#.*:v([0-9]+\.[0-9]+).*#v\1#')"
+    for tag in $(printf '%s\n' "$all" | grep -E "^${series}\." || true); do
+      if [ "$(toolchain_of "$tag")" = "$wanted" ]; then
+        rev="$tag"
+        echo "resolve-mathlib: $tag matches $wanted exactly"
+        break
+      fi
+    done
+    if [ -z "$rev" ]; then
+      rev="$(printf '%s\n' "$all" | grep -E "^${series}\." | head -1 || true)"
+      [ -n "$rev" ] && echo "resolve-mathlib: no exact match; newest in $series series is $rev"
+    fi
+  fi
+  if [ -z "$rev" ]; then
+    rev="$(printf '%s\n' "$all" | head -1 || true)"
+    echo "resolve-mathlib: falling back to newest tag ${rev:-master}"
+  fi
+  rev="${rev:-master}"
+fi
+
+actual="$(toolchain_of "$rev" || true)"
+if [ -z "$actual" ]; then
+  echo "resolve-mathlib: could not read the toolchain for $rev; keeping ${wanted:-project default}" >&2
+  actual="$wanted"
+fi
+
+printf '%s\n' "$actual" > "$TOOLCHAIN_FILE"
+# rewrite only the rev line inside the [[require]] block for mathlib
+python3 - "$LAKEFILE" "$rev" <<'PY'
+import re
+import sys
+
+path, rev = sys.argv[1], sys.argv[2]
+with open(path, encoding="utf-8") as stream:
+    text = stream.read()
+text = re.sub(r'(?m)^rev\s*=\s*".*"$', f'rev = "{rev}"', text, count=1)
+with open(path, "w", encoding="utf-8") as stream:
+    stream.write(text)
+PY
+
+echo "resolve-mathlib: mathlib $rev on $actual"