claude-bot
b9e37952e6
Replaces the previous tree entirely. The pipeline is four separable pieces, each
testable on its own:
1. leanbench/llm.py provider-neutral OpenAI-compatible connector, stdlib only,
with retries, JSON coercion and a scripted client so the
rest of the code is testable without a network.
2. leanbench/detect.py per-page detection of formalizable mathematics. Case 1 is
mathematics asserted as fact (definition/theorem/lemma/...,
deliberately not sub-classified); case 2 is a problem left
open to the reader. Hits come back as spans -- start page
and index, end page and index -- so several per page is
normal. A hit cut off by the page end makes the detector
re-read that page together with the next, repeatedly, so an
item spread over three pages is found whole.
Offsets come from verbatim quotes resolved by
leanbench/anchors.py (exact -> whitespace-insensitive ->
fuzzy), because models quote well and count badly.
3. leanbench/formalize.py + leanbench/verify.py
each span becomes one Lean 4 declaration, judged by the real
compiler: full elaboration for case 1, statement legality
only for case 2, which has no proof to check. Lean's own
diagnostics feed back for a repair attempt. "verified" (Lean
accepted it) and "proved" (accepted with no sorry) are kept
strictly apart.
4. leanbench/db.py indexed SQLite plus JSONL export. Every item is indexed by
the types of its inputs, or by its own type when it is not a
proof, and records the source book (sha256) and the page it
starts on -- so `drop --source H --pages A-B` followed by a
re-ingest redoes exactly those pages and nothing else.
Infrastructure:
- lean/ Lake project pinned to Mathlib.
- docker/ image carrying Lean 4 and a prebuilt Mathlib. The Mathlib
revision and the toolchain are resolved against each other at
build time rather than both hardcoded, so the build does not rot.
docker/smoke.py runs the whole pipeline inside the image against
the real compiler, with the LLM scripted.
- .gitea/workflows ci (lint + tests in a venv, since the runner image's Python is
Debian-managed and refuses installs into itself), docker (build,
verify, optionally push), ingest (submit a book by hand).
The docker workflow is not a throwaway build: it leaves the image, the layer cache,
an idle `lean-bench` container and the `lean-bench-data` / `lean-bench-books`
volumes in place on the runner, and never prunes, so the environment is there to be
used afterwards. Pushing to the registry is optional and cannot fail the build --
Gitea's container registry does not accept the built-in Actions token, so that needs
a REGISTRY_TOKEN secret with write:package.
58 tests; they need neither network nor Lean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ci5T3gWDhhmqDec3e1XtYZ
73 lines
3.0 KiB
Docker
73 lines
3.0 KiB
Docker
# lean-bench: Lean 4 + Mathlib + the ingestion pipeline, in one image.
|
|
#
|
|
# Build from the repository root:
|
|
# docker build -f docker/Dockerfile -t lean-bench:latest .
|
|
#
|
|
# The Mathlib revision and the Lean toolchain are resolved against each other at
|
|
# build time (see docker/resolve-mathlib.sh); override with --build-arg MATHLIB_REV=v4.33.0.
|
|
FROM ubuntu:24.04
|
|
|
|
LABEL org.opencontainers.image.title="lean-bench" \
|
|
org.opencontainers.image.description="Detect formalizable mathematics in books and turn it into verified, indexed Lean 4" \
|
|
org.opencontainers.image.source="https://gitea.entroia.com/Zayden-Jung/lean-bench"
|
|
|
|
ENV DEBIAN_FRONTEND=noninteractive \
|
|
ELAN_HOME=/opt/elan \
|
|
PATH=/opt/elan/bin:$PATH \
|
|
PYTHONUNBUFFERED=1
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
build-essential ca-certificates curl git jq sqlite3 \
|
|
libgmp-dev python3 python3-pip \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# -- elan (the Lean toolchain manager) --------------------------------------
|
|
RUN curl -fsSL https://raw.githubusercontent.com/leanprover/elan/master/elan-init.sh -o /tmp/elan-init.sh \
|
|
&& sh /tmp/elan-init.sh -y --no-modify-path --default-toolchain none \
|
|
&& rm /tmp/elan-init.sh \
|
|
&& elan --version
|
|
|
|
WORKDIR /opt/lean-bench
|
|
|
|
# -- Lean project: toolchain, Mathlib, and its prebuilt cache ---------------
|
|
# Its own layer, because it is the expensive one and it changes rarely.
|
|
ARG MATHLIB_REV=auto
|
|
COPY lean/ lean/
|
|
COPY docker/resolve-mathlib.sh docker/resolve-mathlib.sh
|
|
RUN chmod +x docker/resolve-mathlib.sh && docker/resolve-mathlib.sh lean "$MATHLIB_REV"
|
|
RUN elan toolchain install "$(cat lean/lean-toolchain)" \
|
|
&& elan default "$(cat lean/lean-toolchain)"
|
|
RUN cd lean \
|
|
&& lake update \
|
|
&& (lake exe cache get || echo "no Mathlib cache available; building from source") \
|
|
&& lake build \
|
|
&& lake env lean --version
|
|
|
|
# -- the pipeline -----------------------------------------------------------
|
|
COPY pyproject.toml README.md ./
|
|
COPY leanbench/ leanbench/
|
|
RUN pip install --break-system-packages --no-cache-dir . \
|
|
&& lean-bench --help > /dev/null
|
|
|
|
COPY docker/entrypoint.sh /usr/local/bin/leanbench-entrypoint
|
|
RUN chmod +x /usr/local/bin/leanbench-entrypoint
|
|
|
|
ENV LEANBENCH_LEAN_PROJECT=/opt/lean-bench/lean \
|
|
LEANBENCH_LEAN_COMMAND='["lake","env","lean"]' \
|
|
LEANBENCH_DATABASE=/data/leanbench.sqlite \
|
|
LEANBENCH_LLM_MODEL=qwen-max \
|
|
LEANBENCH_LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 \
|
|
LEANBENCH_LLM_API_KEY_ENV=DASHSCOPE_API_KEY
|
|
|
|
# /books is where you put the source you are submitting; /data holds the database.
|
|
VOLUME ["/data", "/books"]
|
|
WORKDIR /work
|
|
|
|
# Prove the environment works while the image is still being built.
|
|
RUN mkdir -p /work && cd /opt/lean-bench/lean \
|
|
&& printf 'import Mathlib\ntheorem image_smoke (n : Nat) : n + 0 = n := by simp\n' > /tmp/Smoke.lean \
|
|
&& lake env lean /tmp/Smoke.lean && rm /tmp/Smoke.lean
|
|
|
|
ENTRYPOINT ["leanbench-entrypoint"]
|
|
CMD ["doctor"]
|