Zayden-Jung/lean-bench
2
1
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Rebuild lean-bench: pages in, verified indexed Lean out
ci / test (push) Successful in 1m22s
Details
ci / lean-project (push) Successful in 6s
Details
docker / image (push) Failing after 3h0m9s
Details

Browse Source 
Replaces the previous tree entirely. The pipeline is four separable pieces, each
testable on its own:

1. leanbench/llm.py      provider-neutral OpenAI-compatible connector, stdlib only,
                        with retries, JSON coercion and a scripted client so the
                        rest of the code is testable without a network.
2. leanbench/detect.py   per-page detection of formalizable mathematics. Case 1 is
                        mathematics asserted as fact (definition/theorem/lemma/...,
                        deliberately not sub-classified); case 2 is a problem left
                        open to the reader. Hits come back as spans -- start page
                        and index, end page and index -- so several per page is
                        normal. A hit cut off by the page end makes the detector
                        re-read that page together with the next, repeatedly, so an
                        item spread over three pages is found whole.
                        Offsets come from verbatim quotes resolved by
                        leanbench/anchors.py (exact -> whitespace-insensitive ->
                        fuzzy), because models quote well and count badly.
3. leanbench/formalize.py + leanbench/verify.py
                        each span becomes one Lean 4 declaration, judged by the real
                        compiler: full elaboration for case 1, statement legality
                        only for case 2, which has no proof to check. Lean's own
                        diagnostics feed back for a repair attempt. "verified" (Lean
                        accepted it) and "proved" (accepted with no sorry) are kept
                        strictly apart.
4. leanbench/db.py       indexed SQLite plus JSONL export. Every item is indexed by
                        the types of its inputs, or by its own type when it is not a
                        proof, and records the source book (sha256) and the page it
                        starts on -- so `drop --source H --pages A-B` followed by a
                        re-ingest redoes exactly those pages and nothing else.

Infrastructure:

- lean/            Lake project pinned to Mathlib.
- docker/          image carrying Lean 4 and a prebuilt Mathlib. The Mathlib
                   revision and the toolchain are resolved against each other at
                   build time rather than both hardcoded, so the build does not rot.
                   docker/smoke.py runs the whole pipeline inside the image against
                   the real compiler, with the LLM scripted.
- .gitea/workflows ci (lint + tests in a venv, since the runner image's Python is
                   Debian-managed and refuses installs into itself), docker (build,
                   verify, optionally push), ingest (submit a book by hand).

The docker workflow is not a throwaway build: it leaves the image, the layer cache,
an idle `lean-bench` container and the `lean-bench-data` / `lean-bench-books`
volumes in place on the runner, and never prunes, so the environment is there to be
used afterwards. Pushing to the registry is optional and cannot fail the build --
Gitea's container registry does not accept the built-in Actions token, so that needs
a REGISTRY_TOKEN secret with write:package.

58 tests; they need neither network nor Lean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ci5T3gWDhhmqDec3e1XtYZ
This commit is contained in:
claude-bot
2026-09-19 07:16:37 +00:00
commit b9e37952e6
38 changed files with 4548 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/ci.yml
+57
View File
@@ -0,0 +1,57 @@
name: ci
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# The runner image ships a Debian-managed Python (PEP 668), so installing
# into it is refused and `pip install --upgrade pip` cannot replace the
# distro pip. A venv sidesteps both, and costs a couple of seconds.
- name: Set up a virtualenv
run: |
set -eu
python3 --version
python3 -m venv .venv || {
apt-get update -qq
apt-get install -y -qq python3-venv
python3 -m venv .venv
}
.venv/bin/python -m pip install --quiet --upgrade pip
- name: Install
run: .venv/bin/python -m pip install --quiet -e ".[dev]"
- name: Lint
run: .venv/bin/python -m ruff check .
- name: Test
run: .venv/bin/python -m pytest -q
- name: CLI is wired up
run: |
set -eu
.venv/bin/lean-bench --help > /dev/null
.venv/bin/lean-bench doctor --skip-llm --env /dev/null
lean-project:
# Cheap guard: the Lean project's pins must at least resolve.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Resolve the Mathlib / toolchain pair
run: |
set -eu
command -v curl >/dev/null || (apt-get update -qq && apt-get install -y -qq curl)
command -v git >/dev/null || (apt-get update -qq && apt-get install -y -qq git)
chmod +x docker/resolve-mathlib.sh
docker/resolve-mathlib.sh lean "${MATHLIB_REV:-auto}"
echo "toolchain: $(cat lean/lean-toolchain)"
grep -E '^rev' lean/lakefile.toml
.gitea/workflows/docker.yml
+221
View File
@@ -0,0 +1,221 @@
name: docker
# Builds the Lean 4 + Mathlib image and LEAVES IT IN PLACE on the runner host.
#
# This is not a throwaway test build. When the job finishes, the runner still has:
# * the image, tagged lean-bench:latest and lean-bench:sha-<commit>
# * the Docker layer cache, so the next build reuses Mathlib instead of
# re-downloading it (nothing here ever calls `docker system prune`)
# * a long-lived container `lean-bench`, idle and ready for `docker exec`
# * named volumes `lean-bench-data` (the item database) and `lean-bench-books`
#
# so you can submit a book straight afterwards:
# docker exec -e DASHSCOPE_API_KEY=sk-... lean-bench \
# lean-bench ingest /books/your-book.pdf --pages 30-60
#
# Pushing to the registry is optional and never fails the build: the built-in
# Actions token is not accepted by Gitea's container registry, so add a personal
# access token with write:package as the REGISTRY_TOKEN secret to enable it.
on:
push:
branches: [main]
tags: ["v*"]
workflow_dispatch:
inputs:
mathlib_rev:
description: "Mathlib revision (tag, branch, commit, or 'auto')"
required: false
default: "auto"
push_image:
description: "Push to the registry if a token is configured"
required: false
default: "true"
refresh_service:
description: "Recreate the long-lived lean-bench container even if one is running"
required: false
default: "false"
no_cache:
description: "Build without the layer cache (slow: refetches Mathlib)"
required: false
default: "false"
env:
REGISTRY: gitea.entroia.com
IMAGE_NAME: lean-bench
LOCAL_IMAGE: lean-bench
SERVICE_CONTAINER: lean-bench
DATA_VOLUME: lean-bench-data
BOOKS_VOLUME: lean-bench-books
jobs:
image:
runs-on: ubuntu-latest
timeout-minutes: 240
steps:
- uses: actions/checkout@v4
- name: Check the runner can build images
run: |
set -eu
if ! command -v docker >/dev/null; then
echo "::error::No docker CLI on this runner. Give the act_runner the host"
echo "::error::docker socket (-v /var/run/docker.sock:/var/run/docker.sock)"
echo "::error::or run it with the docker-in-docker image."
exit 1
fi
docker version --format 'client {{.Client.Version}} / server {{.Server.Version}}'
echo "disk on the docker host:"
docker system df || true
- name: Work out the tags
id: meta
run: |
set -eu
owner="$(echo "${{ github.repository_owner }}" | tr '[:upper:]' '[:lower:]')"
remote="${REGISTRY}/${owner}/${IMAGE_NAME}"
ref="${GITHUB_REF_NAME:-main}"
sha="$(echo "${GITHUB_SHA:-unknown}" | cut -c1-12)"
case "${GITHUB_REF:-}" in
refs/tags/*) version="${ref}" ;;
*) version="${ref//\//-}" ;;
esac
# Local tags are what stays on the runner; remote tags are for the registry.
local_tags="${LOCAL_IMAGE}:${version} ${LOCAL_IMAGE}:sha-${sha}"
remote_tags="${remote}:${version} ${remote}:sha-${sha}"
if [ "${ref}" = "main" ]; then
local_tags="${local_tags} ${LOCAL_IMAGE}:latest"
remote_tags="${remote_tags} ${remote}:latest"
fi
{
echo "remote=${remote}"
echo "primary=${LOCAL_IMAGE}:${version}"
echo "local_tags=${local_tags}"
echo "remote_tags=${remote_tags}"
} >> "$GITHUB_OUTPUT"
echo "local: ${local_tags}"
echo "remote: ${remote_tags}"
- name: Build
run: |
set -eu
args=""
for tag in ${{ steps.meta.outputs.local_tags }} ${{ steps.meta.outputs.remote_tags }}; do
args="$args -t $tag"
done
# Reuse the previous build's layers explicitly as well as implicitly, so a
# rebuild does not refetch Mathlib.
cache=""
if [ '${{ inputs.no_cache }}' = 'true' ]; then
cache="--no-cache"
elif docker image inspect "${LOCAL_IMAGE}:latest" >/dev/null 2>&1; then
cache="--cache-from ${LOCAL_IMAGE}:latest"
fi
echo "building with: $cache"
docker build \
-f docker/Dockerfile \
--build-arg "MATHLIB_REV=${{ inputs.mathlib_rev || 'auto' }}" \
--build-arg BUILDKIT_INLINE_CACHE=1 \
$cache $args .
- name: Check the image actually works
run: |
set -eu
image='${{ steps.meta.outputs.primary }}'
docker run --rm "$image" doctor --skip-llm
docker run --rm --entrypoint bash "$image" -lc \
'cd "$LEANBENCH_LEAN_PROJECT" && printf "import Mathlib\ntheorem ci_smoke (n : Nat) : n + 0 = n := by simp\n" > /tmp/CI.lean && lake env lean /tmp/CI.lean && echo "lean + mathlib ok"'
# Whole pipeline, scripted LLM, real compiler. Piped in over stdin so the
# job container needs no bind mount into the host daemon.
docker run --rm -i --entrypoint python3 "$image" - < docker/smoke.py
- name: Log in to the registry
id: login
if: ${{ github.event_name != 'workflow_dispatch' || inputs.push_image == 'true' }}
continue-on-error: true
run: |
set -eu
token='${{ secrets.REGISTRY_TOKEN }}'
[ -n "$token" ] || token='${{ secrets.GITEA_TOKEN }}'
[ -n "$token" ] || token='${{ secrets.GITHUB_TOKEN }}'
user='${{ vars.REGISTRY_USER }}'
[ -n "$user" ] || user='${{ github.actor }}'
if [ -z "$token" ]; then
echo "::warning::No registry token; keeping the image on the runner only."
exit 1
fi
echo "$token" | docker login "${REGISTRY}" -u "$user" --password-stdin
- name: Push
id: push
if: ${{ steps.login.outcome == 'success' }}
continue-on-error: true
run: |
set -eu
for tag in ${{ steps.meta.outputs.remote_tags }}; do
echo "pushing $tag"
docker push "$tag"
done
- name: Leave the environment ready for use
run: |
set -eu
image='${{ steps.meta.outputs.primary }}'
docker volume create "${DATA_VOLUME}" >/dev/null
docker volume create "${BOOKS_VOLUME}" >/dev/null
running=""
if docker ps --format '{{.Names}}' | grep -qx "${SERVICE_CONTAINER}"; then
running="yes"
fi
if [ -n "$running" ] && [ '${{ inputs.refresh_service }}' != 'true' ]; then
echo "${SERVICE_CONTAINER} is already running; leaving it alone so nothing"
echo "in flight is interrupted. Re-run with refresh_service=true to move it"
echo "onto the image just built."
else
docker rm -f "${SERVICE_CONTAINER}" >/dev/null 2>&1 || true
docker run -d \
--name "${SERVICE_CONTAINER}" \
--restart unless-stopped \
-v "${DATA_VOLUME}:/data" \
-v "${BOOKS_VOLUME}:/books" \
-e DASHSCOPE_API_KEY \
--entrypoint sleep \
"$image" infinity
echo "${SERVICE_CONTAINER} is up on $image"
fi
docker exec "${SERVICE_CONTAINER}" lean-bench doctor --skip-llm || true
echo "images kept on this runner:"
docker images "${LOCAL_IMAGE}" --format ' {{.Repository}}:{{.Tag}} {{.Size}} {{.CreatedSince}}'
env:
DASHSCOPE_API_KEY: ${{ secrets.DASHSCOPE_API_KEY }}
- name: Summary
if: always()
run: |
{
echo "### lean-bench image"
echo ""
echo "Kept on the runner (not cleaned up):"
for tag in ${{ steps.meta.outputs.local_tags }}; do echo "- \`$tag\`"; done
echo ""
if [ '${{ steps.push.outcome }}' = 'success' ]; then
echo "Pushed to the registry:"
for tag in ${{ steps.meta.outputs.remote_tags }}; do echo "- \`$tag\`"; done
else
echo "**Not pushed.** Gitea's container registry rejects the built-in"
echo "Actions token. Create a personal access token with \`write:package\`"
echo "and add it as the \`REGISTRY_TOKEN\` secret (and \`REGISTRY_USER\`"
echo "as a variable if it is not \`${{ github.actor }}\`). The image is on"
echo "the runner either way."
fi
echo ""
echo "Submit a book to the container that is already running:"
echo '```'
echo "docker cp your-book.pdf lean-bench:/books/"
echo "docker exec -e DASHSCOPE_API_KEY=sk-... lean-bench \\"
echo " lean-bench ingest /books/your-book.pdf --pages 30-60"
echo "docker exec lean-bench lean-bench stats"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
.gitea/workflows/ingest.yml
+129
View File
@@ -0,0 +1,129 @@
name: ingest
# Submit a book and run the whole pipeline, inside the long-lived `lean-bench`
# container the docker workflow leaves running on this runner. The book is copied
# into the `lean-bench-books` volume and the items land in `lean-bench-data`, so
# everything survives the job and is there for the next one -- `--pages` of the
# same book can be redone later without re-ingesting the rest.
#
# The source can be a path inside the repository or a URL the runner can reach.
# Set the DASHSCOPE_API_KEY secret (or whichever key your provider uses) first.
on:
workflow_dispatch:
inputs:
source:
description: "Path in the repo, or a URL, of the book to ingest"
required: true
pages:
description: "Page range, e.g. 30-60 (blank = the whole book)"
required: false
default: ""
title:
description: "Human title for this source"
required: false
default: ""
verify:
description: "Run Lean verification (slow, but the point)"
required: false
default: "true"
image:
description: "Image to use if the lean-bench container is not running"
required: false
default: "lean-bench:latest"
env:
SERVICE_CONTAINER: lean-bench
DATA_VOLUME: lean-bench-data
BOOKS_VOLUME: lean-bench-books
jobs:
ingest:
runs-on: ubuntu-latest
timeout-minutes: 720
steps:
- uses: actions/checkout@v4
- name: Fetch the source
id: source
run: |
set -eu
mkdir -p books out
src='${{ inputs.source }}'
case "$src" in
http://*|https://*)
name="$(basename "${src%%\?*}")"
curl -fsSL --retry 3 -o "books/$name" "$src"
;;
*)
[ -f "$src" ] || { echo "::error::$src is not in the repository"; exit 1; }
name="$(basename "$src")"
cp "$src" "books/$name"
;;
esac
ls -l "books/$name"
echo "name=$name" >> "$GITHUB_OUTPUT"
- name: Make sure the lean-bench container is up
run: |
set -eu
docker volume create "${DATA_VOLUME}" >/dev/null
docker volume create "${BOOKS_VOLUME}" >/dev/null
if ! docker ps --format '{{.Names}}' | grep -qx "${SERVICE_CONTAINER}"; then
image='${{ inputs.image }}'
if ! docker image inspect "$image" >/dev/null 2>&1; then
echo "::error::No $image on this runner. Run the docker workflow first."
exit 1
fi
docker rm -f "${SERVICE_CONTAINER}" >/dev/null 2>&1 || true
docker run -d --name "${SERVICE_CONTAINER}" --restart unless-stopped \
-v "${DATA_VOLUME}:/data" -v "${BOOKS_VOLUME}:/books" \
--entrypoint sleep "$image" infinity
fi
docker exec "${SERVICE_CONTAINER}" lean-bench doctor --skip-llm
- name: Ingest
env:
DASHSCOPE_API_KEY: ${{ secrets.DASHSCOPE_API_KEY }}
LEANBENCH_LLM_MODEL: ${{ vars.LEANBENCH_LLM_MODEL }}
LEANBENCH_LLM_BASE_URL: ${{ vars.LEANBENCH_LLM_BASE_URL }}
run: |
set -eu
[ -n "${DASHSCOPE_API_KEY:-}" ] || { echo "::error::Set the DASHSCOPE_API_KEY secret"; exit 1; }
name='${{ steps.source.outputs.name }}'
docker cp "books/$name" "${SERVICE_CONTAINER}:/books/$name"
set -- ingest "/books/$name" \
--export "/data/${name%.*}.items.jsonl" \
--detections "/data/${name%.*}.detections.jsonl"
# `cond && set -- ...` would abort the step under `set -e` whenever the
# condition is false, so these are spelled out.
if [ -n '${{ inputs.pages }}' ]; then set -- "$@" --pages '${{ inputs.pages }}'; fi
if [ -n '${{ inputs.title }}' ]; then set -- "$@" --title '${{ inputs.title }}'; fi
if [ '${{ inputs.verify }}' != 'true' ]; then set -- "$@" --no-verify; fi
docker exec \
-e DASHSCOPE_API_KEY \
-e LEANBENCH_LLM_MODEL \
-e LEANBENCH_LLM_BASE_URL \
"${SERVICE_CONTAINER}" lean-bench "$@"
docker exec "${SERVICE_CONTAINER}" lean-bench stats
- name: Collect a copy of the results
if: always()
run: |
set -eu
# The originals stay in the lean-bench-data volume; these are just copies
# for the artifact.
docker exec "${SERVICE_CONTAINER}" sh -c 'ls -1 /data' || true
for f in $(docker exec "${SERVICE_CONTAINER}" sh -c 'ls -1 /data' 2>/dev/null || true); do
docker cp "${SERVICE_CONTAINER}:/data/$f" "out/$f" || true
done
ls -l out || true
- uses: actions/upload-artifact@v3
if: always()
with:
name: leanbench-${{ steps.source.outputs.name }}
path: out/
retention-days: 30