Rebuild lean-bench: pages in, verified indexed Lean out
Replaces the previous tree entirely. The pipeline is four separable pieces, each
testable on its own:
1. leanbench/llm.py provider-neutral OpenAI-compatible connector, stdlib only,
with retries, JSON coercion and a scripted client so the
rest of the code is testable without a network.
2. leanbench/detect.py per-page detection of formalizable mathematics. Case 1 is
mathematics asserted as fact (definition/theorem/lemma/...,
deliberately not sub-classified); case 2 is a problem left
open to the reader. Hits come back as spans -- start page
and index, end page and index -- so several per page is
normal. A hit cut off by the page end makes the detector
re-read that page together with the next, repeatedly, so an
item spread over three pages is found whole.
Offsets come from verbatim quotes resolved by
leanbench/anchors.py (exact -> whitespace-insensitive ->
fuzzy), because models quote well and count badly.
3. leanbench/formalize.py + leanbench/verify.py
each span becomes one Lean 4 declaration, judged by the real
compiler: full elaboration for case 1, statement legality
only for case 2, which has no proof to check. Lean's own
diagnostics feed back for a repair attempt. "verified" (Lean
accepted it) and "proved" (accepted with no sorry) are kept
strictly apart.
4. leanbench/db.py indexed SQLite plus JSONL export. Every item is indexed by
the types of its inputs, or by its own type when it is not a
proof, and records the source book (sha256) and the page it
starts on -- so `drop --source H --pages A-B` followed by a
re-ingest redoes exactly those pages and nothing else.
Infrastructure:
- lean/ Lake project pinned to Mathlib.
- docker/ image carrying Lean 4 and a prebuilt Mathlib. The Mathlib
revision and the toolchain are resolved against each other at
build time rather than both hardcoded, so the build does not rot.
docker/smoke.py runs the whole pipeline inside the image against
the real compiler, with the LLM scripted.
- .gitea/workflows ci (lint + tests in a venv, since the runner image's Python is
Debian-managed and refuses installs into itself), docker (build,
verify, optionally push), ingest (submit a book by hand).
The docker workflow is not a throwaway build: it leaves the image, the layer cache,
an idle `lean-bench` container and the `lean-bench-data` / `lean-bench-books`
volumes in place on the runner, and never prunes, so the environment is there to be
used afterwards. Pushing to the registry is optional and cannot fail the build --
Gitea's container registry does not accept the built-in Actions token, so that needs
a REGISTRY_TOKEN secret with write:package.
58 tests; they need neither network nor Lean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ci5T3gWDhhmqDec3e1XtYZ
This commit is contained in:
@@ -0,0 +1,129 @@
|
||||
name: ingest
|
||||
|
||||
# Submit a book and run the whole pipeline, inside the long-lived `lean-bench`
|
||||
# container the docker workflow leaves running on this runner. The book is copied
|
||||
# into the `lean-bench-books` volume and the items land in `lean-bench-data`, so
|
||||
# everything survives the job and is there for the next one -- `--pages` of the
|
||||
# same book can be redone later without re-ingesting the rest.
|
||||
#
|
||||
# The source can be a path inside the repository or a URL the runner can reach.
|
||||
# Set the DASHSCOPE_API_KEY secret (or whichever key your provider uses) first.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
source:
|
||||
description: "Path in the repo, or a URL, of the book to ingest"
|
||||
required: true
|
||||
pages:
|
||||
description: "Page range, e.g. 30-60 (blank = the whole book)"
|
||||
required: false
|
||||
default: ""
|
||||
title:
|
||||
description: "Human title for this source"
|
||||
required: false
|
||||
default: ""
|
||||
verify:
|
||||
description: "Run Lean verification (slow, but the point)"
|
||||
required: false
|
||||
default: "true"
|
||||
image:
|
||||
description: "Image to use if the lean-bench container is not running"
|
||||
required: false
|
||||
default: "lean-bench:latest"
|
||||
|
||||
env:
|
||||
SERVICE_CONTAINER: lean-bench
|
||||
DATA_VOLUME: lean-bench-data
|
||||
BOOKS_VOLUME: lean-bench-books
|
||||
|
||||
jobs:
|
||||
ingest:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 720
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Fetch the source
|
||||
id: source
|
||||
run: |
|
||||
set -eu
|
||||
mkdir -p books out
|
||||
src='${{ inputs.source }}'
|
||||
case "$src" in
|
||||
http://*|https://*)
|
||||
name="$(basename "${src%%\?*}")"
|
||||
curl -fsSL --retry 3 -o "books/$name" "$src"
|
||||
;;
|
||||
*)
|
||||
[ -f "$src" ] || { echo "::error::$src is not in the repository"; exit 1; }
|
||||
name="$(basename "$src")"
|
||||
cp "$src" "books/$name"
|
||||
;;
|
||||
esac
|
||||
ls -l "books/$name"
|
||||
echo "name=$name" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Make sure the lean-bench container is up
|
||||
run: |
|
||||
set -eu
|
||||
docker volume create "${DATA_VOLUME}" >/dev/null
|
||||
docker volume create "${BOOKS_VOLUME}" >/dev/null
|
||||
if ! docker ps --format '{{.Names}}' | grep -qx "${SERVICE_CONTAINER}"; then
|
||||
image='${{ inputs.image }}'
|
||||
if ! docker image inspect "$image" >/dev/null 2>&1; then
|
||||
echo "::error::No $image on this runner. Run the docker workflow first."
|
||||
exit 1
|
||||
fi
|
||||
docker rm -f "${SERVICE_CONTAINER}" >/dev/null 2>&1 || true
|
||||
docker run -d --name "${SERVICE_CONTAINER}" --restart unless-stopped \
|
||||
-v "${DATA_VOLUME}:/data" -v "${BOOKS_VOLUME}:/books" \
|
||||
--entrypoint sleep "$image" infinity
|
||||
fi
|
||||
docker exec "${SERVICE_CONTAINER}" lean-bench doctor --skip-llm
|
||||
|
||||
- name: Ingest
|
||||
env:
|
||||
DASHSCOPE_API_KEY: ${{ secrets.DASHSCOPE_API_KEY }}
|
||||
LEANBENCH_LLM_MODEL: ${{ vars.LEANBENCH_LLM_MODEL }}
|
||||
LEANBENCH_LLM_BASE_URL: ${{ vars.LEANBENCH_LLM_BASE_URL }}
|
||||
run: |
|
||||
set -eu
|
||||
[ -n "${DASHSCOPE_API_KEY:-}" ] || { echo "::error::Set the DASHSCOPE_API_KEY secret"; exit 1; }
|
||||
name='${{ steps.source.outputs.name }}'
|
||||
docker cp "books/$name" "${SERVICE_CONTAINER}:/books/$name"
|
||||
|
||||
set -- ingest "/books/$name" \
|
||||
--export "/data/${name%.*}.items.jsonl" \
|
||||
--detections "/data/${name%.*}.detections.jsonl"
|
||||
# `cond && set -- ...` would abort the step under `set -e` whenever the
|
||||
# condition is false, so these are spelled out.
|
||||
if [ -n '${{ inputs.pages }}' ]; then set -- "$@" --pages '${{ inputs.pages }}'; fi
|
||||
if [ -n '${{ inputs.title }}' ]; then set -- "$@" --title '${{ inputs.title }}'; fi
|
||||
if [ '${{ inputs.verify }}' != 'true' ]; then set -- "$@" --no-verify; fi
|
||||
|
||||
docker exec \
|
||||
-e DASHSCOPE_API_KEY \
|
||||
-e LEANBENCH_LLM_MODEL \
|
||||
-e LEANBENCH_LLM_BASE_URL \
|
||||
"${SERVICE_CONTAINER}" lean-bench "$@"
|
||||
docker exec "${SERVICE_CONTAINER}" lean-bench stats
|
||||
|
||||
- name: Collect a copy of the results
|
||||
if: always()
|
||||
run: |
|
||||
set -eu
|
||||
# The originals stay in the lean-bench-data volume; these are just copies
|
||||
# for the artifact.
|
||||
docker exec "${SERVICE_CONTAINER}" sh -c 'ls -1 /data' || true
|
||||
for f in $(docker exec "${SERVICE_CONTAINER}" sh -c 'ls -1 /data' 2>/dev/null || true); do
|
||||
docker cp "${SERVICE_CONTAINER}:/data/$f" "out/$f" || true
|
||||
done
|
||||
ls -l out || true
|
||||
|
||||
- uses: actions/upload-artifact@v3
|
||||
if: always()
|
||||
with:
|
||||
name: leanbench-${{ steps.source.outputs.name }}
|
||||
path: out/
|
||||
retention-days: 30
|
||||
Reference in New Issue
Block a user