d3555d9a52
Replaces the previous tree entirely. The pipeline is four separable pieces, each
testable on its own:
1. leanbench/llm.py provider-neutral OpenAI-compatible connector, stdlib only,
with retries, JSON coercion and a scripted client so the
rest of the code is testable without a network.
2. leanbench/detect.py per-page detection of formalizable mathematics. Case 1 is
mathematics asserted as fact (definition/theorem/lemma/...,
deliberately not sub-classified); case 2 is a problem left
open to the reader. Hits come back as spans -- start page
and index, end page and index -- so several per page is
normal. A hit cut off by the page end makes the detector
re-read that page together with the next, repeatedly, so an
item spread over three pages is found whole.
Offsets come from verbatim quotes resolved by
leanbench/anchors.py (exact -> whitespace-insensitive ->
fuzzy), because models quote well and count badly.
3. leanbench/formalize.py + leanbench/verify.py
each span becomes one Lean 4 declaration, judged by the real
compiler: full elaboration for case 1, statement legality
only for case 2, which has no proof to check. Lean's own
diagnostics feed back for a repair attempt. "verified" (Lean
accepted it) and "proved" (accepted with no sorry) are kept
strictly apart.
4. leanbench/db.py indexed SQLite plus JSONL export. Every item is indexed by
the types of its inputs, or by its own type when it is not a
proof, and records the source book (sha256) and the page it
starts on -- so `drop --source H --pages A-B` followed by a
re-ingest redoes exactly those pages and nothing else.
Also: lean/ Lake project pinned to Mathlib, docker/ image carrying Lean 4 and a
prebuilt Mathlib (the Mathlib revision and the toolchain are resolved against each
other at build time rather than both hardcoded), and .gitea/workflows for tests,
image build and push to the Gitea registry, and a manual "submit a book" run.
58 tests; they need neither network nor Lean.
101 lines
3.4 KiB
YAML
101 lines
3.4 KiB
YAML
name: ingest
|
|
|
|
# Submit a book and run the whole pipeline on the runner, inside the image the
|
|
# docker workflow built. The database and the JSONL export come back as artifacts.
|
|
#
|
|
# The source can be a path inside the repository or a URL the runner can reach.
|
|
# Set the DASHSCOPE_API_KEY secret (or whichever key your provider uses) first.
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
inputs:
|
|
source:
|
|
description: "Path in the repo, or a URL, of the book to ingest"
|
|
required: true
|
|
pages:
|
|
description: "Page range, e.g. 30-60 (blank = the whole book)"
|
|
required: false
|
|
default: ""
|
|
title:
|
|
description: "Human title for this source"
|
|
required: false
|
|
default: ""
|
|
image:
|
|
description: "Image to run"
|
|
required: false
|
|
default: "gitea.entroia.com/zayden-jung/lean-bench:latest"
|
|
verify:
|
|
description: "Run Lean verification (slow but the point)"
|
|
required: false
|
|
default: "true"
|
|
|
|
jobs:
|
|
ingest:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- name: Fetch the source
|
|
id: source
|
|
run: |
|
|
set -eu
|
|
mkdir -p books data out
|
|
src='${{ inputs.source }}'
|
|
case "$src" in
|
|
http://*|https://*)
|
|
name="$(basename "${src%%\?*}")"
|
|
curl -fsSL --retry 3 -o "books/$name" "$src"
|
|
;;
|
|
*)
|
|
[ -f "$src" ] || { echo "::error::$src is not in the repository"; exit 1; }
|
|
name="$(basename "$src")"
|
|
cp "$src" "books/$name"
|
|
;;
|
|
esac
|
|
ls -l "books/$name"
|
|
echo "name=$name" >> "$GITHUB_OUTPUT"
|
|
|
|
- name: Log in to the registry
|
|
run: |
|
|
set -eu
|
|
token='${{ secrets.GITEA_TOKEN }}'
|
|
[ -n "$token" ] || token='${{ secrets.GITHUB_TOKEN }}'
|
|
echo "$token" | docker login gitea.entroia.com -u '${{ github.actor }}' --password-stdin
|
|
|
|
- name: Ingest
|
|
env:
|
|
DASHSCOPE_API_KEY: ${{ secrets.DASHSCOPE_API_KEY }}
|
|
LEANBENCH_LLM_MODEL: ${{ vars.LEANBENCH_LLM_MODEL }}
|
|
LEANBENCH_LLM_BASE_URL: ${{ vars.LEANBENCH_LLM_BASE_URL }}
|
|
run: |
|
|
set -eu
|
|
[ -n "${DASHSCOPE_API_KEY:-}" ] || { echo "::error::Set the DASHSCOPE_API_KEY secret"; exit 1; }
|
|
docker pull '${{ inputs.image }}'
|
|
args="ingest /books/${{ steps.source.outputs.name }} --export /data/items.jsonl --detections /data/detections.jsonl"
|
|
[ -n "${{ inputs.pages }}" ] && args="$args --pages ${{ inputs.pages }}"
|
|
[ -n "${{ inputs.title }}" ] && args="$args --title '${{ inputs.title }}'"
|
|
[ '${{ inputs.verify }}' = 'true' ] || args="$args --no-verify"
|
|
docker run --rm \
|
|
-e DASHSCOPE_API_KEY \
|
|
-e LEANBENCH_LLM_MODEL \
|
|
-e LEANBENCH_LLM_BASE_URL \
|
|
-v "$PWD/books:/books:ro" \
|
|
-v "$PWD/data:/data" \
|
|
'${{ inputs.image }}' \
|
|
sh -c "lean-bench $args && lean-bench stats"
|
|
|
|
- name: Collect
|
|
if: always()
|
|
run: |
|
|
set -eu
|
|
cp -f data/*.jsonl out/ 2>/dev/null || true
|
|
cp -f data/leanbench.sqlite out/ 2>/dev/null || true
|
|
ls -l out || true
|
|
|
|
- uses: actions/upload-artifact@v3
|
|
if: always()
|
|
with:
|
|
name: leanbench-${{ steps.source.outputs.name }}
|
|
path: out/
|
|
retention-days: 30
|