Logic Skeleton Bench tests whether a solver transfers the same abstract structure across different surface stories. It first creates a typed intermediate representation with a deterministic gold answer, asks one LLM to render that representation as a natural-language problem, asks another LLM to solve only the rendered problem, and then scores the answer.

The renderer and solver are intentionally separate clients. Each can use a different model, API endpoint, key, and OpenAI-compatible API style, so model-pair heatmaps can be added without changing the skeleton generator.

What the first version contains

Three finite skeleton families are included:

transitive_order: a chain of directional relations;
membership_inheritance: membership propagated through subset relations;
sequential_state_update: ordered arithmetic state changes.

Every generated skeleton contains its facts, rules, query, permitted answer labels, deterministic gold_answer, and proof trace. The renderer receives a copy with gold_answer and proof removed.

length means logical complexity, not natural-language tokens. It is the number of proof-bearing facts or state updates. Presets are short=2, medium=4, and long=6; an integer from 1 through 20 is also accepted.

Local use

Python 3.11 or newer is sufficient; the package has no runtime dependencies.

Generate three short skeletons without calling an LLM:

python3 -m logicbench generate --count 3 --length short --seed 20260922

Run the checked-in live task:

export OPENAI_API_KEY='...'
export RENDERER_MODEL='your-renderer-model'
export SOLVER_MODEL='your-solver-model'
python3 -m logicbench run tasks/short-smoke.json

Role-specific values override the shared fallback:

Variable	Meaning
`RENDERER_API_KEY`, `SOLVER_API_KEY`	Optional separate keys; each falls back to `OPENAI_API_KEY`
`RENDERER_BASE_URL`, `SOLVER_BASE_URL`	Optional endpoints; each falls back to `OPENAI_BASE_URL`, then `https://api.openai.com/v1`
`RENDERER_API_STYLE`, `SOLVER_API_STYLE`	`responses` (default) or `chat_completions`
`RENDERER_MODEL`, `SOLVER_MODEL`	Required, independently selected model IDs

Outputs are written under the task's output_dir:

skeletons.jsonl is written before any LLM call and includes gold answers;
results.jsonl contains rendered problems, predictions, and per-case scores;
summary.json contains aggregate accuracy and complexity metadata.

Gitea Actions one-shot test

The workflow .gitea/workflows/short-eval.yml has only workflow_dispatch, so it consumes model quota only when someone manually clicks Run workflow. It runs three cases with length: short and has a 15-minute timeout.

Create these repository Actions settings:

Secret OPENAI_API_KEY, or separate secrets RENDERER_API_KEY and SOLVER_API_KEY.
Variables RENDERER_MODEL and SOLVER_MODEL.
Optional variables RENDERER_BASE_URL, SOLVER_BASE_URL, RENDERER_API_STYLE, and SOLVER_API_STYLE. Leave base URLs empty for OpenAI; use responses as the API style unless a compatible provider only supports chat completions.

The workflow targets runs-on: logic-bench. Add this label to the existing act_runner using one of these methods:

In the Gitea UI, open the runner settings, edit its labels, and add logic-bench:docker://docker.gitea.com/runner-images:ubuntu-latest.
Or add the same string to runner.labels in the runner's config.yaml, preserving existing labels, then restart act_runner with that config.
Or start the daemon with --labels / GITEA_RUNNER_LABELS; this has higher precedence than the config file and must include every label the runner should continue advertising.

Example config fragment:

runner:
  labels:
    - "logic-bench:docker://docker.gitea.com/runner-images:ubuntu-latest"

After restarting the runner, verify that logic-bench appears in its Gitea settings, then manually dispatch Short abstraction evaluation from the repository's Actions page.

Tests

python3 -m unittest discover -s tests -v