Logic Skeleton Bench tests whether a solver transfers the same abstract structure across different surface stories. It first creates a typed intermediate representation with a deterministic gold answer, asks one LLM to render that representation as a natural-language problem, asks another LLM to solve only the rendered problem, and then scores the answer.

The renderer and solver are intentionally separate clients. Each can use a different model, API endpoint, key, and OpenAI-compatible API style, so model-pair heatmaps can be added without changing the skeleton generator.

What the first version contains

Three finite skeleton families are included:

transitive_order: a chain of directional relations;
membership_inheritance: membership propagated through subset relations;
sequential_state_update: ordered arithmetic state changes.

Every generated skeleton contains its facts, rules, query, permitted answer labels, deterministic gold_answer, and proof trace. The renderer receives a copy with gold_answer and proof removed.

The full field grammar, predicate signatures, generator invariants, and renderer boundary are defined in the Skeleton IR specification.

length means logical complexity, not natural-language tokens. It is the number of proof-bearing facts or state updates. Presets are short=2, medium=4, and long=6; an integer from 1 through 20 is also accepted.

Local use

Python 3.11 or newer is sufficient; the package has no runtime dependencies.

Generate three short skeletons without calling an LLM:

python3 -m logicbench generate --count 3 --length short --seed 20260922

Run the checked-in live task:

export LLM_API_KEY='...'
export LLM_BASE_URL='your-provider-compatible-base-url'
export RENDERER_MODEL='your-renderer-model'
export SOLVER_MODEL='your-solver-model'
python3 -m logicbench run tasks/short-smoke.json

Role-specific values override the shared fallback:

Variable	Meaning
`LLM_API_KEY`	Optional shared key when both roles use the same account/provider
`RENDERER_API_KEY`, `SOLVER_API_KEY`	Optional role-specific keys; each takes precedence over `LLM_API_KEY`
`LLM_BASE_URL`	Optional shared OpenAI-compatible endpoint
`RENDERER_BASE_URL`, `SOLVER_BASE_URL`	Optional role-specific endpoints; each takes precedence over `LLM_BASE_URL`
`RENDERER_API_STYLE`, `SOLVER_API_STYLE`	`responses` (default) or `chat_completions`
`RENDERER_MODEL`, `SOLVER_MODEL`	Required, independently selected model IDs

Outputs are written under the task's output_dir:

skeletons.jsonl is written before any LLM call and includes gold answers;
results.jsonl contains rendered problems, predictions, and per-case scores;
summary.json contains aggregate accuracy and complexity metadata.

Gitea Actions one-shot test

The workflow .gitea/workflows/short-eval.yml has only workflow_dispatch, so it consumes model quota only when someone manually clicks Run workflow. It runs three cases with length: short and has a 15-minute timeout.

The run log prints the rendered questions and scores. The same records, original answer-bearing skeletons, and summary are retained in the short-abstraction-evaluation Actions artifact for inspection.

Create these repository Actions settings:

Secret LLM_API_KEY when both roles share credentials, or role-specific secrets RENDERER_API_KEY and SOLVER_API_KEY when they do not.
Variables RENDERER_MODEL and SOLVER_MODEL.
Variable LLM_BASE_URL for a shared provider endpoint, or role-specific variables RENDERER_BASE_URL and SOLVER_BASE_URL for different providers.
Optional variables RENDERER_API_STYLE and SOLVER_API_STYLE, each set to responses or chat_completions according to that provider/model.

The workflow targets the existing minecraft-and-affine-server runner label. Verify that a runner advertising this label is online, then manually dispatch Short abstraction evaluation from the repository's Actions page.

For a current all-Qwen run, create only the shared LLM_API_KEY secret with the Model Studio key. Set LLM_BASE_URL to the API Host shown for that key and region, set the two model variables independently, and use chat_completions for both API-style variables. For example, a fast renderer and stronger solver can use qwen3.8-flash and qwen3.8-max; both currently support JSON Schema structured output. Do not hard-code the DashScope key or endpoint in the workflow.

Tests

python3 -m unittest discover -s tests -v