logic-skeleton-bench/README.md

Logic Skeleton Bench

Logic Skeleton Bench tests whether a solver transfers the same abstract structure across different surface stories. It first creates a typed intermediate representation with a deterministic gold answer, asks one LLM to render that representation as a natural-language problem, asks another LLM to solve only the rendered problem, and then scores the answer.

The renderer and solver are intentionally separate clients. Each can use a different model, API endpoint, key, and OpenAI-compatible API style, so model-pair heatmaps can be added without changing the skeleton generator.

What the first version contains

Three finite skeleton families are included:

• transitive_order: a chain of directional relations;
• membership_inheritance: membership propagated through subset relations;
• sequential_state_update: ordered arithmetic state changes.

Every generated skeleton contains its facts, rules, query, permitted answer labels, deterministic gold_answer, and proof trace. The renderer receives a copy with gold_answer and proof removed.

The full field grammar, predicate signatures, generator invariants, and renderer boundary are defined in the Skeleton IR specification.

length means logical complexity, not natural-language tokens. It is the number of proof-bearing facts or state updates. Presets are short=2, medium=4, and long=6; an integer from 1 through 20 is also accepted.

Local use

Python 3.11 or newer is sufficient; the package has no runtime dependencies.

Generate three short skeletons without calling an LLM:

python3 -m logicbench generate --count 3 --length short --seed 20260922

Run the checked-in live task:

export RENDERER_API_KEY='...'
export RENDERER_BASE_URL='your-renderer-compatible-base-url'
export RENDERER_MODEL='your-renderer-model'
export SOLVER_API_KEY='...'
export SOLVER_BASE_URL='your-solver-compatible-base-url'
export SOLVER_MODEL='your-solver-model'
python3 -m logicbench run tasks/short-smoke.json

Role-specific values override the shared fallback:

Variable	Meaning
RENDERER_API_KEY, SOLVER_API_KEY	Required independent credentials; values may currently be identical
RENDERER_BASE_URL, SOLVER_BASE_URL	Required independent OpenAI-compatible endpoints; values may currently be identical
RENDERER_API_STYLE, SOLVER_API_STYLE	responses (default) or chat_completions
RENDERER_MODEL, SOLVER_MODEL	Required, independently selected model IDs

Outputs are written under the task's output_dir:

• skeletons.jsonl is written before any LLM call and includes gold answers;
• results.jsonl contains rendered problems, predictions, and per-case scores;
• summary.json contains aggregate accuracy and complexity metadata.

Gitea Actions one-shot test

The workflow .gitea/workflows/short-eval.yml has only workflow_dispatch, so it consumes model quota only when someone manually clicks Run workflow. It runs five cases with length: short and has a 15-minute timeout.

The run log prints every raw answer-bearing skeleton followed by each rendered question, the solver's complete text, its parsed #### <answer> label, the gold answer, and correctness. The same records and summary are retained in the short-abstraction-evaluation Actions artifact for inspection.

Create these repository Actions settings:

1. Secrets RENDERER_API_KEY and SOLVER_API_KEY.
2. Variables RENDERER_BASE_URL, RENDERER_MODEL, and RENDERER_API_STYLE.
3. Variables SOLVER_BASE_URL, SOLVER_MODEL, and SOLVER_API_STYLE.

The workflow targets the existing minecraft-and-affine-server runner label. Verify that a runner advertising this label is online, then manually dispatch Short abstraction evaluation from the repository's Actions page.

For a current all-Qwen run, put the Model Studio key into both role-specific secrets and its API Host into both role-specific base-URL variables. The values may be identical today, but the two client boundaries remain explicit. Use chat_completions for both API-style variables. For example, a fast renderer and stronger solver can use qwen3.8-flash and qwen3.8-max; both currently support JSON Schema structured output. Do not hard-code keys or endpoints in the workflow.

Tests

python3 -m unittest discover -s tests -v