# Logic Skeleton Bench

Logic Skeleton Bench tests whether a solver transfers the same abstract structure across different surface stories. It first creates a typed intermediate representation with a deterministic gold answer, asks one LLM to render that representation as a natural-language problem, asks another LLM to solve only the rendered problem, and then scores the answer.

The renderer and solver are intentionally separate clients. Each can use a different model, API endpoint, key, and OpenAI-compatible API style, so model-pair heatmaps can be added without changing the skeleton generator.

## What the first version contains

Three finite skeleton families are included:

- `transitive_order`: a chain of directional relations;
- `membership_inheritance`: membership propagated through subset relations;
- `sequential_state_update`: ordered arithmetic state changes.

Every generated skeleton contains its facts, rules, query, permitted answer labels, deterministic `gold_answer`, and proof trace. The renderer receives a copy with `gold_answer` and `proof` removed.

The full field grammar, predicate signatures, generator invariants, and renderer boundary are defined in [the Skeleton IR specification](docs/skeleton-language.md).

`length` means logical complexity, not natural-language tokens. It is the number of proof-bearing facts or state updates. Presets are `short=2`, `medium=4`, and `long=6`; an integer from 1 through 20 is also accepted.

## Local use

Python 3.11 or newer is sufficient; the package has no runtime dependencies.

Generate three short skeletons without calling an LLM:

```bash
python3 -m logicbench generate --count 3 --length short --seed 20260922
```

Run the checked-in live task:

```bash
export RENDERER_API_KEY='...'
export RENDERER_BASE_URL='your-renderer-compatible-base-url'
export RENDERER_MODEL='your-renderer-model'
export SOLVER_API_KEY='...'
export SOLVER_BASE_URL='your-solver-compatible-base-url'
export SOLVER_MODEL='your-solver-model'
python3 -m logicbench run tasks/short-smoke.json
```

Role-specific values override the shared fallback:

| Variable | Meaning |
| --- | --- |
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials; values may currently be identical |
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Required independent OpenAI-compatible endpoints; values may currently be identical |
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` (default) or `chat_completions` |
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required, independently selected model IDs |

Outputs are written under the task's `output_dir`:

- `skeletons.jsonl` is written before any LLM call and includes gold answers;
- `results.jsonl` contains rendered problems, predictions, and per-case scores;
- `summary.json` contains aggregate accuracy and complexity metadata.

## Gitea Actions one-shot test

The workflow `.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when someone manually clicks **Run workflow**. It runs five cases with `length: short` and has a 15-minute timeout.

The run log prints every raw answer-bearing skeleton followed by each rendered question, the solver's complete text, its parsed `#### <answer>` label, the gold answer, and correctness. The same records and summary are retained in the `short-abstraction-evaluation` Actions artifact for inspection.

Create these repository Actions settings:

1. Secrets `RENDERER_API_KEY` and `SOLVER_API_KEY`.
2. Variables `RENDERER_BASE_URL`, `RENDERER_MODEL`, and `RENDERER_API_STYLE`.
3. Variables `SOLVER_BASE_URL`, `SOLVER_MODEL`, and `SOLVER_API_STYLE`.

The workflow targets the existing `minecraft-and-affine-server` runner label. Verify that a runner advertising this label is online, then manually dispatch **Short abstraction evaluation** from the repository's Actions page.

For a current all-Qwen run, put the Model Studio key into both role-specific secrets and its API Host into both role-specific base-URL variables. The values may be identical today, but the two client boundaries remain explicit. Use `chat_completions` for both API-style variables. For example, a fast renderer and stronger solver can use `qwen3.8-flash` and `qwen3.8-max`; both currently support JSON Schema structured output. Do not hard-code keys or endpoints in the workflow.

## Tests

```bash
python3 -m unittest discover -s tests -v
```