# Logic Skeleton Bench

Logic Skeleton Bench tests whether a solver transfers the same abstract structure across different surface stories. It first creates a typed intermediate representation with a deterministic gold answer, asks one LLM to render that representation as a natural-language problem, asks another LLM to solve only the rendered problem, and then scores the answer.

The renderer and solver are intentionally separate clients. Each can use a different model, API endpoint, key, and OpenAI-compatible API style, so model-pair heatmaps can be added without changing the skeleton generator.

## What the first version contains

Three finite skeleton families are included:

- `transitive_order`: a chain of directional relations;
- `membership_inheritance`: membership propagated through subset relations;
- `sequential_state_update`: ordered arithmetic state changes.

Every generated skeleton contains its facts, rules, query, permitted answer labels, deterministic `gold_answer`, and proof trace. The renderer receives a copy with `gold_answer` and `proof` removed.

The full field grammar, predicate signatures, generator invariants, and renderer boundary are defined in [the Skeleton IR specification](docs/skeleton-language.md).

`length` means logical complexity, not natural-language tokens. It is the number of proof-bearing facts or state updates. Presets are `short=2`, `medium=4`, and `long=6`; an integer from 1 through 20 is also accepted.

## Local use

Python 3.11 or newer is sufficient; the package has no runtime dependencies.

Generate three short skeletons without calling an LLM:

```bash
python3 -m logicbench generate --count 3 --length short --seed 20260922
```

Run the checked-in live task:

```bash
export LLM_API_KEY='...'
export LLM_BASE_URL='your-provider-compatible-base-url'
export RENDERER_MODEL='your-renderer-model'
export SOLVER_MODEL='your-solver-model'
python3 -m logicbench run tasks/short-smoke.json
```

Role-specific values override the shared fallback:

| Variable | Meaning |
| --- | --- |
| `LLM_API_KEY` | Optional shared key when both roles use the same account/provider |
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Optional role-specific keys; each takes precedence over `LLM_API_KEY` |
| `LLM_BASE_URL` | Optional shared OpenAI-compatible endpoint |
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Optional role-specific endpoints; each takes precedence over `LLM_BASE_URL` |
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` (default) or `chat_completions` |
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required, independently selected model IDs |

Outputs are written under the task's `output_dir`:

- `skeletons.jsonl` is written before any LLM call and includes gold answers;
- `results.jsonl` contains rendered problems, predictions, and per-case scores;
- `summary.json` contains aggregate accuracy and complexity metadata.

## Gitea Actions one-shot test

The workflow `.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when someone manually clicks **Run workflow**. It runs three cases with `length: short` and has a 15-minute timeout.

The run log prints the rendered questions and scores. The same records, original answer-bearing skeletons, and summary are retained in the `short-abstraction-evaluation` Actions artifact for inspection.

Create these repository Actions settings:

1. Secret `LLM_API_KEY` when both roles share credentials, or role-specific secrets `RENDERER_API_KEY` and `SOLVER_API_KEY` when they do not.
2. Variables `RENDERER_MODEL` and `SOLVER_MODEL`.
3. Variable `LLM_BASE_URL` for a shared provider endpoint, or role-specific variables `RENDERER_BASE_URL` and `SOLVER_BASE_URL` for different providers.
4. Optional variables `RENDERER_API_STYLE` and `SOLVER_API_STYLE`, each set to `responses` or `chat_completions` according to that provider/model.

The workflow targets the existing `minecraft-and-affine-server` runner label. Verify that a runner advertising this label is online, then manually dispatch **Short abstraction evaluation** from the repository's Actions page.

For a current all-Qwen run, create only the shared `LLM_API_KEY` secret with the Model Studio key. Set `LLM_BASE_URL` to the API Host shown for that key and region, set the two model variables independently, and use `chat_completions` for both API-style variables. For example, a fast renderer and stronger solver can use `qwen3.8-flash` and `qwen3.8-max`; both currently support JSON Schema structured output. Do not hard-code the DashScope key or endpoint in the workflow.

## Tests

```bash
python3 -m unittest discover -s tests -v
```