# Logic Skeleton Bench

Logic Skeleton Bench tests whether a solver transfers the same abstract structure across different surface stories. It first creates a typed intermediate representation with a deterministic gold answer, asks one LLM to render that representation as a natural-language problem, asks another LLM to solve only the rendered problem, and then scores the answer.

The renderer and solver are intentionally separate clients. Each can use a different model, API endpoint, key, and OpenAI-compatible API style, so model-pair heatmaps can be added without changing the skeleton generator.

## What the first version contains

Three finite skeleton families are included:

- `transitive_order`: a chain of directional relations;
- `membership_inheritance`: membership propagated through subset relations;
- `sequential_state_update`: ordered arithmetic state changes.

Every generated skeleton contains its facts, rules, query, permitted answer labels, deterministic `gold_answer`, and proof trace. The renderer receives a copy with `gold_answer` and `proof` removed.

`length` means logical complexity, not natural-language tokens. It is the number of proof-bearing facts or state updates. Presets are `short=2`, `medium=4`, and `long=6`; an integer from 1 through 20 is also accepted.

## Local use

Python 3.11 or newer is sufficient; the package has no runtime dependencies.

Generate three short skeletons without calling an LLM:

```bash
python3 -m logicbench generate --count 3 --length short --seed 20260922
```

Run the checked-in live task:

```bash
export OPENAI_API_KEY='...'
export RENDERER_MODEL='your-renderer-model'
export SOLVER_MODEL='your-solver-model'
python3 -m logicbench run tasks/short-smoke.json
```

Role-specific values override the shared fallback:

| Variable | Meaning |
| --- | --- |
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Optional separate keys; each falls back to `OPENAI_API_KEY` |
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Optional endpoints; each falls back to `OPENAI_BASE_URL`, then `https://api.openai.com/v1` |
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` (default) or `chat_completions` |
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required, independently selected model IDs |

Outputs are written under the task's `output_dir`:

- `skeletons.jsonl` is written before any LLM call and includes gold answers;
- `results.jsonl` contains rendered problems, predictions, and per-case scores;
- `summary.json` contains aggregate accuracy and complexity metadata.

## Gitea Actions one-shot test

The workflow `.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when someone manually clicks **Run workflow**. It runs three cases with `length: short` and has a 15-minute timeout.

Create these repository Actions settings:

1. Secret `OPENAI_API_KEY`, or separate secrets `RENDERER_API_KEY` and `SOLVER_API_KEY`.
2. Variables `RENDERER_MODEL` and `SOLVER_MODEL`.
3. Optional variables `RENDERER_BASE_URL`, `SOLVER_BASE_URL`, `RENDERER_API_STYLE`, and `SOLVER_API_STYLE`. Leave base URLs empty for OpenAI; use `responses` as the API style unless a compatible provider only supports chat completions.

The workflow targets `runs-on: logic-bench`. Add this label to the existing `act_runner` using one of these methods:

- In the Gitea UI, open the runner settings, edit its labels, and add `logic-bench:docker://docker.gitea.com/runner-images:ubuntu-latest`.
- Or add the same string to `runner.labels` in the runner's `config.yaml`, preserving existing labels, then restart `act_runner` with that config.
- Or start the daemon with `--labels` / `GITEA_RUNNER_LABELS`; this has higher precedence than the config file and must include every label the runner should continue advertising.

Example config fragment:

```yaml
runner:
  labels:
    - "logic-bench:docker://docker.gitea.com/runner-images:ubuntu-latest"
```

After restarting the runner, verify that `logic-bench` appears in its Gitea settings, then manually dispatch **Short abstraction evaluation** from the repository's Actions page.

## Tests

```bash
python3 -m unittest discover -s tests -v
```