88 lines
4.4 KiB
Markdown
88 lines
4.4 KiB
Markdown
# Logic Skeleton Bench
|
|
|
|
Logic Skeleton Bench tests whether a solver transfers the same abstract structure across different surface stories. It first creates a typed intermediate representation with a deterministic gold answer, asks one LLM to render that representation as a natural-language problem, asks another LLM to solve only the rendered problem, and then scores the answer.
|
|
|
|
The renderer and solver are intentionally separate clients. Each can use a different model, API endpoint, key, and OpenAI-compatible API style, so model-pair heatmaps can be added without changing the skeleton generator.
|
|
|
|
## What the first version contains
|
|
|
|
Three finite skeleton families are included:
|
|
|
|
- `transitive_order`: a chain of directional relations;
|
|
- `membership_inheritance`: membership propagated through subset relations;
|
|
- `sequential_state_update`: ordered arithmetic state changes.
|
|
|
|
Every generated skeleton contains its facts, rules, query, permitted answer labels, deterministic `gold_answer`, and proof trace. The renderer receives a copy with `gold_answer` and `proof` removed.
|
|
|
|
The full field grammar, predicate signatures, generator invariants, and renderer boundary are defined in [the Skeleton IR specification](docs/skeleton-language.md).
|
|
|
|
`length` means logical complexity, not natural-language tokens. It is the number of proof-bearing facts or state updates. Presets are `short=2`, `medium=4`, and `long=6`; an integer from 1 through 20 is also accepted.
|
|
|
|
## Local use
|
|
|
|
Python 3.11 or newer is sufficient; the package has no runtime dependencies.
|
|
|
|
Generate three short skeletons without calling an LLM:
|
|
|
|
```bash
|
|
python3 -m logicbench generate --count 3 --length short --seed 20260922
|
|
```
|
|
|
|
Run the checked-in live task:
|
|
|
|
```bash
|
|
export OPENAI_API_KEY='...'
|
|
export RENDERER_MODEL='your-renderer-model'
|
|
export SOLVER_MODEL='your-solver-model'
|
|
python3 -m logicbench run tasks/short-smoke.json
|
|
```
|
|
|
|
Role-specific values override the shared fallback:
|
|
|
|
| Variable | Meaning |
|
|
| --- | --- |
|
|
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Optional separate keys; each falls back to `OPENAI_API_KEY` |
|
|
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Optional endpoints; each falls back to `OPENAI_BASE_URL`, then `https://api.openai.com/v1` |
|
|
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` (default) or `chat_completions` |
|
|
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required, independently selected model IDs |
|
|
|
|
Outputs are written under the task's `output_dir`:
|
|
|
|
- `skeletons.jsonl` is written before any LLM call and includes gold answers;
|
|
- `results.jsonl` contains rendered problems, predictions, and per-case scores;
|
|
- `summary.json` contains aggregate accuracy and complexity metadata.
|
|
|
|
## Gitea Actions one-shot test
|
|
|
|
The workflow `.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when someone manually clicks **Run workflow**. It runs three cases with `length: short` and has a 15-minute timeout.
|
|
|
|
The run log prints the rendered questions and scores. The same records, original answer-bearing skeletons, and summary are retained in the `short-abstraction-evaluation` Actions artifact for inspection.
|
|
|
|
Create these repository Actions settings:
|
|
|
|
1. Secret `OPENAI_API_KEY`, or separate secrets `RENDERER_API_KEY` and `SOLVER_API_KEY`.
|
|
2. Variables `RENDERER_MODEL` and `SOLVER_MODEL`.
|
|
3. Optional variables `RENDERER_BASE_URL`, `SOLVER_BASE_URL`, `RENDERER_API_STYLE`, and `SOLVER_API_STYLE`. Leave base URLs empty for OpenAI; use `responses` as the API style unless a compatible provider only supports chat completions.
|
|
|
|
The workflow targets `runs-on: logic-bench`. Add this label to the existing `act_runner` using one of these methods:
|
|
|
|
- In the Gitea UI, open the runner settings, edit its labels, and add `logic-bench:docker://docker.gitea.com/runner-images:ubuntu-latest`.
|
|
- Or add the same string to `runner.labels` in the runner's `config.yaml`, preserving existing labels, then restart `act_runner` with that config.
|
|
- Or start the daemon with `--labels` / `GITEA_RUNNER_LABELS`; this has higher precedence than the config file and must include every label the runner should continue advertising.
|
|
|
|
Example config fragment:
|
|
|
|
```yaml
|
|
runner:
|
|
labels:
|
|
- "logic-bench:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
|
```
|
|
|
|
After restarting the runner, verify that `logic-bench` appears in its Gitea settings, then manually dispatch **Short abstraction evaluation** from the repository's Actions page.
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
python3 -m unittest discover -s tests -v
|
|
```
|