80 lines
4.5 KiB
Markdown
80 lines
4.5 KiB
Markdown
# Logic Skeleton Bench
|
|
|
|
Logic Skeleton Bench tests whether a solver transfers the same abstract structure across different surface stories. It first creates a typed intermediate representation with a deterministic gold answer, asks one LLM to render that representation as a natural-language problem, asks another LLM to solve only the rendered problem, and then scores the answer.
|
|
|
|
The renderer and solver are intentionally separate clients. Each can use a different model, API endpoint, key, and OpenAI-compatible API style, so model-pair heatmaps can be added without changing the skeleton generator.
|
|
|
|
## What the first version contains
|
|
|
|
Three finite skeleton families are included:
|
|
|
|
- `transitive_order`: a chain of directional relations;
|
|
- `membership_inheritance`: membership propagated through subset relations;
|
|
- `sequential_state_update`: ordered arithmetic state changes.
|
|
|
|
Every generated skeleton contains its facts, rules, query, permitted answer labels, deterministic `gold_answer`, and proof trace. The renderer receives a copy with `gold_answer` and `proof` removed.
|
|
|
|
The full field grammar, predicate signatures, generator invariants, and renderer boundary are defined in [the Skeleton IR specification](docs/skeleton-language.md).
|
|
|
|
`length` means logical complexity, not natural-language tokens. It is the number of proof-bearing facts or state updates. Presets are `short=2`, `medium=4`, and `long=6`; an integer from 1 through 20 is also accepted.
|
|
|
|
## Local use
|
|
|
|
Python 3.11 or newer is sufficient; the package has no runtime dependencies.
|
|
|
|
Generate three short skeletons without calling an LLM:
|
|
|
|
```bash
|
|
python3 -m logicbench generate --count 3 --length short --seed 20260922
|
|
```
|
|
|
|
Run the checked-in live task:
|
|
|
|
```bash
|
|
export LLM_API_KEY='...'
|
|
export LLM_BASE_URL='your-provider-compatible-base-url'
|
|
export RENDERER_MODEL='your-renderer-model'
|
|
export SOLVER_MODEL='your-solver-model'
|
|
python3 -m logicbench run tasks/short-smoke.json
|
|
```
|
|
|
|
Role-specific values override the shared fallback:
|
|
|
|
| Variable | Meaning |
|
|
| --- | --- |
|
|
| `LLM_API_KEY` | Optional shared key when both roles use the same account/provider |
|
|
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Optional role-specific keys; each takes precedence over `LLM_API_KEY` |
|
|
| `LLM_BASE_URL` | Optional shared OpenAI-compatible endpoint |
|
|
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Optional role-specific endpoints; each takes precedence over `LLM_BASE_URL` |
|
|
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` (default) or `chat_completions` |
|
|
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required, independently selected model IDs |
|
|
|
|
Outputs are written under the task's `output_dir`:
|
|
|
|
- `skeletons.jsonl` is written before any LLM call and includes gold answers;
|
|
- `results.jsonl` contains rendered problems, predictions, and per-case scores;
|
|
- `summary.json` contains aggregate accuracy and complexity metadata.
|
|
|
|
## Gitea Actions one-shot test
|
|
|
|
The workflow `.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when someone manually clicks **Run workflow**. It runs three cases with `length: short` and has a 15-minute timeout.
|
|
|
|
The run log prints the rendered questions and scores. The same records, original answer-bearing skeletons, and summary are retained in the `short-abstraction-evaluation` Actions artifact for inspection.
|
|
|
|
Create these repository Actions settings:
|
|
|
|
1. Secret `LLM_API_KEY` when both roles share credentials, or role-specific secrets `RENDERER_API_KEY` and `SOLVER_API_KEY` when they do not.
|
|
2. Variables `RENDERER_MODEL` and `SOLVER_MODEL`.
|
|
3. Variable `LLM_BASE_URL` for a shared provider endpoint, or role-specific variables `RENDERER_BASE_URL` and `SOLVER_BASE_URL` for different providers.
|
|
4. Optional variables `RENDERER_API_STYLE` and `SOLVER_API_STYLE`, each set to `responses` or `chat_completions` according to that provider/model.
|
|
|
|
The workflow targets the existing `minecraft-and-affine-server` runner label. Verify that a runner advertising this label is online, then manually dispatch **Short abstraction evaluation** from the repository's Actions page.
|
|
|
|
For a current all-Qwen run, create only the shared `LLM_API_KEY` secret with the Model Studio key. Set `LLM_BASE_URL` to the API Host shown for that key and region, set the two model variables independently, and use `chat_completions` for both API-style variables. For example, a fast renderer and stronger solver can use `qwen3.8-flash` and `qwen3.8-max`; both currently support JSON Schema structured output. Do not hard-code the DashScope key or endpoint in the workflow.
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
python3 -m unittest discover -s tests -v
|
|
```
|