# Logic Skeleton Bench

Logic Skeleton Bench generates large, evolving fictional worlds and tests whether a model can recover a logical consequence from their natural-language histories.

The benchmark has five stages:

```text
hidden typed program
  → initial world with many entities
  → seeded stochastic evolution
  → deterministic realized event history and gold proof
  → natural-language rendering and solving
```

Probability is used only to decide which candidate events actually occur. Once a seed has produced a history, the question and answer are deterministic. The solver never estimates a probability and never sees repeated-trial frequency tables.

## Current world model

The initial implementation mixes several logic domains in every world:

- directed spatial order (`ABOVE`/`BELOW`);
- membership, subset inheritance, and disjointness;
- locations and movement;
- symmetric connectivity;
- sequential numeric updates;
- Boolean state changes.

The hidden program contains typed relations, operators, quantifiers, inference laws, probabilistic transition rules, and their preconditions. A default short world contains 128 entities and runs for 1000 simulation steps, producing roughly 400 realized events with the checked-in seed.

After evolution, the generator searches the realized state for a query with a shortest proof at least as long as the configured `length`. It rejects directly stated answers and stores the exact proof event IDs internally. The renderer receives all public initial observations and all realized events, but not the program, probabilities, hidden ranks, proof, generator checks, or gold answer.

The full grammar and execution contract are defined in [the dynamic-world IR specification](docs/skeleton-language.md).

## Complexity controls

`length` controls minimum proof length, not entity count, event count, or natural-language tokens:

| Preset | Minimum proof events |
| --- | ---: |
| `short` | 4 |
| `medium` | 6 |
| `long` | 8 |

`entity_count` and `simulation_steps` are separate task fields. Explicit proof lengths from 3 through 10 are accepted.

## Local use

Python 3.11 or newer is sufficient; there are no runtime dependencies.

Generate an answer-bearing world without calling an LLM:

```bash
python3 -m logicbench generate \
  --count 1 \
  --length short \
  --entity-count 128 \
  --simulation-steps 1000 \
  --seed 20260922
```

Run the checked-in live task:

```bash
export RENDERER_API_KEY='...'
export RENDERER_BASE_URL='...'
export RENDERER_MODEL='...'
export SOLVER_API_KEY='...'
export SOLVER_BASE_URL='...'
export SOLVER_MODEL='...'
python3 -m logicbench run tasks/short-smoke.json
```

| Variable | Meaning |
| --- | --- |
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials |
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Required independent OpenAI-compatible endpoints |
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` or `chat_completions` |
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required independent model IDs |
| `RENDERER_TIMEOUT_SECONDS`, `SOLVER_TIMEOUT_SECONDS` | Optional per-request timeout; default `240` |
| `RENDERER_ENABLE_THINKING`, `SOLVER_ENABLE_THINKING` | Optional provider extension; omitted unless set |

The renderer and solver remain independent clients for model-pair heatmaps. The checked-in Qwen workflow explicitly disables solver thinking to keep non-streaming test latency bounded.

## Artifacts

Each run writes:

- `skeletons.jsonl`: hidden program, entities, hidden initial and final states, complete event trace, query, gold answer, proof, and anti-leakage checks;
- `rendered.jsonl`: natural-language worlds, checkpointed immediately after rendering;
- `solver-outputs.jsonl`: complete raw solver responses, checkpointed before parsing;
- `results.jsonl`: parsed labels, parse errors, format compliance, and correctness;
- `summary.json`: aggregate accuracy, parsing metrics, scale, and per-world event counts.

## Gitea Actions

`.gitea/workflows/short-eval.yml` is manual-only (`workflow_dispatch`) and targets `minecraft-and-affine-server`. It runs unit tests followed by five large-world cases. Because rendering hundreds of events is substantially slower than the former toy benchmark, the job timeout is 45 minutes.

Repository Actions settings remain two secrets and six role-specific variables:

- secrets: `RENDERER_API_KEY`, `SOLVER_API_KEY`;
- variables: each role's `BASE_URL`, `MODEL`, and `API_STYLE`.

## Tests

```bash
python3 -m unittest discover -s tests -v
```