Logic Skeleton Bench generates large, evolving fictional worlds and tests whether a model can recover a logical consequence from their natural-language histories.

The benchmark has five stages:

hidden typed program
  → initial world with many entities
  → seeded stochastic evolution
  → deterministic realized event history and gold proof
  → natural-language rendering and solving

Probability is used only to decide which candidate events actually occur. Once a seed has produced a history, the question and answer are deterministic. The solver never estimates a probability and never sees repeated-trial frequency tables.

Current world model

The initial implementation mixes several logic domains in every world:

directed spatial order (ABOVE/BELOW);
membership, subset inheritance, and disjointness;
locations and movement;
symmetric connectivity;
sequential numeric updates;
Boolean state changes.

The hidden program contains typed relations, operators, quantifiers, inference laws, probabilistic transition rules, and their preconditions. A default short world contains 128 entities and runs for 1000 simulation steps, producing roughly 400 realized events with the checked-in seed.

After evolution, the generator searches the realized state for a query with a shortest proof at least as long as the configured length. It rejects directly stated answers and stores the exact proof event IDs internally. The renderer receives all public initial observations and all realized events, but not the program, probabilities, hidden ranks, proof, generator checks, or gold answer.

The full grammar and execution contract are defined in the dynamic-world IR specification.

Complexity controls

length controls minimum proof length, not entity count, event count, or natural-language tokens:

Preset	Minimum proof events
`short`	4
`medium`	6
`long`	8

entity_count and simulation_steps are separate task fields. Explicit proof lengths from 3 through 10 are accepted.

render_chunk_size controls how many initial observations or realized events are sent in one renderer request. It defaults to 60; the renderer first creates one shared setting plan, and every chunk receives the same deterministic symbol-to-name map. It can be set in the task JSON or overridden at run time:

python3 -m logicbench run tasks/short-smoke.json --render-chunk-size 40

Local use

Python 3.11 or newer is sufficient; there are no runtime dependencies.

Generate an answer-bearing world without calling an LLM:

python3 -m logicbench generate \
  --count 1 \
  --length short \
  --entity-count 128 \
  --simulation-steps 1000 \
  --seed 20260922

Run the checked-in live task:

export RENDERER_API_KEY='...'
export RENDERER_BASE_URL='...'
export RENDERER_MODEL='...'
export SOLVER_API_KEY='...'
export SOLVER_BASE_URL='...'
export SOLVER_MODEL='...'
python3 -m logicbench run tasks/short-smoke.json

Variable	Meaning
`RENDERER_API_KEY`, `SOLVER_API_KEY`	Required independent credentials
`RENDERER_BASE_URL`, `SOLVER_BASE_URL`	Required independent OpenAI-compatible endpoints
`RENDERER_API_STYLE`, `SOLVER_API_STYLE`	`responses` or `chat_completions`
`RENDERER_MODEL`, `SOLVER_MODEL`	Required independent model IDs
`RENDERER_TIMEOUT_SECONDS`, `SOLVER_TIMEOUT_SECONDS`	Optional per-request timeout; default `240`
`RENDERER_ENABLE_THINKING`, `SOLVER_ENABLE_THINKING`	Optional provider extension; omitted unless set

The renderer and solver remain independent clients for model-pair heatmaps. The checked-in Qwen workflow explicitly disables solver thinking to keep non-streaming test latency bounded.

Artifacts

Each run writes:

skeletons.jsonl: hidden program, entities, hidden initial and final states, complete event trace, query, gold answer, proof, and anti-leakage checks;
rendered.jsonl: natural-language worlds, checkpointed immediately after rendering;
solver-outputs.jsonl: complete raw solver responses, checkpointed before parsing;
results.jsonl: parsed labels, parse errors, format compliance, and correctness;
summary.json: aggregate accuracy, parsing metrics, scale, and per-world event counts.

Gitea Actions

.gitea/workflows/short-eval.yml is manual-only (workflow_dispatch) and targets minecraft-and-affine-server. It runs unit tests followed by five large-world cases. Because rendering hundreds of events is substantially slower than the former toy benchmark, the job timeout is 45 minutes.

Repository Actions settings remain two secrets and six role-specific variables:

secrets: RENDERER_API_KEY, SOLVER_API_KEY;
variables: each role's BASE_URL, MODEL, and API_STYLE.

Tests

python3 -m unittest discover -s tests -v