The generator now creates an executable hidden stochastic system, runs controlled experiments against it, and builds a behavior skeleton containing only inputs, ordered events, and observed final outcomes. A renderer turns that evidence into a coherent natural-language world. The solver sees the rendered histories and predicts which outcome is more likely for a held-out event combination. It never sees the hidden rules or their probabilities.

The renderer and solver remain independent clients. Each can use a different model, endpoint, key, and OpenAI-compatible API style for future model-pair heatmaps.

Current task model

Every trial starts from the same empty state. Hidden rules can:

trigger on an event;
test an input attribute or an internal state flag;
fire with a hidden probability;
add a record, permanently invalidate it, or set a flag used by a later event.

An invalidation is a persistent veto, so probabilistically triggered actions can conflict. Historical cases report aggregate PRESENT/ABSENT counts from repeated trials. The query uses a new composition of events and asks only which outcome is more likely—not for the exact probability.

The local evaluator computes the true query probability by exact enumeration of all stochastic branches. Random simulation is used only to create the finite observations shown to the model. Both are reproducible from the task seed.

The complete hidden-system language, execution semantics, public behavior format, and non-leakage boundary are specified in the behavioral IR specification.

length is structural complexity, not a natural-language token target. It is the number of distinct ordered query events and hidden event rules. Presets are short=4, medium=6, and long=8; explicit integers from 4 through 10 are accepted.

Local use

Python 3.11 or newer is sufficient; the package has no runtime dependencies.

Generate three short, answer-bearing records without calling an LLM:

python3 -m logicbench generate --count 3 --length short --seed 20260922

Run the checked-in live task:

export RENDERER_API_KEY='...'
export RENDERER_BASE_URL='your-renderer-compatible-base-url'
export RENDERER_MODEL='your-renderer-model'
export SOLVER_API_KEY='...'
export SOLVER_BASE_URL='your-solver-compatible-base-url'
export SOLVER_MODEL='your-solver-model'
python3 -m logicbench run tasks/short-smoke.json

Variable	Meaning
`RENDERER_API_KEY`, `SOLVER_API_KEY`	Required independent credentials; values may be identical
`RENDERER_BASE_URL`, `SOLVER_BASE_URL`	Required independent OpenAI-compatible endpoints
`RENDERER_API_STYLE`, `SOLVER_API_STYLE`	`responses` (default) or `chat_completions`
`RENDERER_MODEL`, `SOLVER_MODEL`	Required, independently selected model IDs
`RENDERER_TIMEOUT_SECONDS`, `SOLVER_TIMEOUT_SECONDS`	Optional per-request timeout; default `240`

Outputs under the task's output_dir are:

skeletons.jsonl: full hidden programs, sampled behavior, exact probabilities, gold labels, and latent traces; written before any LLM call;
results.jsonl: rendered problems, complete solver responses, parsed labels, gold probabilities, and scores;
summary.json: aggregate accuracy and structural metadata.

Gitea Actions one-shot test

.gitea/workflows/short-eval.yml has only workflow_dispatch, so it consumes model quota only when manually started. It targets the minecraft-and-affine-server runner label, runs unit tests and five short live cases, prints the auditable records, and uploads short-abstraction-evaluation.

Configure repository Actions with:

Secrets RENDERER_API_KEY and SOLVER_API_KEY.
Variables RENDERER_BASE_URL, RENDERER_MODEL, and RENDERER_API_STYLE.
Variables SOLVER_BASE_URL, SOLVER_MODEL, and SOLVER_API_STYLE.

For Qwen's OpenAI-compatible API, use the compatible-mode /v1 endpoint and chat_completions. Keys, URLs, and models remain role-specific even when both roles currently use the same provider.

Tests

python3 -m unittest discover -s tests -v