Zayden-Jung/logic-skeleton-bench
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Generate proof-bearing dynamic world histories

Browse Source
This commit is contained in:
Codex Bot
2026-09-23 10:23:09 +08:00
parent 6f929437a2
commit 43aff1e2ee
13 changed files with 1023 additions and 550 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+63 -37
View File
@@ -1,78 +1,104 @@
# Logic Skeleton Bench
Logic Skeleton Bench tests whether a model can infer an abstract mechanism from behavior rather than solve a rule statement with renamed nouns.
Logic Skeleton Bench generates large, evolving fictional worlds and tests whether a model can recover a logical consequence from their natural-language histories.
The generator now creates an executable **hidden stochastic system**, runs controlled experiments against it, and builds a **behavior skeleton** containing only inputs, ordered events, and observed final outcomes. A renderer turns that evidence into a coherent natural-language world. The solver sees the rendered histories and predicts which outcome is more likely for a held-out event combination. It never sees the hidden rules or their probabilities.
The benchmark has five stages:
The renderer and solver remain independent clients. Each can use a different model, endpoint, key, and OpenAI-compatible API style for future model-pair heatmaps.
```text
hidden typed program
→ initial world with many entities
→ seeded stochastic evolution
→ deterministic realized event history and gold proof
→ natural-language rendering and solving
```
## Current task model
Probability is used only to decide which candidate events actually occur. Once a seed has produced a history, the question and answer are deterministic. The solver never estimates a probability and never sees repeated-trial frequency tables.
Every trial starts from the same empty state. Hidden rules can:
## Current world model
- trigger on an event;
- test an input attribute or an internal state flag;
- fire with a hidden probability;
- add a record, permanently invalidate it, or set a flag used by a later event.
The initial implementation mixes several logic domains in every world:
An invalidation is a persistent veto, so probabilistically triggered actions can conflict. Historical cases report aggregate `PRESENT`/`ABSENT` counts from repeated trials. The query uses a new composition of events and asks only which outcome is more likely—not for the exact probability.
- directed spatial order (`ABOVE`/`BELOW`);
- membership, subset inheritance, and disjointness;
- locations and movement;
- symmetric connectivity;
- sequential numeric updates;
- Boolean state changes.
The local evaluator computes the true query probability by exact enumeration of all stochastic branches. Random simulation is used only to create the finite observations shown to the model. Both are reproducible from the task seed.
The hidden program contains typed relations, operators, quantifiers, inference laws, probabilistic transition rules, and their preconditions. A default short world contains 128 entities and runs for 1000 simulation steps, producing roughly 400 realized events with the checked-in seed.
The complete hidden-system language, execution semantics, public behavior format, and non-leakage boundary are specified in [the behavioral IR specification](docs/skeleton-language.md).
After evolution, the generator searches the realized state for a query with a shortest proof at least as long as the configured `length`. It rejects directly stated answers and stores the exact proof event IDs internally. The renderer receives all public initial observations and all realized events, but not the program, probabilities, hidden ranks, proof, generator checks, or gold answer.
`length` is structural complexity, not a natural-language token target. It is the number of distinct ordered query events and hidden event rules. Presets are `short=4`, `medium=6`, and `long=8`; explicit integers from 4 through 10 are accepted.
The full grammar and execution contract are defined in [the dynamic-world IR specification](docs/skeleton-language.md).
## Complexity controls
`length` controls minimum proof length, not entity count, event count, or natural-language tokens:
| Preset | Minimum proof events |
| --- | ---: |
| `short` | 4 |
| `medium` | 6 |
| `long` | 8 |
`entity_count` and `simulation_steps` are separate task fields. Explicit proof lengths from 3 through 10 are accepted.
## Local use
Python 3.11 or newer is sufficient; the package has no runtime dependencies.
Python 3.11 or newer is sufficient; there are no runtime dependencies.
Generate three short, answer-bearing records without calling an LLM:
Generate an answer-bearing world without calling an LLM:
```bash
python3 -m logicbench generate --count 3 --length short --seed 20260922
python3 -m logicbench generate \
--count 1 \
--length short \
--entity-count 128 \
--simulation-steps 1000 \
--seed 20260922
```
Run the checked-in live task:
```bash
export RENDERER_API_KEY='...'
export RENDERER_BASE_URL='your-renderer-compatible-base-url'
export RENDERER_MODEL='your-renderer-model'
export RENDERER_BASE_URL='...'
export RENDERER_MODEL='...'
export SOLVER_API_KEY='...'
export SOLVER_BASE_URL='your-solver-compatible-base-url'
export SOLVER_MODEL='your-solver-model'
export SOLVER_BASE_URL='...'
export SOLVER_MODEL='...'
python3 -m logicbench run tasks/short-smoke.json
```
| Variable | Meaning |
| --- | --- |
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials; values may be identical |
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials |
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Required independent OpenAI-compatible endpoints |
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` (default) or `chat_completions` |
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required, independently selected model IDs |
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` or `chat_completions` |
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required independent model IDs |
| `RENDERER_TIMEOUT_SECONDS`, `SOLVER_TIMEOUT_SECONDS` | Optional per-request timeout; default `240` |
| `RENDERER_ENABLE_THINKING`, `SOLVER_ENABLE_THINKING` | Optional `true`/`false` provider extension; omitted unless set |
| `RENDERER_ENABLE_THINKING`, `SOLVER_ENABLE_THINKING` | Optional provider extension; omitted unless set |
Outputs under the task's `output_dir` are:
The renderer and solver remain independent clients for model-pair heatmaps. The checked-in Qwen workflow explicitly disables solver thinking to keep non-streaming test latency bounded.
- `skeletons.jsonl`: full hidden programs, sampled behavior, exact probabilities, gold labels, and latent traces; written before any LLM call;
- `rendered.jsonl`: each natural-language problem, checkpointed immediately after rendering;
- `solver-outputs.jsonl`: each complete raw solver response, checkpointed before parsing;
- `results.jsonl`: rendered problems, complete solver responses, parsed labels, parse errors, format compliance, gold probabilities, and scores;
- `summary.json`: aggregate accuracy and structural metadata.
## Artifacts
## Gitea Actions one-shot test
Each run writes:
`.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when manually started. It targets the `minecraft-and-affine-server` runner label, runs unit tests and five short live cases, prints the auditable records, and uploads `short-abstraction-evaluation`.
- `skeletons.jsonl`: hidden program, entities, hidden initial and final states, complete event trace, query, gold answer, proof, and anti-leakage checks;
- `rendered.jsonl`: natural-language worlds, checkpointed immediately after rendering;
- `solver-outputs.jsonl`: complete raw solver responses, checkpointed before parsing;
- `results.jsonl`: parsed labels, parse errors, format compliance, and correctness;
- `summary.json`: aggregate accuracy, parsing metrics, scale, and per-world event counts.
Configure repository Actions with:
## Gitea Actions
1. Secrets `RENDERER_API_KEY` and `SOLVER_API_KEY`.
2. Variables `RENDERER_BASE_URL`, `RENDERER_MODEL`, and `RENDERER_API_STYLE`.
3. Variables `SOLVER_BASE_URL`, `SOLVER_MODEL`, and `SOLVER_API_STYLE`.
`.gitea/workflows/short-eval.yml` is manual-only (`workflow_dispatch`) and targets `minecraft-and-affine-server`. It runs unit tests followed by five large-world cases. Because rendering hundreds of events is substantially slower than the former toy benchmark, the job timeout is 45 minutes.
For Qwen's OpenAI-compatible API, use the compatible-mode `/v1` endpoint and `chat_completions`. Keys, URLs, and models remain role-specific even when both roles currently use the same provider. The checked-in short workflow explicitly disables solver thinking to avoid long non-streaming requests; remove or change that workflow value when thinking-mode performance is itself under test.
Repository Actions settings remain two secrets and six role-specific variables:
- secrets: `RENDERER_API_KEY`, `SOLVER_API_KEY`;
- variables: each role's `BASE_URL`, `MODEL`, and `API_STYLE`.
## Tests