Zayden-Jung/logic-skeleton-bench
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace explicit rules with behavioral hidden systems

Browse Source
This commit is contained in:
Codex Bot
2026-09-23 00:10:37 +08:00
parent 3cf3db7b12
commit 9f54f8f5a3
12 changed files with 598 additions and 380 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/short-eval.yml
+2 -2
View File
@@ -1,4 +1,4 @@
name: Short abstraction evaluation
name: Short behavioral-system evaluation
on:
workflow_dispatch:
@@ -26,7 +26,7 @@ jobs:
- name: Run five short live cases
run: python3 -m logicbench run tasks/short-smoke.json
- name: Show raw skeletons, evaluation records, and summary
- name: Show hidden systems, behavior skeletons, results, and summary
if: always()
run: |
test -f artifacts/short-smoke/skeletons.jsonl && cat artifacts/short-smoke/skeletons.jsonl || true
README.md
+25 -26
View File
@@ -1,28 +1,33 @@
# Logic Skeleton Bench
Logic Skeleton Bench tests whether a solver transfers the same abstract structure across different surface stories. It first creates a typed intermediate representation with a deterministic gold answer, asks one LLM to render that representation as a natural-language problem, asks another LLM to solve only the rendered problem, and then scores the answer.
Logic Skeleton Bench tests whether a model can infer an abstract mechanism from behavior rather than solve a rule statement with renamed nouns.
The renderer and solver are intentionally separate clients. Each can use a different model, API endpoint, key, and OpenAI-compatible API style, so model-pair heatmaps can be added without changing the skeleton generator.
The generator now creates an executable **hidden stochastic system**, runs controlled experiments against it, and builds a **behavior skeleton** containing only inputs, ordered events, and observed final outcomes. A renderer turns that evidence into a coherent natural-language world. The solver sees the rendered histories and predicts which outcome is more likely for a held-out event combination. It never sees the hidden rules or their probabilities.
## What the first version contains
The renderer and solver remain independent clients. Each can use a different model, endpoint, key, and OpenAI-compatible API style for future model-pair heatmaps.
Three finite skeleton families are included:
## Current task model
- `transitive_order`: a chain of directional relations;
- `membership_inheritance`: membership propagated through subset relations;
- `sequential_state_update`: ordered arithmetic state changes.
Every trial starts from the same empty state. Hidden rules can:
Every generated skeleton contains its facts, rules, query, permitted answer labels, deterministic `gold_answer`, and proof trace. The renderer receives a copy with `gold_answer` and `proof` removed.
- trigger on an event;
- test an input attribute or an internal state flag;
- fire with a hidden probability;
- add a record, permanently invalidate it, or set a flag used by a later event.
The full field grammar, predicate signatures, generator invariants, and renderer boundary are defined in [the Skeleton IR specification](docs/skeleton-language.md).
An invalidation is a persistent veto, so probabilistically triggered actions can conflict. Historical cases report aggregate `PRESENT`/`ABSENT` counts from repeated trials. The query uses a new composition of events and asks only which outcome is more likely—not for the exact probability.
`length` means logical complexity, not natural-language tokens. It is the number of proof-bearing facts or state updates. Presets are `short=2`, `medium=4`, and `long=6`; an integer from 1 through 20 is also accepted.
The local evaluator computes the true query probability by exact enumeration of all stochastic branches. Random simulation is used only to create the finite observations shown to the model. Both are reproducible from the task seed.
The complete hidden-system language, execution semantics, public behavior format, and non-leakage boundary are specified in [the behavioral IR specification](docs/skeleton-language.md).
`length` is structural complexity, not a natural-language token target. It is the number of distinct ordered query events and hidden event rules. Presets are `short=4`, `medium=6`, and `long=8`; explicit integers from 4 through 10 are accepted.
## Local use
Python 3.11 or newer is sufficient; the package has no runtime dependencies.
Generate three short skeletons without calling an LLM:
Generate three short, answer-bearing records without calling an LLM:
```bash
python3 -m logicbench generate --count 3 --length short --seed 20260922
@@ -40,36 +45,30 @@ export SOLVER_MODEL='your-solver-model'
python3 -m logicbench run tasks/short-smoke.json
```
Role-specific values override the shared fallback:
| Variable | Meaning |
| --- | --- |
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials; values may currently be identical |
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Required independent OpenAI-compatible endpoints; values may currently be identical |
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials; values may be identical |
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Required independent OpenAI-compatible endpoints |
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` (default) or `chat_completions` |
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required, independently selected model IDs |
Outputs are written under the task's `output_dir`:
Outputs under the task's `output_dir` are:
- `skeletons.jsonl` is written before any LLM call and includes gold answers;
- `results.jsonl` contains rendered problems, predictions, and per-case scores;
- `summary.json` contains aggregate accuracy and complexity metadata.
- `skeletons.jsonl`: full hidden programs, sampled behavior, exact probabilities, gold labels, and latent traces; written before any LLM call;
- `results.jsonl`: rendered problems, complete solver responses, parsed labels, gold probabilities, and scores;
- `summary.json`: aggregate accuracy and structural metadata.
## Gitea Actions one-shot test
The workflow `.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when someone manually clicks **Run workflow**. It runs five cases with `length: short` and has a 15-minute timeout.
`.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when manually started. It targets the `minecraft-and-affine-server` runner label, runs unit tests and five short live cases, prints the auditable records, and uploads `short-abstraction-evaluation`.
The run log prints every raw answer-bearing skeleton followed by each rendered question, the solver's complete text, its parsed `#### <answer>` label, the gold answer, and correctness. The same records and summary are retained in the `short-abstraction-evaluation` Actions artifact for inspection.
Create these repository Actions settings:
Configure repository Actions with:
1. Secrets `RENDERER_API_KEY` and `SOLVER_API_KEY`.
2. Variables `RENDERER_BASE_URL`, `RENDERER_MODEL`, and `RENDERER_API_STYLE`.
3. Variables `SOLVER_BASE_URL`, `SOLVER_MODEL`, and `SOLVER_API_STYLE`.
The workflow targets the existing `minecraft-and-affine-server` runner label. Verify that a runner advertising this label is online, then manually dispatch **Short abstraction evaluation** from the repository's Actions page.
For a current all-Qwen run, put the Model Studio key into both role-specific secrets and its API Host into both role-specific base-URL variables. The values may be identical today, but the two client boundaries remain explicit. Use `chat_completions` for both API-style variables. For example, a fast renderer and stronger solver can use `qwen3.8-flash` and `qwen3.8-max`; both currently support JSON Schema structured output. Do not hard-code keys or endpoints in the workflow.
For Qwen's OpenAI-compatible API, use the compatible-mode `/v1` endpoint and `chat_completions`. Keys, URLs, and models remain role-specific even when both roles currently use the same provider.
## Tests
docs/skeleton-language.md
+181 -222
View File
@@ -1,248 +1,207 @@
# Skeleton Generator and Intermediate Language
# Hidden System and Behavioral Skeleton Language
This document specifies the answer-bearing logical skeleton generated before any natural-language rendering. The skeleton is the experimental ground truth; natural language is only one view of it.
This document specifies version 0.2 of the benchmark. The central change is that natural-language rendering no longer receives a system's rules. It receives only behavior produced by executing those rules.
## Pipeline contract
The benchmark follows this order:
1. A deterministic local generator selects a logic family and constructs a typed skeleton.
2. The generator computes `gold_answer` and a proof trace from the structure itself.
3. The complete skeleton is written to `skeletons.jsonl` before any model call.
4. A redacted renderer payload removes `gold_answer` and `proof`.
5. The renderer converts the remaining structure into a self-contained natural-language problem.
6. The solver sees only that rendered problem and returns one permitted answer label.
7. The evaluator compares the label with the original `gold_answer`.
This ordering prevents the renderer from deciding or leaking the label. It also makes a failed or semantically inaccurate rendering auditable against its source skeleton.
## Abstract grammar
The current intermediate language can be summarized as:
## Three-layer contract
```text
Skeleton ::= {
id, family, complexity, entities,
facts, rules, query, answer_options,
gold_answer, proof
HiddenSystem --execute/sample--> BehaviorSkeleton --LLM render--> Natural-language problem
| | |
never public observations only solver input
|
exact enumeration --> gold probability --> majority label --> evaluator only
```
Generation follows this order:
1. A seeded local generator builds an executable `HiddenSystem`.
2. The simulator runs controlled support cases repeatedly and records aggregate outcomes.
3. A held-out query combines known events in a combination not present in the histories.
4. An exact local enumerator computes the query's true outcome distribution and gold label.
5. The full answer-bearing record is written before either LLM is called.
6. `renderer_payload()` removes the program, exact probability, gold label, and latent trace.
7. The renderer expresses only the public behavioral evidence in a surface domain.
8. The solver infers a mechanism from those observations and selects the more likely outcome.
This is system identification followed by transfer, not direct deduction from stated laws.
## Hidden system grammar
```text
HiddenSystem ::= {
initial_record,
conflict_policy,
rules: HiddenRule+
}
entities ::= Map<Symbol, TypeName>
facts ::= Fact+
Fact ::= { predicate: PredicateName, arguments: SymbolOrLiteral+ }
rules ::= RuleText*
query ::= { predicate: PredicateName, arguments: SymbolOrLiteral* }
answer_options ::= AnswerLabel+
gold_answer ::= one member of answer_options
proof ::= ProofStep+
HiddenRule ::= {
id,
on_event,
conditions: Condition*,
probability,
action,
target
}
Condition ::= {
source: "attribute" | "state",
key,
operator: "EQ" | "LE" | "GT" | "IS_SET",
value
}
action ::= "ADD_RECORD" | "INVALIDATE_RECORD" | "SET_FLAG"
probability ::= real number in [0, 1]
```
`Symbol`, `TypeName`, `PredicateName`, and `AnswerLabel` are case-sensitive strings. Facts and queries use positional arguments. A predicate's arity and accepted argument types are defined by its logic family.
The grammar is finite, while different rule selections, parameters, event sequences, attributes, and stochastic outcomes generate an unbounded set of behavioral structures.
The implementation currently serializes Python tuples as JSON arrays. Each JSONL line is one complete skeleton.
### State and execution
## Field semantics and invariants
### `id`
A stable run-local identifier such as `order-0000`. It consists of a family prefix and generation index. It is not a semantic input to either model.
### `family`
The name of the generator and inference system that owns the skeleton. Different families may reuse a predicate spelling only when its semantics and argument types are identical.
### `complexity`
The intended structural length of the required proof, not a token count and not a target natural-language length.
For every current family:
Each independent trial starts with:
```text
complexity = number of proof-bearing facts or ordered transformations
= len(proof)
record = ABSENT
invalidated = false
flags = {}
```
Named presets are:
Events execute from left to right. For each event, matching rules execute in stored order:
| Preset | Complexity |
1. evaluate all conditions against immutable attributes and current state;
2. if they hold, sample that rule's Bernoulli trigger;
3. if triggered, apply its action immediately.
Actions mean:
- `SET_FLAG(x)`: add internal flag `x`; later rules can test it;
- `ADD_RECORD`: make the record present unless it was previously invalidated;
- `INVALIDATE_RECORD`: remove the record and permanently set `invalidated=true`.
The current `PERSISTENT_INVALIDATION_VETO` policy therefore implements a one-vote veto. A later add cannot restore an invalidated record. Event order matters because a flag can enable a later rule and an early veto persists.
Probabilities belong to hidden rule activation. They are not printed in the rendered problem. The task asks whether `PRESENT` or `ABSENT` has probability greater than one half, not for a numerical probability.
## Exact gold computation versus sampled evidence
These are deliberately separate:
- Support histories are finite empirical data. The seeded simulator samples each case 48 times for short tasks and 64 times for longer tasks.
- The gold query probability is not estimated by Monte Carlo. The evaluator propagates a probability distribution over every reachable internal state, branching at every applicable stochastic rule.
For each state with mass `m` and applicable rule probability `p`, exact enumeration creates:
```text
inactive branch: same state, mass m(1-p)
active branch: action(state), mass mp
```
Equal resulting states are merged after each rule. At the end, the masses of all states whose record is present are summed. The answer is `PRESENT` when that sum exceeds `0.5`, otherwise `ABSENT`. Current generation rejects cases within `0.08` of the boundary to avoid fragile labels.
## Behavioral skeleton grammar
```text
BehaviorSkeleton ::= {
id,
family,
complexity,
event_vocabulary,
attribute_vocabulary,
observation_protocol,
hidden_system,
support_cases: BehaviorCase+,
query_case: BehaviorCase,
answer_options: ["PRESENT", "ABSENT"],
gold_answer,
gold_present_probability,
latent_trace
}
BehaviorCase ::= {
id,
attributes,
events,
trials,
observed_present,
observed_absent
}
```
Support cases have integer observation counts that sum to `trials`. The query has `null` observation counts because its result is what the solver must predict.
The public payload contains:
```text
family, complexity,
event_vocabulary, attribute_vocabulary,
observation_protocol, support_cases,
query_case, answer_options
```
It never contains:
```text
id, hidden_system, gold_answer,
gold_present_probability, latent_trace
```
## Current generated structure
Every system has at least four interacting rules:
- an unconditional probabilistic add event;
- an attribute-gated probabilistic veto event;
- an event that probabilistically sets an invisible flag;
- a later event whose effect requires that flag.
Longer configurations add attribute-gated add or veto rules. Support evidence includes:
- every event in isolation;
- pairwise event combinations;
- both orders of the state-gated pair;
- low/high attribute counterfactual controls;
- repeated outcomes rather than rule annotations.
The query applies all events in order. Its complete combination is held out from support evidence, so the model must compose effects learned from smaller interventions.
Generated indices alternate between present-favored and absent-favored systems to prevent a one-label benchmark. The index is redacted from the renderer payload so this balancing mechanism cannot become a shortcut. It affects hidden parameter ranges, never the answer after generation; the exact enumerator remains authoritative.
## Complexity and determinism
`complexity` is the number of distinct events in the held-out sequence and the minimum number of corresponding hidden rules. It is independent of rendered token count.
| Preset | Ordered query events |
| --- | ---: |
| `short` | 2 |
| `medium` | 4 |
| `long` | 6 |
| `short` | 4 |
| `medium` | 6 |
| `long` | 8 |
Explicit integer values from 1 through 20 are accepted. Future families must document any deviation from the proof-length interpretation; ideally they should not deviate.
Explicit values from 4 through 10 are accepted. `generate_skeletons(count, length, seed)` uses a local pseudorandom generator; the same code version and arguments produce equivalent hidden systems, samples, queries, and gold answers.
### `entities`
## Renderer obligations
The symbol table. Every non-literal argument appearing in facts or the query must be declared here.
The renderer may rename event codes, attributes, states, and outcomes into a coherent scenario. It must preserve every numeric count, attribute value, event order, trial boundary, query input, and answer label.
Examples of current types are `Entity`, `Set`, and `Counter`. Types exist to distinguish superficially similar but structurally different expressions, such as membership of an entity in a set versus a subset relation between two sets.
It must not:
### `facts`
- receive or state hidden rules;
- turn empirical correlations into declared laws;
- expose trigger probabilities, internal flags, latent traces, or the answer;
- collapse a repeated experiment into a deterministic statement;
- add causal or real-world assumptions;
- show the exact query combination as an already observed history.
Atomic premises or state instructions. Their meaning is family-specific.
Retaining the complete pre-render record makes rendering fidelity independently auditable.
Fact order is semantically irrelevant for declarative families such as transitive order and set membership. Fact order is semantically significant for procedural families such as sequential state updates. The renderer must preserve ordering whenever the family's rules say that order matters.
## Extension points
### `rules`
The next DSL extensions can remain compatible with the three-layer boundary:
The inference laws needed to solve the generated instance. They are included explicitly so that solving does not require real-world assumptions.
- multiple entities and relations between them;
- delayed actions, expiry, and multiple clock phases;
- priority, last-write, merge, and fallback conflict policies;
- partial observation and noisy sensors distinct from stochastic behavior;
- candidate-program enumeration to report identifiability;
- counterfactual and intervention queries in addition to outcome prediction.
Rules are human-readable in version 0.1. A future version can replace or supplement them with machine-executable rule expressions. Until then, the local generator remains the authoritative implementation of the rule.
### `query`
The proposition or value to determine. Its predicate declares the requested operation; its arguments bind the relevant entities.
### `answer_options`
The complete set of legal output labels. The renderer must reproduce these labels exactly, although it may explain them in the surface story. The solver's structured-output schema restricts its response to this set.
### `gold_answer`
The standard answer computed locally during generation. It must be a member of `answer_options`. Neither the renderer nor the solver may receive it.
### `proof`
A minimal intended sequence of premises or transformations sufficient to derive the gold answer. It is primarily an audit trail and a complexity witness. It is not currently checked by the solver and is never sent to the renderer.
## Current predicate signatures
The notation below uses `E:Entity`, `S:Set`, `C:Counter`, and `N:Integer`.
### Transitive order
```text
LEFT_OF(E, E)
RELATION(E, E) -> {LEFT_OF, RIGHT_OF, UNKNOWN}
```
Rules:
```text
LEFT_OF(x, y) and LEFT_OF(y, z) imply LEFT_OF(x, z)
RIGHT_OF(x, y) iff LEFT_OF(y, x)
```
The current generator creates a directed chain of exactly `complexity` edges and asks for the relation between its endpoints. The edge list may be shuffled because the relation is declarative. The standard answer is computed as `LEFT_OF` before rendering.
Example:
```json
{
"id": "order-0000",
"family": "transitive_order",
"complexity": 2,
"entities": {"E0": "Entity", "E1": "Entity", "E2": "Entity"},
"facts": [
{"predicate": "LEFT_OF", "arguments": ["E0", "E1"]},
{"predicate": "LEFT_OF", "arguments": ["E1", "E2"]}
],
"rules": [
"LEFT_OF is transitive",
"RIGHT_OF(x,y) iff LEFT_OF(y,x)"
],
"query": {"predicate": "RELATION", "arguments": ["E0", "E2"]},
"answer_options": ["LEFT_OF", "RIGHT_OF", "UNKNOWN"],
"gold_answer": "LEFT_OF",
"proof": ["LEFT_OF(E0,E1)", "LEFT_OF(E1,E2)"]
}
```
### Membership inheritance
```text
MEMBER_OF(E, S)
SUBSET_OF(S, S)
MEMBER_OF(E, S) -> {YES, NO, UNKNOWN}
```
Rule:
```text
MEMBER_OF(x, A) and SUBSET_OF(A, B) imply MEMBER_OF(x, B)
```
The generator emits one membership fact followed logically by `complexity - 1` subset links. It asks whether the entity belongs to the final set. Premises may be shuffled. The standard answer is `YES`.
### Sequential state update
```text
INITIAL_VALUE(C, N)
INCREMENT(C, N)
DECREMENT(C, N)
FINAL_VALUE(C) -> IntegerLabel
```
The generator chooses an initial non-negative integer and then emits exactly `complexity` ordered updates. An update that would make the state negative is replaced with an increment. The standard answer is produced by executing the updates locally. Two distinct adjacent integer distractors are added to `answer_options`.
## Generator determinism
`generate_skeletons(count, length, seed)` uses a local pseudorandom generator initialized from `seed`. Given the same code version, count, length, and seed, it produces byte-equivalent semantic records.
Families are selected round-robin in the initial implementation:
```text
0 -> transitive_order
1 -> membership_inheritance
2 -> sequential_state_update
3 -> transitive_order
...
```
Randomness currently affects premise order and arithmetic values or operations. It never delegates answer generation to an LLM.
## Renderer boundary
The renderer receives:
```text
id, family, complexity, entities, facts,
rules, query, answer_options
```
It does not receive:
```text
gold_answer, proof
```
The renderer may:
- invent concrete or fictional names;
- replace abstract entities with objects in a coherent scene;
- express formal predicates in natural language;
- reorder declarative premises without changing their meaning.
The renderer must not:
- add or remove facts;
- reverse relation direction;
- reorder procedural state updates;
- introduce real-world assumptions needed for the answer;
- expose or hint at the standard answer;
- rename the permitted answer labels.
Renderer errors are a separate experimental failure mode from solver errors. Retaining both the skeleton and rendered problem makes those failures distinguishable during review.
## Adding a logic family
A new family generator should satisfy all of these conditions:
1. Declare every symbol and type used by its facts and query.
2. Generate a satisfiable instance unless inconsistency is the property being tested.
3. Compute the gold answer locally without an LLM.
4. Emit a proof trace whose length agrees with `complexity`.
5. Provide finite, unambiguous answer labels suitable for exact-match scoring.
6. State whether fact order is semantically significant.
7. Avoid relying on culturally specific or real-world knowledge.
8. Add tests for determinism, type/arity invariants, gold-answer membership, and at least one boundary complexity.
The family function signature is:
```python
def generator(index: int, complexity: int, rng: random.Random) -> Skeleton:
...
```
Register it in `GENERATORS` to include it in round-robin generation. A later heatmap runner can reuse the same stored skeleton set across every renderer/solver model pair, ensuring that model comparisons operate on identical logical inputs.
Any extension must keep gold computation local, expose behavior rather than source rules, preserve seeded reproducibility, and add tests for execution order, conflict semantics, and renderer redaction.
logicbench/__init__.py
+11 -3
View File
@@ -1,6 +1,14 @@
"""Logic skeleton benchmark."""
"""Behavioral hidden-system benchmark."""
from .models import Fact, Query, Skeleton
from .models import BehaviorCase, BehaviorSkeleton, Condition, HiddenRule, HiddenSystem
from .skeletons import generate_skeletons, resolve_length
__all__ = ["Fact", "Query", "Skeleton", "generate_skeletons", "resolve_length"]
__all__ = [
"BehaviorCase",
"BehaviorSkeleton",
"Condition",
"HiddenRule",
"HiddenSystem",
"generate_skeletons",
"resolve_length",
]
logicbench/cli.py
+3 -1
View File
@@ -23,7 +23,9 @@ def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(prog="logicbench")
subparsers = parser.add_subparsers(dest="command", required=True)
generate = subparsers.add_parser("generate", help="generate answer-bearing skeleton JSONL")
generate = subparsers.add_parser(
"generate", help="generate hidden systems and answer-bearing behavior skeletons"
)
generate.add_argument("--count", type=int, default=3)
generate.add_argument("--length", default="short")
generate.add_argument("--seed", type=int, default=0)
logicbench/models.py
+59 -17
View File
@@ -5,36 +5,78 @@ from typing import Any
@dataclass(frozen=True)
class Fact:
predicate: str
arguments: tuple[str, ...]
class Condition:
"""A predicate over immutable inputs or mutable simulator state."""
source: str
key: str
operator: str
value: Any
@dataclass(frozen=True)
class Query:
predicate: str
arguments: tuple[str, ...]
class HiddenRule:
"""One executable rule in the hidden system language."""
id: str
on_event: str
conditions: tuple[Condition, ...]
probability: float
action: str
target: str
@dataclass(frozen=True)
class Skeleton:
class HiddenSystem:
"""The latent program. This object must never be sent to the renderer."""
initial_record: str
conflict_policy: str
rules: tuple[HiddenRule, ...]
@dataclass(frozen=True)
class BehaviorCase:
"""Observable inputs and aggregate output from repeated independent trials."""
id: str
attributes: dict[str, Any]
events: tuple[str, ...]
trials: int
observed_present: int | None
observed_absent: int | None
@dataclass(frozen=True)
class BehaviorSkeleton:
"""A hidden program plus its public behavioral evidence and answer-bearing query."""
id: str
family: str
complexity: int
entities: dict[str, str]
facts: tuple[Fact, ...]
rules: tuple[str, ...]
query: Query
event_vocabulary: tuple[str, ...]
attribute_vocabulary: dict[str, str]
observation_protocol: tuple[str, ...]
hidden_system: HiddenSystem
support_cases: tuple[BehaviorCase, ...]
query_case: BehaviorCase
answer_options: tuple[str, ...]
gold_answer: str
proof: tuple[str, ...]
gold_present_probability: float
latent_trace: tuple[str, ...]
def to_dict(self) -> dict[str, Any]:
return asdict(self)
def renderer_payload(self) -> dict[str, Any]:
"""Return everything needed to render the problem, but never the answer."""
payload = self.to_dict()
payload.pop("gold_answer")
payload.pop("proof")
return payload
"""Expose behavior only: no program, probability, trace, or answer."""
return {
"family": self.family,
"complexity": self.complexity,
"event_vocabulary": self.event_vocabulary,
"attribute_vocabulary": self.attribute_vocabulary,
"observation_protocol": self.observation_protocol,
"support_cases": [asdict(case) for case in self.support_cases],
"query_case": asdict(self.query_case),
"answer_options": self.answer_options,
}
logicbench/pipeline.py
+15 -8
View File
@@ -5,7 +5,7 @@ from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Protocol
from .models import Skeleton
from .models import BehaviorSkeleton
from .skeletons import generate_skeletons
@@ -30,6 +30,7 @@ class EvaluationResult:
rendered_problem: str
answer_options: tuple[str, ...]
gold_answer: str
gold_present_probability: float
solver_output: str
model_answer: str
correct: bool
@@ -43,15 +44,18 @@ RENDER_SCHEMA = {
}
def render_skeleton(client: JsonModel, skeleton: Skeleton) -> str:
def render_skeleton(client: JsonModel, skeleton: BehaviorSkeleton) -> str:
prompt = json.dumps(skeleton.renderer_payload(), ensure_ascii=False, indent=2)
response = client.request_json(
system=(
"You turn typed logical intermediate representations into self-contained natural-language "
"reasoning problems. Invent a concrete or fictional surface scenario and rename entities, but "
"preserve every fact, its order when order matters, the rules, the query, and the exact answer "
"labels. Do not solve the problem, hint at the answer, mention a hidden skeleton, or add facts. "
"End by requiring exactly one of the supplied answer labels."
"You turn behavioral observations from an unknown stochastic system into one self-contained "
"natural-language inductive reasoning problem. Invent a coherent concrete or fictional domain. "
"Give each event code and attribute a stable surface name, and preserve every case's attributes, "
"event order, trial count, observed outcome counts, query, and exact answer labels. The histories "
"are empirical observations, not explicit system laws: do not state or invent the hidden rules, "
"trigger probabilities, intermediate flags, exact query probability, or answer. Make clear that "
"all trials use one unchanged system and that the question asks which final outcome is more likely, "
"not for an exact probability. End by requiring exactly one supplied answer label."
),
user=f"Render this intermediate representation:\n{prompt}",
schema_name="rendered_problem",
@@ -76,7 +80,9 @@ def parse_marked_answer(output: str, answer_options: tuple[str, ...]) -> str:
def solve_problem(client: JsonModel, problem: str, answer_options: tuple[str, ...]) -> tuple[str, str]:
output = client.request_text(
system=(
"Solve the reasoning problem from the stated facts only. Give a concise explanation, then put "
"Infer the stable hidden mechanism from the repeated behavioral observations, then predict which "
"query outcome is more likely. Event order, attributes, stochastic frequencies, interactions, and "
"counterfactual controls may matter. Do not demand an exact probability. Give a concise explanation, then put "
"the final answer on the last non-empty line in the exact form '#### <answer>', where <answer> "
"is exactly one permitted answer label. Do not use unstated real-world assumptions."
),
@@ -116,6 +122,7 @@ def run_evaluation(
rendered_problem=problem,
answer_options=skeleton.answer_options,
gold_answer=skeleton.gold_answer,
gold_present_probability=skeleton.gold_present_probability,
solver_output=solver_output,
model_answer=answer,
correct=answer == skeleton.gold_answer,
logicbench/skeletons.py
+236 -73
View File
@@ -1,16 +1,26 @@
from __future__ import annotations
import random
from collections.abc import Callable
from dataclasses import dataclass
from itertools import combinations
from typing import Any
from .models import Fact, Query, Skeleton
from .models import BehaviorCase, BehaviorSkeleton, Condition, HiddenRule, HiddenSystem
LENGTH_PRESETS = {"short": 2, "medium": 4, "long": 6}
LENGTH_PRESETS = {"short": 4, "medium": 6, "long": 8}
ANSWER_OPTIONS = ("PRESENT", "ABSENT")
@dataclass(frozen=True)
class _State:
present: bool = False
invalidated: bool = False
flags: frozenset[str] = frozenset()
def resolve_length(value: str | int) -> int:
"""Resolve a named preset or an explicit number of proof-bearing operations."""
"""Resolve the number of distinct events in the held-out query."""
if isinstance(value, int):
result = value
elif value in LENGTH_PRESETS:
@@ -21,91 +31,244 @@ def resolve_length(value: str | int) -> int:
except ValueError as exc:
choices = ", ".join(LENGTH_PRESETS)
raise ValueError(f"length must be an integer or one of: {choices}") from exc
if not 1 <= result <= 20:
raise ValueError("length must be between 1 and 20")
if not 4 <= result <= 10:
raise ValueError("length must be between 4 and 10")
return result
def _order_chain(index: int, complexity: int, rng: random.Random) -> Skeleton:
nodes = [f"E{i}" for i in range(complexity + 1)]
facts = [Fact("LEFT_OF", (nodes[i], nodes[i + 1])) for i in range(complexity)]
rng.shuffle(facts)
return Skeleton(
id=f"order-{index:04d}",
family="transitive_order",
complexity=complexity,
entities={node: "Entity" for node in nodes},
facts=tuple(facts),
rules=("LEFT_OF is transitive", "RIGHT_OF(x,y) iff LEFT_OF(y,x)"),
query=Query("RELATION", (nodes[0], nodes[-1])),
answer_options=("LEFT_OF", "RIGHT_OF", "UNKNOWN"),
gold_answer="LEFT_OF",
proof=tuple(f"LEFT_OF({nodes[i]},{nodes[i + 1]})" for i in range(complexity)),
def _condition_holds(condition: Condition, attributes: dict[str, Any], state: _State) -> bool:
if condition.source == "attribute":
actual = attributes.get(condition.key)
elif condition.source == "state":
actual = condition.key in state.flags
else:
raise ValueError(f"unknown condition source: {condition.source}")
operations = {
"EQ": lambda left, right: left == right,
"LE": lambda left, right: left <= right,
"GT": lambda left, right: left > right,
"IS_SET": lambda left, right: left is bool(right),
}
try:
return operations[condition.operator](actual, condition.value)
except KeyError as exc:
raise ValueError(f"unknown condition operator: {condition.operator}") from exc
def _apply_action(state: _State, rule: HiddenRule) -> _State:
if rule.action == "SET_FLAG":
return _State(state.present, state.invalidated, state.flags | {rule.target})
if rule.action == "ADD_RECORD":
if state.invalidated:
return state
return _State(True, False, state.flags)
if rule.action == "INVALIDATE_RECORD":
return _State(False, True, state.flags)
raise ValueError(f"unknown action: {rule.action}")
def exact_present_probability(
system: HiddenSystem, attributes: dict[str, Any], events: tuple[str, ...]
) -> float:
"""Enumerate all stochastic branches while respecting event and state order."""
initial = _State(present=system.initial_record == "PRESENT")
distribution: dict[_State, float] = {initial: 1.0}
for event in events:
for rule in (item for item in system.rules if item.on_event == event):
next_distribution: dict[_State, float] = {}
for state, mass in distribution.items():
if not all(_condition_holds(item, attributes, state) for item in rule.conditions):
next_distribution[state] = next_distribution.get(state, 0.0) + mass
continue
inactive_mass = mass * (1.0 - rule.probability)
active_state = _apply_action(state, rule)
next_distribution[state] = next_distribution.get(state, 0.0) + inactive_mass
next_distribution[active_state] = (
next_distribution.get(active_state, 0.0) + mass * rule.probability
)
distribution = next_distribution
return sum(mass for state, mass in distribution.items() if state.present)
def _simulate_once(
system: HiddenSystem,
attributes: dict[str, Any],
events: tuple[str, ...],
rng: random.Random,
) -> bool:
state = _State(present=system.initial_record == "PRESENT")
for event in events:
for rule in (item for item in system.rules if item.on_event == event):
if all(_condition_holds(item, attributes, state) for item in rule.conditions):
if rng.random() < rule.probability:
state = _apply_action(state, rule)
return state.present
def _observe_case(
case_id: str,
system: HiddenSystem,
attributes: dict[str, Any],
events: tuple[str, ...],
trials: int,
rng: random.Random,
) -> BehaviorCase:
present = sum(_simulate_once(system, attributes, events, rng) for _ in range(trials))
return BehaviorCase(
id=case_id,
attributes=attributes,
events=events,
trials=trials,
observed_present=present,
observed_absent=trials - present,
)
def _set_chain(index: int, complexity: int, rng: random.Random) -> Skeleton:
item = "E0"
sets = [f"S{i}" for i in range(complexity)]
facts = [Fact("MEMBER_OF", (item, sets[0]))]
facts.extend(Fact("SUBSET_OF", (sets[i], sets[i + 1])) for i in range(complexity - 1))
rng.shuffle(facts)
return Skeleton(
id=f"set-{index:04d}",
family="membership_inheritance",
complexity=complexity,
entities={item: "Entity", **{name: "Set" for name in sets}},
facts=tuple(facts),
rules=("MEMBER_OF(x,A) and SUBSET_OF(A,B) imply MEMBER_OF(x,B)",),
query=Query("MEMBER_OF", (item, sets[-1])),
answer_options=("YES", "NO", "UNKNOWN"),
gold_answer="YES",
proof=tuple(
[f"MEMBER_OF({item},{sets[0]})"]
+ [f"SUBSET_OF({sets[i]},{sets[i + 1]})" for i in range(complexity - 1)]
def _build_system(index: int, complexity: int, rng: random.Random) -> HiddenSystem:
"""Build a finite stochastic program with a gated rule and veto conflicts."""
present_favored = index % 2 == 0
add_probability = rng.choice((0.85, 0.9, 0.95)) if present_favored else rng.choice((0.65, 0.75))
veto_probability = rng.choice((0.15, 0.25)) if present_favored else rng.choice((0.7, 0.8, 0.9))
marker_probability = rng.choice((0.6, 0.75, 0.9))
gated_action = "ADD_RECORD" if present_favored else "INVALIDATE_RECORD"
gated_probability = rng.choice((0.6, 0.75, 0.85))
rules = [
HiddenRule("R0", "E0", (), add_probability, "ADD_RECORD", "record"),
HiddenRule(
"R1",
"E1",
(Condition("attribute", "load", "LE", 4),),
veto_probability,
"INVALIDATE_RECORD",
"record",
),
HiddenRule("R2", "E2", (), marker_probability, "SET_FLAG", "marked"),
HiddenRule(
"R3",
"E3",
(Condition("state", "marked", "IS_SET", True),),
gated_probability,
gated_action,
"record",
),
]
for number in range(4, complexity):
action = "ADD_RECORD" if (number + index) % 3 else "INVALIDATE_RECORD"
probability = rng.choice((0.2, 0.35, 0.5))
condition = Condition("attribute", "class", "EQ", "priority")
rules.append(
HiddenRule(f"R{number}", f"E{number}", (condition,), probability, action, "record")
)
return HiddenSystem(
initial_record="ABSENT",
conflict_policy="PERSISTENT_INVALIDATION_VETO",
rules=tuple(rules),
)
def _state_updates(index: int, complexity: int, rng: random.Random) -> Skeleton:
start = rng.randint(3, 9)
operations: list[tuple[str, int]] = []
value = start
for _ in range(complexity):
amount = rng.randint(1, 4)
predicate = rng.choice(("INCREMENT", "DECREMENT"))
if predicate == "DECREMENT" and value - amount < 0:
predicate = "INCREMENT"
value = value + amount if predicate == "INCREMENT" else value - amount
operations.append((predicate, amount))
def _support_event_sets(complexity: int) -> list[tuple[str, ...]]:
events = [f"E{number}" for number in range(complexity)]
cases: list[tuple[str, ...]] = [(event,) for event in events]
cases.extend((("E2", "E3"), ("E3", "E2"), ("E0", "E1")))
cases.extend(tuple(pair) for pair in combinations(events, 2) if tuple(pair) not in cases)
return cases
facts = [Fact("INITIAL_VALUE", ("E0", str(start)))]
facts.extend(Fact(predicate, ("E0", str(amount))) for predicate, amount in operations)
lower_distractor = value - 1 if value > 0 else value + 2
return Skeleton(
id=f"state-{index:04d}",
family="sequential_state_update",
def _generate_one(index: int, complexity: int, rng: random.Random) -> BehaviorSkeleton:
system = _build_system(index, complexity, rng)
low = {"load": 3, "class": "priority"}
high = {"load": 8, "class": "standard"}
query_attributes = dict(low)
query_events = tuple(f"E{number}" for number in range(complexity))
probability = exact_present_probability(system, query_attributes, query_events)
if abs(probability - 0.5) < 0.08:
raise RuntimeError("generated query has an insufficient probability margin")
gold = "PRESENT" if probability > 0.5 else "ABSENT"
trials = 48 if complexity <= 4 else 64
support: list[BehaviorCase] = []
for case_number, events in enumerate(_support_event_sets(complexity)):
attributes = low if case_number % 3 else high
support.append(
_observe_case(
f"history-{case_number:03d}", system, dict(attributes), events, trials, rng
)
)
# Keeping events fixed while changing load makes the veto guard observable.
next_id = len(support)
support.append(
_observe_case(f"history-{next_id:03d}", system, dict(low), ("E0", "E1"), trials, rng)
)
support.append(
_observe_case(
f"history-{next_id + 1:03d}", system, dict(high), ("E0", "E1"), trials, rng
)
)
support.append(
_observe_case(
f"history-{next_id + 2:03d}",
system,
dict(low),
("E0", "E2", "E3"),
trials,
rng,
)
)
query = BehaviorCase(
id="query",
attributes=query_attributes,
events=query_events,
trials=1,
observed_present=None,
observed_absent=None,
)
trace = tuple(
[f"query events: {' -> '.join(query_events)}"]
+ [
f"{rule.id}: on {rule.on_event}, p={rule.probability}, {rule.action}({rule.target})"
for rule in system.rules
]
+ [f"exact P(PRESENT)={probability:.8f}", f"gold={gold}"]
)
return BehaviorSkeleton(
id=f"dynamic-{index:04d}",
family="stochastic_behavioral_system",
complexity=complexity,
entities={"E0": "Counter"},
facts=tuple(facts),
rules=("Apply updates in the listed order",),
query=Query("FINAL_VALUE", ("E0",)),
answer_options=(str(value), str(value + 1), str(lower_distractor)),
gold_answer=str(value),
proof=tuple(f"{predicate}({amount})" for predicate, amount in operations),
event_vocabulary=tuple(f"E{number}" for number in range(complexity)),
attribute_vocabulary={"load": "integer", "class": "category"},
observation_protocol=(
"Each trial begins with an absent record and no internal flags.",
"Events occur from left to right; the same hidden system is used in every trial.",
"PRESENT and ABSENT counts are final observations over independent repeated trials.",
"For the query, choose which final outcome is more likely; an exact probability is not required.",
),
hidden_system=system,
support_cases=tuple(support),
query_case=query,
answer_options=ANSWER_OPTIONS,
gold_answer=gold,
gold_present_probability=probability,
latent_trace=trace,
)
GENERATORS: tuple[Callable[[int, int, random.Random], Skeleton], ...] = (
_order_chain,
_set_chain,
_state_updates,
)
def generate_skeletons(count: int, length: str | int, seed: int = 0) -> list[Skeleton]:
def generate_skeletons(count: int, length: str | int, seed: int = 0) -> list[BehaviorSkeleton]:
if count < 1:
raise ValueError("count must be positive")
complexity = resolve_length(length)
rng = random.Random(seed)
return [GENERATORS[i % len(GENERATORS)](i, complexity, rng) for i in range(count)]
generated: list[BehaviorSkeleton] = []
for index in range(count):
for _attempt in range(100):
try:
generated.append(_generate_one(index, complexity, rng))
break
except RuntimeError as exc:
if "insufficient probability margin" not in str(exc):
raise
else:
raise RuntimeError("could not generate a query with a stable probability margin")
return generated
pyproject.toml
+2 -2
View File
@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
[project]
name = "logic-skeleton-bench"
version = "0.1.0"
description = "Generate logical skeletons, render them as stories, and evaluate LLM solvers."
version = "0.2.0"
description = "Infer stochastic hidden systems from rendered behavioral observations."
readme = "README.md"
requires-python = ">=3.11"
license = { text = "MIT" }
tasks/short-smoke.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "short-smoke",
"description": "Render and solve five short examples across the initial skeleton families.",
"description": "Infer five short stochastic hidden systems from behavioral histories.",
"count": 5,
"length": "short",
"seed": 20260922,
tests/test_pipeline.py
+10 -11
View File
@@ -19,19 +19,18 @@ class QueueClient:
class PipelineTests(unittest.TestCase):
def test_end_to_end_writes_skeletons_before_scored_results(self):
renderer = QueueClient([
{"problem": "Choose LEFT_OF, RIGHT_OF, or UNKNOWN."},
{"problem": "Choose YES, NO, or UNKNOWN."},
{"problem": "Choose 9, 10, or 8."},
{"problem": "Choose LEFT_OF, RIGHT_OF, or UNKNOWN."},
{"problem": "Choose YES, NO, or UNKNOWN."},
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
])
# Seed 20260922 makes the third gold answer 9.
solver = QueueClient([
"The relation follows by transitivity.\n#### LEFT_OF",
"Membership propagates through the subset.\n#### YES",
"The two increments produce nine.\n#### 9",
"The relation follows by transitivity.\n#### LEFT_OF",
"Membership propagates through the subset.\n#### YES",
"The positive behavior dominates.\n#### PRESENT",
"The veto behavior dominates.\n#### ABSENT",
"The positive behavior dominates.\n#### PRESENT",
"The veto behavior dominates.\n#### ABSENT",
"The positive behavior dominates.\n#### PRESENT",
])
with tempfile.TemporaryDirectory() as directory:
output = Path(directory)
tests/test_skeletons.py
+53 -14
View File
@@ -1,36 +1,75 @@
import unittest
from logicbench.skeletons import generate_skeletons, resolve_length
from logicbench.models import Condition, HiddenRule, HiddenSystem
from logicbench.skeletons import exact_present_probability, generate_skeletons, resolve_length
class SkeletonTests(unittest.TestCase):
def test_short_means_two_proof_bearing_operations(self):
def test_short_means_four_query_events(self):
skeletons = generate_skeletons(count=3, length="short", seed=42)
self.assertEqual([item.complexity for item in skeletons], [2, 2, 2])
self.assertEqual([item.family for item in skeletons], [
"transitive_order",
"membership_inheritance",
"sequential_state_update",
])
self.assertTrue(all(len(item.proof) == 2 for item in skeletons))
self.assertEqual([item.complexity for item in skeletons], [4, 4, 4])
self.assertTrue(all(len(item.query_case.events) == 4 for item in skeletons))
self.assertEqual(
[item.gold_answer for item in skeletons], ["PRESENT", "ABSENT", "PRESENT"]
)
def test_generation_is_deterministic(self):
first = [item.to_dict() for item in generate_skeletons(6, 4, seed=7)]
second = [item.to_dict() for item in generate_skeletons(6, 4, seed=7)]
self.assertEqual(first, second)
def test_renderer_never_receives_gold_or_proof(self):
def test_renderer_receives_behavior_but_no_latent_information(self):
skeleton = generate_skeletons(1, "short", seed=1)[0]
payload = skeleton.renderer_payload()
self.assertNotIn("gold_answer", payload)
self.assertNotIn("proof", payload)
self.assertIn("answer_options", payload)
for secret in (
"id",
"hidden_system",
"gold_answer",
"gold_present_probability",
"latent_trace",
):
self.assertNotIn(secret, payload)
self.assertIn("support_cases", payload)
self.assertIsNone(payload["query_case"]["observed_present"])
def test_exact_enumerator_applies_persistent_veto(self):
system = HiddenSystem(
initial_record="ABSENT",
conflict_policy="PERSISTENT_INVALIDATION_VETO",
rules=(
HiddenRule("add", "A", (), 0.8, "ADD_RECORD", "record"),
HiddenRule("veto", "B", (), 0.25, "INVALIDATE_RECORD", "record"),
),
)
self.assertAlmostEqual(exact_present_probability(system, {}, ("A", "B")), 0.6)
self.assertAlmostEqual(exact_present_probability(system, {}, ("B", "A")), 0.6)
def test_exact_enumerator_respects_stateful_event_order(self):
system = HiddenSystem(
initial_record="ABSENT",
conflict_policy="PERSISTENT_INVALIDATION_VETO",
rules=(
HiddenRule("mark", "A", (), 0.5, "SET_FLAG", "ready"),
HiddenRule(
"add",
"B",
(Condition("state", "ready", "IS_SET", True),),
0.8,
"ADD_RECORD",
"record",
),
),
)
self.assertAlmostEqual(exact_present_probability(system, {}, ("A", "B")), 0.4)
self.assertEqual(exact_present_probability(system, {}, ("B", "A")), 0.0)
def test_length_validation(self):
self.assertEqual(resolve_length("medium"), 4)
self.assertEqual(resolve_length("medium"), 6)
self.assertEqual(resolve_length("5"), 5)
with self.assertRaises(ValueError):
resolve_length("huge")
with self.assertRaises(ValueError):
resolve_length(3)
if __name__ == "__main__":