Generate proof-bearing dynamic world histories

This commit is contained in:
2026-09-23 10:23:09 +08:00
parent 6f929437a2
commit 43aff1e2ee
13 changed files with 1023 additions and 550 deletions
+5 -5
View File
@@ -1,4 +1,4 @@
name: Short behavioral-system evaluation name: Dynamic world reasoning evaluation
on: on:
workflow_dispatch: workflow_dispatch:
@@ -6,7 +6,7 @@ on:
jobs: jobs:
short-eval: short-eval:
runs-on: minecraft-and-affine-server runs-on: minecraft-and-affine-server
timeout-minutes: 15 timeout-minutes: 45
env: env:
RENDERER_API_KEY: ${{ secrets.RENDERER_API_KEY }} RENDERER_API_KEY: ${{ secrets.RENDERER_API_KEY }}
RENDERER_BASE_URL: ${{ vars.RENDERER_BASE_URL }} RENDERER_BASE_URL: ${{ vars.RENDERER_BASE_URL }}
@@ -24,10 +24,10 @@ jobs:
- name: Run unit tests - name: Run unit tests
run: python3 -m unittest discover -s tests -v run: python3 -m unittest discover -s tests -v
- name: Run five short live cases - name: Run five large-world live cases
run: python3 -m logicbench run tasks/short-smoke.json run: python3 -m logicbench run tasks/short-smoke.json
- name: Show hidden systems, behavior skeletons, results, and summary - name: Show dynamic worlds, rendered histories, results, and summary
if: always() if: always()
run: | run: |
test -f artifacts/short-smoke/skeletons.jsonl && cat artifacts/short-smoke/skeletons.jsonl || true test -f artifacts/short-smoke/skeletons.jsonl && cat artifacts/short-smoke/skeletons.jsonl || true
@@ -40,5 +40,5 @@ jobs:
if: always() if: always()
uses: actions/upload-artifact@v3 uses: actions/upload-artifact@v3
with: with:
name: short-abstraction-evaluation name: dynamic-world-evaluation
path: artifacts/short-smoke/ path: artifacts/short-smoke/
+63 -37
View File
@@ -1,78 +1,104 @@
# Logic Skeleton Bench # Logic Skeleton Bench
Logic Skeleton Bench tests whether a model can infer an abstract mechanism from behavior rather than solve a rule statement with renamed nouns. Logic Skeleton Bench generates large, evolving fictional worlds and tests whether a model can recover a logical consequence from their natural-language histories.
The generator now creates an executable **hidden stochastic system**, runs controlled experiments against it, and builds a **behavior skeleton** containing only inputs, ordered events, and observed final outcomes. A renderer turns that evidence into a coherent natural-language world. The solver sees the rendered histories and predicts which outcome is more likely for a held-out event combination. It never sees the hidden rules or their probabilities. The benchmark has five stages:
The renderer and solver remain independent clients. Each can use a different model, endpoint, key, and OpenAI-compatible API style for future model-pair heatmaps. ```text
hidden typed program
→ initial world with many entities
→ seeded stochastic evolution
→ deterministic realized event history and gold proof
→ natural-language rendering and solving
```
## Current task model Probability is used only to decide which candidate events actually occur. Once a seed has produced a history, the question and answer are deterministic. The solver never estimates a probability and never sees repeated-trial frequency tables.
Every trial starts from the same empty state. Hidden rules can: ## Current world model
- trigger on an event; The initial implementation mixes several logic domains in every world:
- test an input attribute or an internal state flag;
- fire with a hidden probability;
- add a record, permanently invalidate it, or set a flag used by a later event.
An invalidation is a persistent veto, so probabilistically triggered actions can conflict. Historical cases report aggregate `PRESENT`/`ABSENT` counts from repeated trials. The query uses a new composition of events and asks only which outcome is more likely—not for the exact probability. - directed spatial order (`ABOVE`/`BELOW`);
- membership, subset inheritance, and disjointness;
- locations and movement;
- symmetric connectivity;
- sequential numeric updates;
- Boolean state changes.
The local evaluator computes the true query probability by exact enumeration of all stochastic branches. Random simulation is used only to create the finite observations shown to the model. Both are reproducible from the task seed. The hidden program contains typed relations, operators, quantifiers, inference laws, probabilistic transition rules, and their preconditions. A default short world contains 128 entities and runs for 1000 simulation steps, producing roughly 400 realized events with the checked-in seed.
The complete hidden-system language, execution semantics, public behavior format, and non-leakage boundary are specified in [the behavioral IR specification](docs/skeleton-language.md). After evolution, the generator searches the realized state for a query with a shortest proof at least as long as the configured `length`. It rejects directly stated answers and stores the exact proof event IDs internally. The renderer receives all public initial observations and all realized events, but not the program, probabilities, hidden ranks, proof, generator checks, or gold answer.
`length` is structural complexity, not a natural-language token target. It is the number of distinct ordered query events and hidden event rules. Presets are `short=4`, `medium=6`, and `long=8`; explicit integers from 4 through 10 are accepted. The full grammar and execution contract are defined in [the dynamic-world IR specification](docs/skeleton-language.md).
## Complexity controls
`length` controls minimum proof length, not entity count, event count, or natural-language tokens:
| Preset | Minimum proof events |
| --- | ---: |
| `short` | 4 |
| `medium` | 6 |
| `long` | 8 |
`entity_count` and `simulation_steps` are separate task fields. Explicit proof lengths from 3 through 10 are accepted.
## Local use ## Local use
Python 3.11 or newer is sufficient; the package has no runtime dependencies. Python 3.11 or newer is sufficient; there are no runtime dependencies.
Generate three short, answer-bearing records without calling an LLM: Generate an answer-bearing world without calling an LLM:
```bash ```bash
python3 -m logicbench generate --count 3 --length short --seed 20260922 python3 -m logicbench generate \
--count 1 \
--length short \
--entity-count 128 \
--simulation-steps 1000 \
--seed 20260922
``` ```
Run the checked-in live task: Run the checked-in live task:
```bash ```bash
export RENDERER_API_KEY='...' export RENDERER_API_KEY='...'
export RENDERER_BASE_URL='your-renderer-compatible-base-url' export RENDERER_BASE_URL='...'
export RENDERER_MODEL='your-renderer-model' export RENDERER_MODEL='...'
export SOLVER_API_KEY='...' export SOLVER_API_KEY='...'
export SOLVER_BASE_URL='your-solver-compatible-base-url' export SOLVER_BASE_URL='...'
export SOLVER_MODEL='your-solver-model' export SOLVER_MODEL='...'
python3 -m logicbench run tasks/short-smoke.json python3 -m logicbench run tasks/short-smoke.json
``` ```
| Variable | Meaning | | Variable | Meaning |
| --- | --- | | --- | --- |
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials; values may be identical | | `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials |
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Required independent OpenAI-compatible endpoints | | `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Required independent OpenAI-compatible endpoints |
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` (default) or `chat_completions` | | `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` or `chat_completions` |
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required, independently selected model IDs | | `RENDERER_MODEL`, `SOLVER_MODEL` | Required independent model IDs |
| `RENDERER_TIMEOUT_SECONDS`, `SOLVER_TIMEOUT_SECONDS` | Optional per-request timeout; default `240` | | `RENDERER_TIMEOUT_SECONDS`, `SOLVER_TIMEOUT_SECONDS` | Optional per-request timeout; default `240` |
| `RENDERER_ENABLE_THINKING`, `SOLVER_ENABLE_THINKING` | Optional `true`/`false` provider extension; omitted unless set | | `RENDERER_ENABLE_THINKING`, `SOLVER_ENABLE_THINKING` | Optional provider extension; omitted unless set |
Outputs under the task's `output_dir` are: The renderer and solver remain independent clients for model-pair heatmaps. The checked-in Qwen workflow explicitly disables solver thinking to keep non-streaming test latency bounded.
- `skeletons.jsonl`: full hidden programs, sampled behavior, exact probabilities, gold labels, and latent traces; written before any LLM call; ## Artifacts
- `rendered.jsonl`: each natural-language problem, checkpointed immediately after rendering;
- `solver-outputs.jsonl`: each complete raw solver response, checkpointed before parsing;
- `results.jsonl`: rendered problems, complete solver responses, parsed labels, parse errors, format compliance, gold probabilities, and scores;
- `summary.json`: aggregate accuracy and structural metadata.
## Gitea Actions one-shot test Each run writes:
`.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when manually started. It targets the `minecraft-and-affine-server` runner label, runs unit tests and five short live cases, prints the auditable records, and uploads `short-abstraction-evaluation`. - `skeletons.jsonl`: hidden program, entities, hidden initial and final states, complete event trace, query, gold answer, proof, and anti-leakage checks;
- `rendered.jsonl`: natural-language worlds, checkpointed immediately after rendering;
- `solver-outputs.jsonl`: complete raw solver responses, checkpointed before parsing;
- `results.jsonl`: parsed labels, parse errors, format compliance, and correctness;
- `summary.json`: aggregate accuracy, parsing metrics, scale, and per-world event counts.
Configure repository Actions with: ## Gitea Actions
1. Secrets `RENDERER_API_KEY` and `SOLVER_API_KEY`. `.gitea/workflows/short-eval.yml` is manual-only (`workflow_dispatch`) and targets `minecraft-and-affine-server`. It runs unit tests followed by five large-world cases. Because rendering hundreds of events is substantially slower than the former toy benchmark, the job timeout is 45 minutes.
2. Variables `RENDERER_BASE_URL`, `RENDERER_MODEL`, and `RENDERER_API_STYLE`.
3. Variables `SOLVER_BASE_URL`, `SOLVER_MODEL`, and `SOLVER_API_STYLE`.
For Qwen's OpenAI-compatible API, use the compatible-mode `/v1` endpoint and `chat_completions`. Keys, URLs, and models remain role-specific even when both roles currently use the same provider. The checked-in short workflow explicitly disables solver thinking to avoid long non-streaming requests; remove or change that workflow value when thinking-mode performance is itself under test. Repository Actions settings remain two secrets and six role-specific variables:
- secrets: `RENDERER_API_KEY`, `SOLVER_API_KEY`;
- variables: each role's `BASE_URL`, `MODEL`, and `API_STYLE`.
## Tests ## Tests
+193 -154
View File
@@ -1,207 +1,246 @@
# Hidden System and Behavioral Skeleton Language # Dynamic World Generator and Intermediate Language
This document specifies version 0.2 of the benchmark. The central change is that natural-language rendering no longer receives a system's rules. It receives only behavior produced by executing those rules. This specification defines the answer-bearing world representation generated before natural-language rendering.
## Three-layer contract ## Pipeline boundary
```text ```text
HiddenSystem --execute/sample--> BehaviorSkeleton --LLM render--> Natural-language problem HiddenProgram + seed
| | | ↓
never public observations only solver input Initial WorldState
| ↓ stochastic transition sampling
exact enumeration --> gold probability --> majority label --> evaluator only Realized Event Trace
↓ deterministic closure and query selection
Query + Gold Proof
↓ redact program, hidden state, proof, and answer
Renderer Payload
↓
Natural-language history → Solver
``` ```
Generation follows this order: Randomness affects which events occur. It never makes the final question probabilistic: after the trace is realized, the evaluator asks whether a proposition follows in that particular world.
1. A seeded local generator builds an executable `HiddenSystem`. ## Hidden program
2. The simulator runs controlled support cases repeatedly and records aggregate outcomes.
3. A held-out query combines known events in a combination not present in the histories.
4. An exact local enumerator computes the query's true outcome distribution and gold label.
5. The full answer-bearing record is written before either LLM is called.
6. `renderer_payload()` removes the program, exact probability, gold label, and latent trace.
7. The renderer expresses only the public behavioral evidence in a surface domain.
8. The solver infers a mechanism from those observations and selects the more likely outcome.
This is system identification followed by transfer, not direct deduction from stated laws.
## Hidden system grammar
```text ```text
HiddenSystem ::= { HiddenProgram ::= {
initial_record, types: TypeName+,
conflict_policy, relations: PredicateName+,
rules: HiddenRule+ operators: OperatorName+,
quantifiers: QuantifierName+,
structures: StructureName+,
inference_rules: RuleExpression+,
transition_rules: TransitionRule+
} }
HiddenRule ::= { TransitionRule ::= {
id, id,
on_event, trigger,
conditions: Condition*, preconditions: Expression*,
probability, probability: Number[0,1],
action, effects: Expression+
target
} }
```
Condition ::= { The current finite vocabulary includes:
source: "attribute" | "state",
key, ```text
operator: "EQ" | "LE" | "GT" | "IS_SET", types:
value Entity, Set, Region, Number, Boolean, State
relations:
EQUAL, NOT_EQUAL, ABOVE, BELOW,
MEMBER, SUBSET, DISJOINT,
ADJACENT, CONNECTED, CAUSES, LOCATED_IN
operators:
NOT, AND, OR, IMPLIES,
ADD, SUBTRACT, COMPARE,
UNION, INTERSECTION,
MOVE, SWAP, SET, COMPOSE
quantifiers:
ALL, EXISTS
structures:
CHAIN, BRANCH, MERGE, CYCLE, NEST, CONDITION, SEQUENCE
```
Not every primitive must affect every query. The point of the shared world is that many logic domains evolve concurrently and become realistic distractors for one proof-bearing target.
## Entities and initial state
```text
Entity ::= { id, type }
WorldState ::= {
locations,
directed_order_relations,
memberships,
subset_relations,
disjoint_relations,
connections,
numeric_values,
boolean_flags,
hidden_generation_attributes
} }
action ::= "ADD_RECORD" | "INVALIDATE_RECORD" | "SET_FLAG"
probability ::= real number in [0, 1]
``` ```
The grammar is finite, while different rule selections, parameters, event sequences, attributes, and stochastic outcomes generate an unbounded set of behavioral structures. The default 128-entity allocation includes ordinary entities, sets, regions, and numeric counters. Entity IDs are abstract symbols; the renderer must assign stable names within its chosen scenario.
### State and execution Some initial facts are public, such as locations and counter values. Each has an `I####` audit ID. Latent generation attributes, such as levels used to keep randomly generated order and subset graphs consistent, are stored for audit but removed from the renderer payload.
Each independent trial starts with: ## Event language
```text ```text
record = ABSENT WorldEvent ::= {
invalidated = false id,
flags = {} time,
kind,
participants,
arguments
}
``` ```
Events execute from left to right. For each event, matching rules execute in stored order: Current realized event forms are:
1. evaluate all conditions against immutable attributes and current state;
2. if they hold, sample that rule's Bernoulli trigger;
3. if triggered, apply its action immediately.
Actions mean:
- `SET_FLAG(x)`: add internal flag `x`; later rules can test it;
- `ADD_RECORD`: make the record present unless it was previously invalidated;
- `INVALIDATE_RECORD`: remove the record and permanently set `invalidated=true`.
The current `PERSISTENT_INVALIDATION_VETO` policy therefore implements a one-vote veto. A later add cannot restore an invalidated record. Event order matters because a flag can enable a later rule and an early veto persists.
Probabilities belong to hidden rule activation. They are not printed in the rendered problem. The task asks whether `PRESENT` or `ABSENT` has probability greater than one half, not for a numerical probability.
## Exact gold computation versus sampled evidence
These are deliberately separate:
- Support histories are finite empirical data. The seeded simulator samples each case 48 times for short tasks and 64 times for longer tasks.
- The gold query probability is not estimated by Monte Carlo. The evaluator propagates a probability distribution over every reachable internal state, branching at every applicable stochastic rule.
For each state with mass `m` and applicable rule probability `p`, exact enumeration creates:
```text ```text
inactive branch: same state, mass m(1-p) MOVE(entity, origin, destination)
active branch: action(state), mass mp RELATE(ABOVE, higher, lower, region)
RELATE(MEMBER, entity, set)
RELATE(SUBSET, child_set, parent_set)
RELATE(DISJOINT, left_set, right_set)
RELATE(CONNECTED, left_entity, right_entity)
UPDATE_NUMBER(counter, ADD|SUBTRACT, amount, before, after)
SET_STATE(entity, flag, before, after)
``` ```
Equal resulting states are merged after each rule. At the end, the masses of all states whose record is present are summed. The answer is `PRESENT` when that sum exceeds `0.5`, otherwise `ABSENT`. Current generation rejects cases within `0.08` of the boundary to avoid fragile labels. Every simulator step selects a candidate transition type. Preconditions are evaluated, then the transition's probability determines whether an event actually happens. Only realized events enter the trace. A fixed seed makes the complete trace reproducible.
## Behavioral skeleton grammar The default short configuration runs 1000 candidate steps and currently realizes about 400 events. Thus probability controls world evolution, while the resulting event history is an ordinary deterministic collection of facts and changes.
## Current inference semantics
```text ```text
BehaviorSkeleton ::= { ABOVE(x,y) ∧ ABOVE(y,z) → ABOVE(x,z)
ABOVE(x,y) → BELOW(y,x)
MEMBER(x,A) ∧ SUBSET(A,B) → MEMBER(x,B)
MEMBER(x,A) ∧ DISJOINT(A,B) → ¬MEMBER(x,B)
CONNECTED(x,y) ↔ CONNECTED(y,x)
numeric updates execute in timestamp order
later SET_STATE operations replace the prior value of the same flag
MOVE replaces the current region of its participant
```
`ABOVE` and `SUBSET` events are generated from adjacent hidden levels. This allows long chains while preventing contradictory cycles. The levels are generator machinery, not observable premises.
## Query generation
```text
Query ::= {
predicate,
arguments,
time
}
```
After simulation, the query builder searches the realized graphs rather than inserting a prefabricated puzzle. Current query families are:
- `spatial_order`: transitive `ABOVE` proof;
- reverse `spatial_order`: a `NO` answer proved by an opposite-direction chain;
- `membership_inheritance`: membership propagated through nested subsets;
- `membership_disjointness`: inherited membership in one set plus disjointness from the queried set.
For a requested complexity `k`, the selected proof contains at least `k` realized event IDs. The queried proposition itself must not appear as a direct event. The full record stores both the event IDs and normalized proof facts.
Current generation checks record that:
- no event directly states the query;
- proof length meets the requested minimum;
- proof events are a small fraction of the full history;
- simple direct-fact and too-short-proof baselines fail.
These checks are structural. They should be expanded with executable solver baselines as the benchmark grows.
## Full skeleton
```text
WorldSkeleton ::= {
id, id,
family, family,
complexity, complexity,
event_vocabulary, entity_count,
attribute_vocabulary, simulation_steps,
observation_protocol, hidden_program,
hidden_system, entities,
support_cases: BehaviorCase+, initial_state,
query_case: BehaviorCase,
answer_options: ["PRESENT", "ABSENT"],
gold_answer,
gold_present_probability,
latent_trace
}
BehaviorCase ::= {
id,
attributes,
events, events,
trials, final_state,
observed_present, query,
observed_absent answer_options,
gold_answer,
proof_event_ids,
proof_facts,
generator_checks
} }
``` ```
Support cases have integer observation counts that sum to `trials`. The query has `null` observation counts because its result is what the solver must predict. `skeletons.jsonl` contains this complete record for reproducibility and auditing.
The public payload contains: ## Renderer boundary
The renderer receives only:
```text ```text
family, complexity, family
event_vocabulary, attribute_vocabulary, complexity
observation_protocol, support_cases, entity_types
query_case, answer_options initial_observations
events
query
answer_options
``` ```
It never contains: It does not receive:
```text ```text
id, hidden_system, gold_answer, world id
gold_present_probability, latent_trace hidden program or transition probabilities
latent initial-state attributes
complete final state
gold answer
proof event IDs or proof facts
generator checks
``` ```
## Current generated structure The renderer must map every symbol to one stable surface name, preserve all events and their direction and ordering, and produce one chronological fictional history. Every rendered initial fact and event line retains its `I####` or `V#####` audit ID. The evaluator rejects a rendering when any expected ID is absent. The renderer may compress formatting, but it may not summarize away distractors, state hidden rules, expose the proof, or directly assert the queried proposition.
Every system has at least four interacting rules: ## Complexity dimensions
- an unconditional probabilistic add event; The axes are deliberately independent:
- an attribute-gated probabilistic veto event;
- an event that probabilistically sets an invisible flag;
- a later event whose effect requires that flag.
Longer configurations add attribute-gated add or veto rules. Support evidence includes: ```text
length minimum gold-proof events
entity_count total typed entities
simulation_steps candidate evolution steps
event_count realized events; measured after simulation
```
- every event in isolation; Named proof presets are `short=4`, `medium=6`, and `long=8`. The task file separately controls world size and simulation duration.
- pairwise event combinations;
- both orders of the state-gated pair;
- low/high attribute counterfactual controls;
- repeated outcomes rather than rule annotations.
The query applies all events in order. Its complete combination is held out from support evidence, so the model must compose effects learned from smaller interventions. ## Planned extensions
Generated indices alternate between present-favored and absent-favored systems to prevent a one-label benchmark. The index is redacted from the renderer payload so this balancing mechanism cannot become a shortcut. It affects hidden parameter ranges, never the answer after generation; the exact enumerator remains authoritative. The IR is intended to grow without changing the visibility boundary. Useful next additions include:
## Complexity and determinism - relation deletion and expiry;
- explicit event causation chains;
`complexity` is the number of distinct events in the held-out sequence and the minimum number of corresponding hidden rules. It is independent of rendered token count. - multi-effect transactions and conflict policies;
- union and intersection queries;
| Preset | Ordered query events | - quantified `ALL` and `EXISTS` questions;
| --- | ---: | - temporal queries about intermediate rather than final state;
| `short` | 4 | - counterfactual interventions;
| `medium` | 6 | - executable nearest-neighbor, keyword, local-window, and shallow-proof baselines;
| `long` | 8 | - renderer fidelity verification against every source event.
Explicit values from 4 through 10 are accepted. `generate_skeletons(count, length, seed)` uses a local pseudorandom generator; the same code version and arguments produce equivalent hidden systems, samples, queries, and gold answers.
## Renderer obligations
The renderer may rename event codes, attributes, states, and outcomes into a coherent scenario. It must preserve every numeric count, attribute value, event order, trial boundary, query input, and answer label.
It must not:
- receive or state hidden rules;
- turn empirical correlations into declared laws;
- expose trigger probabilities, internal flags, latent traces, or the answer;
- collapse a repeated experiment into a deterministic statement;
- add causal or real-world assumptions;
- show the exact query combination as an already observed history.
Retaining the complete pre-render record makes rendering fidelity independently auditable.
## Extension points
The next DSL extensions can remain compatible with the three-layer boundary:
- multiple entities and relations between them;
- delayed actions, expiry, and multiple clock phases;
- priority, last-write, merge, and fallback conflict policies;
- partial observation and noisy sensors distinct from stochastic behavior;
- candidate-program enumeration to report identifiability;
- counterfactual and intervention queries in addition to outcome prediction.
Any extension must keep gold computation local, expose behavior rather than source rules, preserve seeded reproducibility, and add tests for execution order, conflict semantics, and renderer redaction.
+7 -6
View File
@@ -1,14 +1,15 @@
"""Behavioral hidden-system benchmark.""" """Behavioral hidden-system benchmark."""
from .models import BehaviorCase, BehaviorSkeleton, Condition, HiddenRule, HiddenSystem from .models import Entity, HiddenProgram, Query, TransitionRule, WorldEvent, WorldSkeleton
from .skeletons import generate_skeletons, resolve_length from .skeletons import generate_skeletons, resolve_length
__all__ = [ __all__ = [
"BehaviorCase", "Entity",
"BehaviorSkeleton", "HiddenProgram",
"Condition", "Query",
"HiddenRule", "TransitionRule",
"HiddenSystem", "WorldEvent",
"WorldSkeleton",
"generate_skeletons", "generate_skeletons",
"resolve_length", "resolve_length",
] ]
+11 -1
View File
@@ -29,6 +29,8 @@ def build_parser() -> argparse.ArgumentParser:
generate.add_argument("--count", type=int, default=3) generate.add_argument("--count", type=int, default=3)
generate.add_argument("--length", default="short") generate.add_argument("--length", default="short")
generate.add_argument("--seed", type=int, default=0) generate.add_argument("--seed", type=int, default=0)
generate.add_argument("--entity-count", type=int, default=128)
generate.add_argument("--simulation-steps", type=int, default=1000)
generate.add_argument("--output", type=Path) generate.add_argument("--output", type=Path)
run = subparsers.add_parser("run", help="run renderer and solver from a task JSON file") run = subparsers.add_parser("run", help="run renderer and solver from a task JSON file")
@@ -39,7 +41,13 @@ def build_parser() -> argparse.ArgumentParser:
def main(argv: list[str] | None = None) -> int: def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv) args = build_parser().parse_args(argv)
if args.command == "generate": if args.command == "generate":
skeletons = generate_skeletons(args.count, args.length, args.seed) skeletons = generate_skeletons(
args.count,
args.length,
args.seed,
entity_count=args.entity_count,
simulation_steps=args.simulation_steps,
)
lines = [json.dumps(item.to_dict(), ensure_ascii=False) for item in skeletons] lines = [json.dumps(item.to_dict(), ensure_ascii=False) for item in skeletons]
text = "\n".join(lines) + "\n" text = "\n".join(lines) + "\n"
if args.output: if args.output:
@@ -59,6 +67,8 @@ def main(argv: list[str] | None = None) -> int:
length=task["length"], length=task["length"],
seed=int(task["seed"]), seed=int(task["seed"]),
output_dir=Path(task["output_dir"]), output_dir=Path(task["output_dir"]),
entity_count=int(task.get("entity_count", 128)),
simulation_steps=int(task.get("simulation_steps", 1000)),
) )
print(json.dumps(summary, ensure_ascii=False, indent=2)) print(json.dumps(summary, ensure_ascii=False, indent=2))
return 0 return 0
+2 -2
View File
@@ -81,7 +81,7 @@ class LLMClient:
"schema": schema, "schema": schema,
} }
}, },
"max_output_tokens": 2500, "max_output_tokens": 12000,
} }
else: else:
path = "/chat/completions" path = "/chat/completions"
@@ -95,7 +95,7 @@ class LLMClient:
"type": "json_schema", "type": "json_schema",
"json_schema": {"name": schema_name, "strict": True, "schema": schema}, "json_schema": {"name": schema_name, "strict": True, "schema": schema},
}, },
"max_tokens": 2500, "max_tokens": 12000,
} }
if self.config.enable_thinking is not None: if self.config.enable_thinking is not None:
+43 -44
View File
@@ -5,78 +5,77 @@ from typing import Any
@dataclass(frozen=True) @dataclass(frozen=True)
class Condition: class Entity:
"""A predicate over immutable inputs or mutable simulator state.""" id: str
type: str
source: str
key: str
operator: str
value: Any
@dataclass(frozen=True) @dataclass(frozen=True)
class HiddenRule: class TransitionRule:
"""One executable rule in the hidden system language."""
id: str id: str
on_event: str trigger: str
conditions: tuple[Condition, ...] preconditions: tuple[str, ...]
probability: float probability: float
action: str effects: tuple[str, ...]
target: str
@dataclass(frozen=True) @dataclass(frozen=True)
class HiddenSystem: class HiddenProgram:
"""The latent program. This object must never be sent to the renderer.""" types: tuple[str, ...]
relations: tuple[str, ...]
initial_record: str operators: tuple[str, ...]
conflict_policy: str quantifiers: tuple[str, ...]
rules: tuple[HiddenRule, ...] structures: tuple[str, ...]
inference_rules: tuple[str, ...]
transition_rules: tuple[TransitionRule, ...]
@dataclass(frozen=True) @dataclass(frozen=True)
class BehaviorCase: class WorldEvent:
"""Observable inputs and aggregate output from repeated independent trials."""
id: str id: str
attributes: dict[str, Any] time: int
events: tuple[str, ...] kind: str
trials: int participants: tuple[str, ...]
observed_present: int | None arguments: dict[str, Any]
observed_absent: int | None
@dataclass(frozen=True) @dataclass(frozen=True)
class BehaviorSkeleton: class Query:
"""A hidden program plus its public behavioral evidence and answer-bearing query.""" predicate: str
arguments: tuple[str, ...]
time: int
@dataclass(frozen=True)
class WorldSkeleton:
id: str id: str
family: str family: str
complexity: int complexity: int
event_vocabulary: tuple[str, ...] entity_count: int
attribute_vocabulary: dict[str, str] simulation_steps: int
observation_protocol: tuple[str, ...] hidden_program: HiddenProgram
hidden_system: HiddenSystem entities: tuple[Entity, ...]
support_cases: tuple[BehaviorCase, ...] initial_state: dict[str, Any]
query_case: BehaviorCase events: tuple[WorldEvent, ...]
final_state: dict[str, Any]
query: Query
answer_options: tuple[str, ...] answer_options: tuple[str, ...]
gold_answer: str gold_answer: str
gold_present_probability: float proof_event_ids: tuple[str, ...]
latent_trace: tuple[str, ...] proof_facts: tuple[str, ...]
generator_checks: dict[str, Any]
def to_dict(self) -> dict[str, Any]: def to_dict(self) -> dict[str, Any]:
return asdict(self) return asdict(self)
def renderer_payload(self) -> dict[str, Any]: def renderer_payload(self) -> dict[str, Any]:
"""Expose behavior only: no program, probability, trace, or answer.""" """Expose realized history, never the source program, proof, or answer."""
return { return {
"family": self.family, "family": self.family,
"complexity": self.complexity, "complexity": self.complexity,
"event_vocabulary": self.event_vocabulary, "entity_types": {entity.id: entity.type for entity in self.entities},
"attribute_vocabulary": self.attribute_vocabulary, "initial_observations": self.initial_state["public_observations"],
"observation_protocol": self.observation_protocol, "events": [asdict(event) for event in self.events],
"support_cases": [asdict(case) for case in self.support_cases], "query": asdict(self.query),
"query_case": asdict(self.query_case),
"answer_options": self.answer_options, "answer_options": self.answer_options,
} }
+38 -20
View File
@@ -6,7 +6,7 @@ from dataclasses import asdict, dataclass
from pathlib import Path from pathlib import Path
from typing import Any, Protocol from typing import Any, Protocol
from .models import BehaviorSkeleton from .models import WorldSkeleton
from .skeletons import generate_skeletons from .skeletons import generate_skeletons
@@ -31,7 +31,6 @@ class EvaluationResult:
rendered_problem: str rendered_problem: str
answer_options: tuple[str, ...] answer_options: tuple[str, ...]
gold_answer: str gold_answer: str
gold_present_probability: float
solver_output: str solver_output: str
model_answer: str | None model_answer: str | None
format_compliant: bool format_compliant: bool
@@ -47,20 +46,20 @@ RENDER_SCHEMA = {
} }
def render_skeleton(client: JsonModel, skeleton: BehaviorSkeleton) -> str: def render_skeleton(client: JsonModel, skeleton: WorldSkeleton) -> str:
prompt = json.dumps(skeleton.renderer_payload(), ensure_ascii=False, indent=2) prompt = json.dumps(skeleton.renderer_payload(), ensure_ascii=False, indent=2)
response = client.request_json( response = client.request_json(
system=( system=(
"You turn behavioral observations from an unknown stochastic system into one self-contained " "Turn a large formal history from one simulated world into a self-contained natural-language "
"natural-language inductive reasoning problem. Invent a coherent concrete or fictional domain. " "reasoning problem. Invent one coherent concrete or fictional setting and assign stable names to "
"Give each event code and attribute a stable surface name, and preserve every case's attributes, " "every symbol. Render every initial observation and every realized event exactly once, preserving "
"event order, trial count, observed outcome counts, query, and exact answer labels. The histories " "time order, participants, relation direction, numeric values, and state changes. Render the query "
"are empirical observations, not explicit system laws: do not state or invent the hidden rules, " "at the end. The events are facts that actually occurred, not repeated probability experiments. "
"trigger probabilities, intermediate flags, exact query probability, or answer. Make clear that " "Never mention the generator, hidden program, probabilities, proof path, gold answer, or authoring "
"all trials use one unchanged system and that the question asks which final outcome is more likely, " "instructions. Do not directly state the queried proposition. Begin each rendered fact or event line "
"not for an exact probability. Present repeated histories in a compact table rather than verbose " "with its original I#### or V##### audit ID. Use compact chronological log lines rather than "
"paragraphs. State the supplied answer labels, but do not impose a response format or forbid an " "explanatory prose, but do not omit apparent distractors. State the answer labels; the "
"explanation; the evaluator supplies its own output protocol to the solver." "evaluator separately supplies the solver response protocol."
), ),
user=f"Render this intermediate representation:\n{prompt}", user=f"Render this intermediate representation:\n{prompt}",
schema_name="rendered_problem", schema_name="rendered_problem",
@@ -69,7 +68,16 @@ def render_skeleton(client: JsonModel, skeleton: BehaviorSkeleton) -> str:
problem = response.get("problem") problem = response.get("problem")
if not isinstance(problem, str) or not problem.strip(): if not isinstance(problem, str) or not problem.strip():
raise RuntimeError("renderer returned an empty problem") raise RuntimeError("renderer returned an empty problem")
return problem.strip() problem = problem.strip()
expected_ids = [item["id"] for item in skeleton.initial_state["public_observations"]]
expected_ids.extend(event.id for event in skeleton.events)
missing = [identifier for identifier in expected_ids if identifier not in problem]
if missing:
preview = ", ".join(missing[:10])
raise RuntimeError(
f"renderer omitted {len(missing)} initial observations or events; first missing IDs: {preview}"
)
return problem
def parse_marked_answer(output: str, answer_options: tuple[str, ...]) -> str: def parse_marked_answer(output: str, answer_options: tuple[str, ...]) -> str:
@@ -110,10 +118,10 @@ def solve_problem(client: JsonModel, problem: str, answer_options: tuple[str, ..
def request_solver_output(client: JsonModel, problem: str, answer_options: tuple[str, ...]) -> str: def request_solver_output(client: JsonModel, problem: str, answer_options: tuple[str, ...]) -> str:
return client.request_text( return client.request_text(
system=( system=(
"Infer the stable hidden mechanism from the repeated behavioral observations, then predict which " "Solve the question from the described world history. Track entity identity, relation direction, "
"query outcome is more likely. Event order, attributes, stochastic frequencies, interactions, and " "temporal updates, set inheritance, ordering, and relevant compositions while ignoring distractors. "
"counterfactual controls may matter. Do not demand an exact probability. Keep the explanation under " "Use only the stated observations and ordinary meanings explicitly established by the story. Keep "
"200 words, then put " "the explanation under 200 words, then put "
"the final answer on the last non-empty line in the exact form '#### <answer>', where <answer> " "the final answer on the last non-empty line in the exact form '#### <answer>', where <answer> "
"is exactly one permitted answer label. Do not use unstated real-world assumptions." "is exactly one permitted answer label. Do not use unstated real-world assumptions."
), ),
@@ -135,9 +143,17 @@ def run_evaluation(
length: str | int, length: str | int,
seed: int, seed: int,
output_dir: Path, output_dir: Path,
entity_count: int = 128,
simulation_steps: int = 1000,
) -> dict[str, Any]: ) -> dict[str, Any]:
output_dir.mkdir(parents=True, exist_ok=True) output_dir.mkdir(parents=True, exist_ok=True)
skeletons = generate_skeletons(count=count, length=length, seed=seed) skeletons = generate_skeletons(
count=count,
length=length,
seed=seed,
entity_count=entity_count,
simulation_steps=simulation_steps,
)
_write_jsonl(output_dir / "skeletons.jsonl", [item.to_dict() for item in skeletons]) _write_jsonl(output_dir / "skeletons.jsonl", [item.to_dict() for item in skeletons])
results: list[EvaluationResult] = [] results: list[EvaluationResult] = []
@@ -165,7 +181,6 @@ def run_evaluation(
rendered_problem=problem, rendered_problem=problem,
answer_options=skeleton.answer_options, answer_options=skeleton.answer_options,
gold_answer=skeleton.gold_answer, gold_answer=skeleton.gold_answer,
gold_present_probability=skeleton.gold_present_probability,
solver_output=solver_output, solver_output=solver_output,
model_answer=answer, model_answer=answer,
format_compliant=format_compliant, format_compliant=format_compliant,
@@ -189,6 +204,9 @@ def run_evaluation(
"length": length, "length": length,
"complexity": skeletons[0].complexity, "complexity": skeletons[0].complexity,
"seed": seed, "seed": seed,
"entity_count": entity_count,
"simulation_steps": simulation_steps,
"event_counts": [len(item.events) for item in skeletons],
} }
(output_dir / "summary.json").write_text( (output_dir / "summary.json").write_text(
json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8" json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
+579 -216
View File
@@ -1,26 +1,33 @@
from __future__ import annotations from __future__ import annotations
import random import random
from dataclasses import dataclass from collections import deque
from itertools import combinations from dataclasses import dataclass, field
from typing import Any from typing import Any
from .models import BehaviorCase, BehaviorSkeleton, Condition, HiddenRule, HiddenSystem from .models import Entity, HiddenProgram, Query, TransitionRule, WorldEvent, WorldSkeleton
LENGTH_PRESETS = {"short": 4, "medium": 6, "long": 8} LENGTH_PRESETS = {"short": 4, "medium": 6, "long": 8}
ANSWER_OPTIONS = ("PRESENT", "ABSENT") ANSWER_OPTIONS = ("YES", "NO")
@dataclass(frozen=True) @dataclass
class _State: class _World:
present: bool = False locations: dict[str, str]
invalidated: bool = False order_level: dict[str, int]
flags: frozenset[str] = frozenset() set_level: dict[str, int]
above: dict[tuple[str, str], str] = field(default_factory=dict)
subsets: dict[tuple[str, str], str] = field(default_factory=dict)
members: dict[tuple[str, str], str] = field(default_factory=dict)
disjoint: dict[frozenset[str], str] = field(default_factory=dict)
connected: set[frozenset[str]] = field(default_factory=set)
numbers: dict[str, int] = field(default_factory=dict)
flags: dict[tuple[str, str], bool] = field(default_factory=dict)
def resolve_length(value: str | int) -> int: def resolve_length(value: str | int) -> int:
"""Resolve the number of distinct events in the held-out query.""" """Resolve required proof length, independently of event-log or text length."""
if isinstance(value, int): if isinstance(value, int):
result = value result = value
elif value in LENGTH_PRESETS: elif value in LENGTH_PRESETS:
@@ -29,246 +36,602 @@ def resolve_length(value: str | int) -> int:
try: try:
result = int(value) result = int(value)
except ValueError as exc: except ValueError as exc:
choices = ", ".join(LENGTH_PRESETS) raise ValueError(f"length must be an integer or one of: {', '.join(LENGTH_PRESETS)}") from exc
raise ValueError(f"length must be an integer or one of: {choices}") from exc if not 3 <= result <= 10:
if not 4 <= result <= 10: raise ValueError("length must be between 3 and 10")
raise ValueError("length must be between 4 and 10")
return result return result
def _condition_holds(condition: Condition, attributes: dict[str, Any], state: _State) -> bool: def _program() -> HiddenProgram:
if condition.source == "attribute": return HiddenProgram(
actual = attributes.get(condition.key) types=("Entity", "Set", "Region", "Number", "Boolean", "State"),
elif condition.source == "state": relations=(
actual = condition.key in state.flags "EQUAL",
else: "NOT_EQUAL",
raise ValueError(f"unknown condition source: {condition.source}") "ABOVE",
"BELOW",
"MEMBER",
"SUBSET",
"DISJOINT",
"ADJACENT",
"CONNECTED",
"CAUSES",
"LOCATED_IN",
),
operators=(
"NOT",
"AND",
"OR",
"IMPLIES",
"ADD",
"SUBTRACT",
"COMPARE",
"UNION",
"INTERSECTION",
"MOVE",
"SWAP",
"SET",
"COMPOSE",
),
quantifiers=("ALL", "EXISTS"),
structures=("CHAIN", "BRANCH", "MERGE", "CYCLE", "NEST", "CONDITION", "SEQUENCE"),
inference_rules=(
"ABOVE(x,y) AND ABOVE(y,z) IMPLIES ABOVE(x,z)",
"ABOVE(x,y) IMPLIES BELOW(y,x)",
"MEMBER(x,A) AND SUBSET(A,B) IMPLIES MEMBER(x,B)",
"MEMBER(x,A) AND DISJOINT(A,B) IMPLIES NOT MEMBER(x,B)",
"CONNECTED is symmetric",
"Numeric updates compose in temporal order",
),
transition_rules=(
TransitionRule(
"move",
"SELECT_ENTITY",
("target region differs from current region",),
0.72,
("MOVE(entity, region)",),
),
TransitionRule(
"encounter-order",
"COLOCATED_PAIR",
("entities share a region", "latent order levels are adjacent"),
0.78,
("SET ABOVE(higher, lower)", "SET BELOW(lower, higher)"),
),
TransitionRule(
"membership",
"ENTITY_SET_CONTACT",
("set is at base level",),
0.68,
("SET MEMBER(entity, set)",),
),
TransitionRule(
"subset",
"SET_PAIR_CONTACT",
("set levels are adjacent",),
0.76,
("SET SUBSET(lower, higher)",),
),
TransitionRule(
"disjoint",
"SET_PAIR_CONTACT",
("sets have equal level", "sets are distinct"),
0.64,
("SET DISJOINT(left, right)",),
),
TransitionRule(
"numeric-update",
"COUNTER_PULSE",
(),
0.82,
("ADD or SUBTRACT(counter, amount)",),
),
TransitionRule(
"connect",
"ENTITY_PAIR_CONTACT",
(),
0.55,
("SET CONNECTED(left, right)",),
),
TransitionRule(
"state-change",
"ENTITY_SIGNAL",
(),
0.58,
("SET Boolean state",),
),
),
)
operations = {
"EQ": lambda left, right: left == right, def _entities(entity_count: int) -> tuple[list[Entity], dict[str, list[str]]]:
"LE": lambda left, right: left <= right, if entity_count < 48:
"GT": lambda left, right: left > right, raise ValueError("entity_count must be at least 48")
"IS_SET": lambda left, right: left is bool(right), region_count = max(4, entity_count // 32)
set_count = max(24, entity_count // 5)
counter_count = max(8, entity_count // 16)
agent_count = entity_count - region_count - set_count - counter_count
groups = {
"Entity": [f"E{i:03d}" for i in range(agent_count)],
"Set": [f"S{i:03d}" for i in range(set_count)],
"Region": [f"R{i:03d}" for i in range(region_count)],
"Number": [f"N{i:03d}" for i in range(counter_count)],
} }
try: entities = [Entity(identifier, kind) for kind, values in groups.items() for identifier in values]
return operations[condition.operator](actual, condition.value) return entities, groups
except KeyError as exc:
raise ValueError(f"unknown condition operator: {condition.operator}") from exc
def _apply_action(state: _State, rule: HiddenRule) -> _State: def _initial_world(
if rule.action == "SET_FLAG": groups: dict[str, list[str]], rng: random.Random, proof_length: int
return _State(state.present, state.invalidated, state.flags | {rule.target}) ) -> tuple[_World, dict[str, Any]]:
if rule.action == "ADD_RECORD": agents, sets, regions, counters = (
if state.invalidated: groups["Entity"],
return state groups["Set"],
return _State(True, False, state.flags) groups["Region"],
if rule.action == "INVALIDATE_RECORD": groups["Number"],
return _State(False, True, state.flags)
raise ValueError(f"unknown action: {rule.action}")
def exact_present_probability(
system: HiddenSystem, attributes: dict[str, Any], events: tuple[str, ...]
) -> float:
"""Enumerate all stochastic branches while respecting event and state order."""
initial = _State(present=system.initial_record == "PRESENT")
distribution: dict[_State, float] = {initial: 1.0}
for event in events:
for rule in (item for item in system.rules if item.on_event == event):
next_distribution: dict[_State, float] = {}
for state, mass in distribution.items():
if not all(_condition_holds(item, attributes, state) for item in rule.conditions):
next_distribution[state] = next_distribution.get(state, 0.0) + mass
continue
inactive_mass = mass * (1.0 - rule.probability)
active_state = _apply_action(state, rule)
next_distribution[state] = next_distribution.get(state, 0.0) + inactive_mass
next_distribution[active_state] = (
next_distribution.get(active_state, 0.0) + mass * rule.probability
)
distribution = next_distribution
return sum(mass for state, mass in distribution.items() if state.present)
def _simulate_once(
system: HiddenSystem,
attributes: dict[str, Any],
events: tuple[str, ...],
rng: random.Random,
) -> bool:
state = _State(present=system.initial_record == "PRESENT")
for event in events:
for rule in (item for item in system.rules if item.on_event == event):
if all(_condition_holds(item, attributes, state) for item in rule.conditions):
if rng.random() < rule.probability:
state = _apply_action(state, rule)
return state.present
def _observe_case(
case_id: str,
system: HiddenSystem,
attributes: dict[str, Any],
events: tuple[str, ...],
trials: int,
rng: random.Random,
) -> BehaviorCase:
present = sum(_simulate_once(system, attributes, events, rng) for _ in range(trials))
return BehaviorCase(
id=case_id,
attributes=attributes,
events=events,
trials=trials,
observed_present=present,
observed_absent=trials - present,
) )
order_levels = max(proof_length + 2, 8)
set_levels = max(proof_length + 1, 8)
def _build_system(index: int, complexity: int, rng: random.Random) -> HiddenSystem: locations = {agent: rng.choice(regions) for agent in agents}
"""Build a finite stochastic program with a gated rule and veto conflicts.""" world = _World(
present_favored = index % 2 == 0 locations=locations,
add_probability = rng.choice((0.85, 0.9, 0.95)) if present_favored else rng.choice((0.65, 0.75)) order_level={agent: index % order_levels for index, agent in enumerate(agents)},
veto_probability = rng.choice((0.15, 0.25)) if present_favored else rng.choice((0.7, 0.8, 0.9)) set_level={item: index % set_levels for index, item in enumerate(sets)},
marker_probability = rng.choice((0.6, 0.75, 0.9)) numbers={counter: rng.randint(10, 90) for counter in counters},
gated_action = "ADD_RECORD" if present_favored else "INVALIDATE_RECORD" )
gated_probability = rng.choice((0.6, 0.75, 0.85)) public = [
{"id": f"I{index:04d}", "predicate": "LOCATED_IN", "arguments": [agent, region]}
rules = [ for index, (agent, region) in enumerate(locations.items())
HiddenRule("R0", "E0", (), add_probability, "ADD_RECORD", "record"),
HiddenRule(
"R1",
"E1",
(Condition("attribute", "load", "LE", 4),),
veto_probability,
"INVALIDATE_RECORD",
"record",
),
HiddenRule("R2", "E2", (), marker_probability, "SET_FLAG", "marked"),
HiddenRule(
"R3",
"E3",
(Condition("state", "marked", "IS_SET", True),),
gated_probability,
gated_action,
"record",
),
] ]
for number in range(4, complexity): offset = len(public)
action = "ADD_RECORD" if (number + index) % 3 else "INVALIDATE_RECORD" public.extend(
probability = rng.choice((0.2, 0.35, 0.5)) {
condition = Condition("attribute", "class", "EQ", "priority") "id": f"I{offset + index:04d}",
rules.append( "predicate": "INITIAL_VALUE",
HiddenRule(f"R{number}", f"E{number}", (condition,), probability, action, "record") "arguments": [counter, value],
) }
return HiddenSystem( for index, (counter, value) in enumerate(world.numbers.items())
initial_record="ABSENT",
conflict_policy="PERSISTENT_INVALIDATION_VETO",
rules=tuple(rules),
) )
hidden = {
"order_level": world.order_level,
"set_level": world.set_level,
"initial_locations": dict(locations),
"initial_numbers": dict(world.numbers),
}
return world, {"public_observations": public, "hidden_state": hidden}
def _support_event_sets(complexity: int) -> list[tuple[str, ...]]: def _event(
events = [f"E{number}" for number in range(complexity)] events: list[WorldEvent],
cases: list[tuple[str, ...]] = [(event,) for event in events] time: int,
cases.extend((("E2", "E3"), ("E3", "E2"), ("E0", "E1"))) kind: str,
cases.extend(tuple(pair) for pair in combinations(events, 2) if tuple(pair) not in cases) participants: tuple[str, ...],
return cases **arguments: Any,
) -> WorldEvent:
item = WorldEvent(f"V{len(events):05d}", time, kind, participants, arguments)
events.append(item)
return item
def _generate_one(index: int, complexity: int, rng: random.Random) -> BehaviorSkeleton: def _simulate(
system = _build_system(index, complexity, rng) world: _World,
low = {"load": 3, "class": "priority"} groups: dict[str, list[str]],
high = {"load": 8, "class": "standard"} steps: int,
query_attributes = dict(low) rng: random.Random,
query_events = tuple(f"E{number}" for number in range(complexity)) ) -> list[WorldEvent]:
probability = exact_present_probability(system, query_attributes, query_events) agents, sets, regions, counters = (
if abs(probability - 0.5) < 0.08: groups["Entity"],
raise RuntimeError("generated query has an insufficient probability margin") groups["Set"],
gold = "PRESENT" if probability > 0.5 else "ABSENT" groups["Region"],
groups["Number"],
)
events: list[WorldEvent] = []
kinds = ("ENCOUNTER", "MOVE", "MEMBER", "SUBSET", "DISJOINT", "NUMBER", "CONNECT", "STATE")
weights = (30, 12, 11, 15, 7, 11, 8, 6)
probabilities = {
"ENCOUNTER": 0.78,
"MOVE": 0.72,
"MEMBER": 0.68,
"SUBSET": 0.76,
"DISJOINT": 0.64,
"NUMBER": 0.82,
"CONNECT": 0.55,
"STATE": 0.58,
}
trials = 48 if complexity <= 4 else 64 for time in range(1, steps + 1):
support: list[BehaviorCase] = [] kind = rng.choices(kinds, weights=weights, k=1)[0]
for case_number, events in enumerate(_support_event_sets(complexity)): if rng.random() >= probabilities[kind]:
attributes = low if case_number % 3 else high continue
support.append(
_observe_case( if kind == "MOVE":
f"history-{case_number:03d}", system, dict(attributes), events, trials, rng agent = rng.choice(agents)
choices = [region for region in regions if region != world.locations[agent]]
destination = rng.choice(choices)
origin = world.locations[agent]
world.locations[agent] = destination
_event(events, time, "MOVE", (agent, origin, destination), origin=origin, destination=destination)
elif kind == "ENCOUNTER":
lower_level = rng.randrange(max(world.order_level.values()))
lower = [item for item in agents if world.order_level[item] == lower_level]
higher = [item for item in agents if world.order_level[item] == lower_level + 1]
if not lower or not higher:
continue
low, high = rng.choice(lower), rng.choice(higher)
if world.locations[low] != world.locations[high] or (high, low) in world.above:
continue
item = _event(
events,
time,
"RELATE",
(high, low),
predicate="ABOVE",
subject=high,
object=low,
region=world.locations[high],
)
world.above[(high, low)] = item.id
elif kind == "MEMBER":
agent = rng.choice(agents)
base_sets = [item for item in sets if world.set_level[item] == 0]
container = rng.choice(base_sets)
if (agent, container) in world.members:
continue
item = _event(
events,
time,
"RELATE",
(agent, container),
predicate="MEMBER",
subject=agent,
object=container,
)
world.members[(agent, container)] = item.id
elif kind == "SUBSET":
lower_level = rng.randrange(max(world.set_level.values()))
lower = [item for item in sets if world.set_level[item] == lower_level]
higher = [item for item in sets if world.set_level[item] == lower_level + 1]
child, parent = rng.choice(lower), rng.choice(higher)
if (child, parent) in world.subsets or any(left == child for left, _right in world.subsets):
continue
item = _event(
events,
time,
"RELATE",
(child, parent),
predicate="SUBSET",
subject=child,
object=parent,
)
world.subsets[(child, parent)] = item.id
elif kind == "DISJOINT":
level = rng.randrange(max(world.set_level.values()) + 1)
same_level = [item for item in sets if world.set_level[item] == level]
if len(same_level) < 2:
continue
left, right = rng.sample(same_level, 2)
key = frozenset((left, right))
if key in world.disjoint:
continue
item = _event(
events,
time,
"RELATE",
(left, right),
predicate="DISJOINT",
subject=left,
object=right,
)
world.disjoint[key] = item.id
elif kind == "NUMBER":
counter = rng.choice(counters)
delta = rng.choice((-7, -5, -3, 2, 4, 6))
before = world.numbers[counter]
after = max(0, before + delta)
world.numbers[counter] = after
_event(
events,
time,
"UPDATE_NUMBER",
(counter,),
operation="ADD" if delta >= 0 else "SUBTRACT",
amount=abs(delta),
before=before,
after=after,
)
elif kind == "CONNECT":
left, right = rng.sample(agents, 2)
key = frozenset((left, right))
if key in world.connected:
continue
world.connected.add(key)
_event(
events,
time,
"RELATE",
(left, right),
predicate="CONNECTED",
subject=left,
object=right,
)
else:
agent = rng.choice(agents)
flag = rng.choice(("ACTIVE", "SEALED", "VERIFIED"))
before = world.flags.get((agent, flag), False)
after = not before
world.flags[(agent, flag)] = after
_event(
events,
time,
"SET_STATE",
(agent,),
state=flag,
before=before,
after=after,
)
return events
def _shortest_path(
edges: dict[tuple[str, str], str], minimum: int
) -> tuple[str, str, tuple[str, ...], tuple[str, ...]] | None:
adjacency: dict[str, list[tuple[str, str]]] = {}
for (left, right), event_id in edges.items():
adjacency.setdefault(left, []).append((right, event_id))
candidates: list[tuple[str, str, tuple[str, ...], tuple[str, ...]]] = []
for source in adjacency:
queue = deque([(source, (source,), ())])
seen = {source}
while queue:
node, nodes, event_ids = queue.popleft()
if len(event_ids) >= minimum:
candidates.append((source, node, nodes, event_ids))
continue
for target, event_id in adjacency.get(node, []):
if target not in seen:
seen.add(target)
queue.append((target, nodes + (target,), event_ids + (event_id,)))
if not candidates:
return None
candidates.sort(key=lambda item: (len(item[3]), item[0], item[1]))
return candidates[0]
def _membership_path(
world: _World, minimum: int
) -> tuple[str, str, tuple[str, ...], tuple[str, ...]] | None:
subset_path = _shortest_path(world.subsets, minimum - 1)
if subset_path is None:
return None
candidates = []
for (entity, base_set), member_event in world.members.items():
path = _shortest_path_from(world.subsets, base_set, minimum - 1)
if path:
target, nodes, event_ids = path
candidates.append((entity, target, (member_event,) + event_ids, (entity,) + nodes))
if not candidates:
return None
candidates.sort(key=lambda item: (len(item[2]), item[0], item[1]))
entity, target, event_ids, nodes = candidates[0]
return entity, target, nodes, event_ids
def _membership_negative(
world: _World, minimum: int
) -> tuple[str, str, tuple[str, ...], tuple[str, ...]] | None:
adjacency: dict[str, list[tuple[str, str]]] = {}
for (left, right), event_id in world.subsets.items():
adjacency.setdefault(left, []).append((right, event_id))
for (entity, base_set), member_event in sorted(world.members.items()):
reachable = {base_set}
queue = deque([(base_set, (base_set,), ())])
paths: list[tuple[str, tuple[str, ...], tuple[str, ...]]] = []
while queue:
node, nodes, event_ids = queue.popleft()
paths.append((node, nodes, event_ids))
for target, event_id in adjacency.get(node, []):
if target not in reachable:
reachable.add(target)
queue.append((target, nodes + (target,), event_ids + (event_id,)))
for known_set, nodes, subset_events in paths:
if 1 + len(subset_events) < minimum - 1:
continue
for pair, disjoint_event in world.disjoint.items():
if known_set not in pair:
continue
other = next(item for item in pair if item != known_set)
if other in reachable:
continue
proof_ids = (member_event,) + subset_events + (disjoint_event,)
proof_nodes = (entity,) + nodes + (other,)
return entity, other, proof_nodes, proof_ids
return None
def _shortest_path_from(
edges: dict[tuple[str, str], str], source: str, minimum: int
) -> tuple[str, tuple[str, ...], tuple[str, ...]] | None:
adjacency: dict[str, list[tuple[str, str]]] = {}
for (left, right), event_id in edges.items():
adjacency.setdefault(left, []).append((right, event_id))
queue = deque([(source, (source,), ())])
seen = {source}
while queue:
node, nodes, event_ids = queue.popleft()
if len(event_ids) >= minimum:
return node, nodes, event_ids
for target, event_id in adjacency.get(node, []):
if target not in seen:
seen.add(target)
queue.append((target, nodes + (target,), event_ids + (event_id,)))
return None
def _choose_query(
index: int, complexity: int, world: _World, events: list[WorldEvent]
) -> tuple[Query, str, tuple[str, ...], tuple[str, ...], str] | None:
if index % 2 == 0:
path = _shortest_path(world.above, complexity)
if path is None:
return None
source, target, nodes, proof_ids = path
positive = index % 4 == 0
arguments = (source, target) if positive else (target, source)
gold = "YES" if positive else "NO"
facts = tuple(
f"ABOVE({nodes[position]},{nodes[position + 1]})"
for position in range(len(nodes) - 1)
)
if not positive:
facts += (f"ABOVE({source},{target}) therefore NOT ABOVE({target},{source})",)
return Query("ABOVE", arguments, events[-1].time), gold, proof_ids, facts, "spatial_order"
if index % 4 == 1:
negative = _membership_negative(world, complexity)
if negative is None:
return None
entity, target, nodes, proof_ids = negative
known_set, disjoint_set = nodes[-2], nodes[-1]
facts = [f"MEMBER({entity},{nodes[1]})"]
facts.extend(
f"SUBSET({nodes[position]},{nodes[position + 1]})"
for position in range(1, len(nodes) - 2)
)
facts.extend(
(
f"DISJOINT({known_set},{disjoint_set})",
f"therefore NOT MEMBER({entity},{target})",
) )
) )
# Keeping events fixed while changing load makes the veto guard observable. return (
next_id = len(support) Query("MEMBER", (entity, target), events[-1].time),
support.append( "NO",
_observe_case(f"history-{next_id:03d}", system, dict(low), ("E0", "E1"), trials, rng) proof_ids,
) tuple(facts),
support.append( "membership_disjointness",
_observe_case(
f"history-{next_id + 1:03d}", system, dict(high), ("E0", "E1"), trials, rng
) )
membership = _membership_path(world, complexity)
if membership is None:
return None
entity, target, nodes, proof_ids = membership
facts = [f"MEMBER({entity},{nodes[1]})"]
facts.extend(
f"SUBSET({nodes[position]},{nodes[position + 1]})"
for position in range(1, len(nodes) - 1)
) )
support.append( return (
_observe_case( Query("MEMBER", (entity, target), events[-1].time),
f"history-{next_id + 2:03d}", "YES",
system, proof_ids,
dict(low), tuple(facts),
("E0", "E2", "E3"), "membership_inheritance",
trials,
rng,
)
) )
query = BehaviorCase(
id="query", def _final_state(world: _World) -> dict[str, Any]:
attributes=query_attributes, return {
events=query_events, "locations": dict(world.locations),
trials=1, "above": [[left, right] for left, right in sorted(world.above)],
observed_present=None, "subsets": [[left, right] for left, right in sorted(world.subsets)],
observed_absent=None, "members": [[entity, container] for entity, container in sorted(world.members)],
"disjoint": [sorted(pair) for pair in sorted(world.disjoint, key=lambda item: sorted(item))],
"connected": [sorted(pair) for pair in sorted(world.connected, key=lambda item: sorted(item))],
"numbers": dict(world.numbers),
"flags": [
{"entity": entity, "state": state, "value": value}
for (entity, state), value in sorted(world.flags.items())
],
}
def _generate_one(
index: int,
complexity: int,
entity_count: int,
simulation_steps: int,
rng: random.Random,
) -> WorldSkeleton:
entities, groups = _entities(entity_count)
world, initial_state = _initial_world(groups, rng, complexity)
events = _simulate(world, groups, simulation_steps, rng)
if not events:
raise RuntimeError("simulation produced no events")
selected = _choose_query(index, complexity, world, events)
if selected is None:
raise RuntimeError("simulation did not produce a sufficiently deep proof")
query, gold, proof_ids, proof_facts, query_family = selected
direct = any(
event.kind == "RELATE"
and event.arguments.get("predicate") == query.predicate
and (event.arguments.get("subject"), event.arguments.get("object")) == query.arguments
for event in events
) )
trace = tuple( if direct:
[f"query events: {' -> '.join(query_events)}"] raise RuntimeError("query was directly stated in the event history")
+ [ checks = {
f"{rule.id}: on {rule.on_event}, p={rule.probability}, {rule.action}({rule.target})" "direct_query_event_absent": True,
for rule in system.rules "proof_length": len(proof_ids),
] "minimum_proof_length": complexity,
+ [f"exact P(PRESENT)={probability:.8f}", f"gold={gold}"] "event_count": len(events),
) "proof_event_ratio": len(proof_ids) / len(events),
return BehaviorSkeleton( "query_family": query_family,
id=f"dynamic-{index:04d}", "weak_baselines": {
family="stochastic_behavioral_system", "direct_fact": False,
"proof_shorter_than_requested": len(proof_ids) < complexity,
"proof_is_large_fraction_of_history": len(proof_ids) / len(events) >= 0.05,
},
}
return WorldSkeleton(
id=f"world-{index:04d}",
family="dynamic_multi_logic_world",
complexity=complexity, complexity=complexity,
event_vocabulary=tuple(f"E{number}" for number in range(complexity)), entity_count=entity_count,
attribute_vocabulary={"load": "integer", "class": "category"}, simulation_steps=simulation_steps,
observation_protocol=( hidden_program=_program(),
"Each trial begins with an absent record and no internal flags.", entities=tuple(entities),
"Events occur from left to right; the same hidden system is used in every trial.", initial_state=initial_state,
"PRESENT and ABSENT counts are final observations over independent repeated trials.", events=tuple(events),
"For the query, choose which final outcome is more likely; an exact probability is not required.", final_state=_final_state(world),
), query=query,
hidden_system=system,
support_cases=tuple(support),
query_case=query,
answer_options=ANSWER_OPTIONS, answer_options=ANSWER_OPTIONS,
gold_answer=gold, gold_answer=gold,
gold_present_probability=probability, proof_event_ids=proof_ids,
latent_trace=trace, proof_facts=proof_facts,
generator_checks=checks,
) )
def generate_skeletons(count: int, length: str | int, seed: int = 0) -> list[BehaviorSkeleton]: def generate_skeletons(
count: int,
length: str | int,
seed: int = 0,
entity_count: int = 128,
simulation_steps: int = 1000,
) -> list[WorldSkeleton]:
if count < 1: if count < 1:
raise ValueError("count must be positive") raise ValueError("count must be positive")
if simulation_steps < 100:
raise ValueError("simulation_steps must be at least 100")
complexity = resolve_length(length) complexity = resolve_length(length)
rng = random.Random(seed) rng = random.Random(seed)
generated: list[BehaviorSkeleton] = [] generated: list[WorldSkeleton] = []
for index in range(count): for index in range(count):
for _attempt in range(100): for _attempt in range(60):
try: try:
generated.append(_generate_one(index, complexity, rng)) generated.append(
_generate_one(index, complexity, entity_count, simulation_steps, rng)
)
break break
except RuntimeError as exc: except RuntimeError:
if "insufficient probability margin" not in str(exc): continue
raise
else: else:
raise RuntimeError("could not generate a query with a stable probability margin") raise RuntimeError("could not generate a world with the requested proof complexity")
return generated return generated
+2 -2
View File
@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "logic-skeleton-bench" name = "logic-skeleton-bench"
version = "0.2.0" version = "0.3.0"
description = "Infer stochastic hidden systems from rendered behavioral observations." description = "Generate and solve proof-bearing histories from large evolving worlds."
readme = "README.md" readme = "README.md"
requires-python = ">=3.11" requires-python = ">=3.11"
license = { text = "MIT" } license = { text = "MIT" }
+3 -1
View File
@@ -1,8 +1,10 @@
{ {
"name": "short-smoke", "name": "short-smoke",
"description": "Infer five short stochastic hidden systems from behavioral histories.", "description": "Solve five proof-bearing queries from large simulated world histories.",
"count": 5, "count": 5,
"length": "short", "length": "short",
"entity_count": 128,
"simulation_steps": 1000,
"seed": 20260922, "seed": 20260922,
"output_dir": "artifacts/short-smoke" "output_dir": "artifacts/short-smoke"
} }
+30 -14
View File
@@ -1,3 +1,4 @@
import json
import tempfile import tempfile
import unittest import unittest
from pathlib import Path from pathlib import Path
@@ -21,21 +22,23 @@ class FailingSolver:
raise RuntimeError("solver failed") raise RuntimeError("solver failed")
class AuditRenderer:
def request_json(self, **kwargs):
payload = json.loads(kwargs["user"][kwargs["user"].find("{") :])
identifiers = [item["id"] for item in payload["initial_observations"]]
identifiers.extend(item["id"] for item in payload["events"])
return {"problem": "A compact behavioral problem.\n" + "\n".join(identifiers)}
class PipelineTests(unittest.TestCase): class PipelineTests(unittest.TestCase):
def test_end_to_end_writes_skeletons_before_scored_results(self): def test_end_to_end_writes_skeletons_before_scored_results(self):
renderer = QueueClient([ renderer = AuditRenderer()
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
])
solver = QueueClient([ solver = QueueClient([
"The positive behavior dominates.\n#### PRESENT", "The relation follows through the event chain.\n#### YES",
"The veto behavior dominates.\n#### ABSENT", "The target set is disjoint from the inherited membership.\n#### NO",
"The positive behavior dominates.\n#### PRESENT", "The reverse ordering is false.\n#### NO",
"The veto behavior dominates.\n#### ABSENT", "Membership propagates through the nested sets.\n#### YES",
"The positive behavior dominates.\n#### PRESENT", "The relation follows through the event chain.\n#### YES",
]) ])
with tempfile.TemporaryDirectory() as directory: with tempfile.TemporaryDirectory() as directory:
output = Path(directory) output = Path(directory)
@@ -70,7 +73,7 @@ class PipelineTests(unittest.TestCase):
self.assertEqual(parse_solver_answer("Reasoning.\n#### YES", ("YES", "NO")), ("YES", True)) self.assertEqual(parse_solver_answer("Reasoning.\n#### YES", ("YES", "NO")), ("YES", True))
def test_rendered_problem_is_checkpointed_before_solver_failure(self): def test_rendered_problem_is_checkpointed_before_solver_failure(self):
renderer = QueueClient([{"problem": "A compact behavioral problem."}]) renderer = AuditRenderer()
with tempfile.TemporaryDirectory() as directory: with tempfile.TemporaryDirectory() as directory:
output = Path(directory) output = Path(directory)
with self.assertRaisesRegex(RuntimeError, "solver failed"): with self.assertRaisesRegex(RuntimeError, "solver failed"):
@@ -85,7 +88,7 @@ class PipelineTests(unittest.TestCase):
self.assertIn("A compact behavioral problem.", (output / "rendered.jsonl").read_text()) self.assertIn("A compact behavioral problem.", (output / "rendered.jsonl").read_text())
def test_unparseable_solver_output_is_scored_without_stopping_batch(self): def test_unparseable_solver_output_is_scored_without_stopping_batch(self):
renderer = QueueClient([{"problem": "A compact behavioral problem."}]) renderer = AuditRenderer()
solver = QueueClient(["The response ended before its final label."]) solver = QueueClient(["The response ended before its final label."])
with tempfile.TemporaryDirectory() as directory: with tempfile.TemporaryDirectory() as directory:
output = Path(directory) output = Path(directory)
@@ -101,6 +104,19 @@ class PipelineTests(unittest.TestCase):
self.assertEqual(summary["correct"], 0) self.assertEqual(summary["correct"], 0)
self.assertIn('"parse_error":', (output / "results.jsonl").read_text()) self.assertIn('"parse_error":', (output / "results.jsonl").read_text())
def test_renderer_omission_is_rejected_before_solving(self):
renderer = QueueClient([{"problem": "This rendering omitted its audit IDs."}])
with tempfile.TemporaryDirectory() as directory:
with self.assertRaisesRegex(RuntimeError, "renderer omitted"):
run_evaluation(
renderer=renderer,
solver=FailingSolver(),
count=1,
length="short",
seed=1,
output_dir=Path(directory),
)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+47 -48
View File
@@ -1,75 +1,74 @@
import unittest import unittest
from logicbench.models import Condition, HiddenRule, HiddenSystem from logicbench.skeletons import generate_skeletons, resolve_length
from logicbench.skeletons import exact_present_probability, generate_skeletons, resolve_length
class SkeletonTests(unittest.TestCase): class SkeletonTests(unittest.TestCase):
def test_short_means_four_query_events(self): def test_short_world_has_many_entities_and_realized_events(self):
skeletons = generate_skeletons(count=3, length="short", seed=42) skeleton = generate_skeletons(1, "short", seed=42)[0]
self.assertEqual([item.complexity for item in skeletons], [4, 4, 4]) self.assertEqual(skeleton.entity_count, 128)
self.assertTrue(all(len(item.query_case.events) == 4 for item in skeletons)) self.assertEqual(skeleton.simulation_steps, 1000)
self.assertEqual( self.assertGreater(len(skeleton.events), 300)
[item.gold_answer for item in skeletons], ["PRESENT", "ABSENT", "PRESENT"] self.assertGreaterEqual(len(skeleton.proof_event_ids), 4)
) self.assertLess(skeleton.generator_checks["proof_event_ratio"], 0.05)
def test_generation_is_deterministic(self): def test_generation_is_deterministic(self):
first = [item.to_dict() for item in generate_skeletons(6, 4, seed=7)] first = [item.to_dict() for item in generate_skeletons(4, 4, seed=7)]
second = [item.to_dict() for item in generate_skeletons(6, 4, seed=7)] second = [item.to_dict() for item in generate_skeletons(4, 4, seed=7)]
self.assertEqual(first, second) self.assertEqual(first, second)
def test_renderer_receives_behavior_but_no_latent_information(self): def test_renderer_receives_realized_history_but_no_program_or_answer(self):
skeleton = generate_skeletons(1, "short", seed=1)[0] skeleton = generate_skeletons(1, "short", seed=1)[0]
payload = skeleton.renderer_payload() payload = skeleton.renderer_payload()
for secret in ( for secret in (
"id", "id",
"hidden_system", "hidden_program",
"hidden_state",
"final_state",
"gold_answer", "gold_answer",
"gold_present_probability", "proof_event_ids",
"latent_trace", "proof_facts",
"generator_checks",
): ):
self.assertNotIn(secret, payload) self.assertNotIn(secret, payload)
self.assertIn("support_cases", payload) self.assertEqual(len(payload["events"]), len(skeleton.events))
self.assertIsNone(payload["query_case"]["observed_present"]) self.assertEqual(len(payload["entity_types"]), 128)
def test_exact_enumerator_applies_persistent_veto(self): def test_query_is_not_directly_stated_and_proof_events_exist(self):
system = HiddenSystem( for skeleton in generate_skeletons(5, "short", seed=20260922):
initial_record="ABSENT", query_pair = skeleton.query.arguments
conflict_policy="PERSISTENT_INVALIDATION_VETO", direct = [
rules=( event
HiddenRule("add", "A", (), 0.8, "ADD_RECORD", "record"), for event in skeleton.events
HiddenRule("veto", "B", (), 0.25, "INVALIDATE_RECORD", "record"), if event.kind == "RELATE"
), and event.arguments.get("predicate") == skeleton.query.predicate
) and (event.arguments.get("subject"), event.arguments.get("object")) == query_pair
self.assertAlmostEqual(exact_present_probability(system, {}, ("A", "B")), 0.6) ]
self.assertAlmostEqual(exact_present_probability(system, {}, ("B", "A")), 0.6) self.assertFalse(direct)
event_ids = {event.id for event in skeleton.events}
self.assertTrue(set(skeleton.proof_event_ids) <= event_ids)
self.assertGreaterEqual(len(skeleton.proof_event_ids), skeleton.complexity)
def test_exact_enumerator_respects_stateful_event_order(self): def test_world_contains_multiple_logic_domains(self):
system = HiddenSystem( skeleton = generate_skeletons(1, "short", seed=3)[0]
initial_record="ABSENT", predicates = {
conflict_policy="PERSISTENT_INVALIDATION_VETO", event.arguments.get("predicate")
rules=( for event in skeleton.events
HiddenRule("mark", "A", (), 0.5, "SET_FLAG", "ready"), if event.kind == "RELATE"
HiddenRule( }
"add", self.assertTrue({"ABOVE", "MEMBER", "SUBSET", "CONNECTED"} <= predicates)
"B", kinds = {event.kind for event in skeleton.events}
(Condition("state", "ready", "IS_SET", True),), self.assertTrue({"MOVE", "UPDATE_NUMBER", "SET_STATE"} <= kinds)
0.8,
"ADD_RECORD",
"record",
),
),
)
self.assertAlmostEqual(exact_present_probability(system, {}, ("A", "B")), 0.4)
self.assertEqual(exact_present_probability(system, {}, ("B", "A")), 0.0)
def test_length_validation(self): def test_length_and_scale_validation(self):
self.assertEqual(resolve_length("medium"), 6) self.assertEqual(resolve_length("medium"), 6)
self.assertEqual(resolve_length("5"), 5) self.assertEqual(resolve_length("5"), 5)
with self.assertRaises(ValueError): with self.assertRaises(ValueError):
resolve_length("huge") resolve_length("huge")
with self.assertRaises(ValueError): with self.assertRaises(ValueError):
resolve_length(3) resolve_length(2)
with self.assertRaises(ValueError):
generate_skeletons(1, "short", entity_count=20)
if __name__ == "__main__": if __name__ == "__main__":