Generate proof-bearing dynamic world histories
This commit is contained in:
@@ -1,4 +1,4 @@
|
||||
name: Short behavioral-system evaluation
|
||||
name: Dynamic world reasoning evaluation
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
@@ -6,7 +6,7 @@ on:
|
||||
jobs:
|
||||
short-eval:
|
||||
runs-on: minecraft-and-affine-server
|
||||
timeout-minutes: 15
|
||||
timeout-minutes: 45
|
||||
env:
|
||||
RENDERER_API_KEY: ${{ secrets.RENDERER_API_KEY }}
|
||||
RENDERER_BASE_URL: ${{ vars.RENDERER_BASE_URL }}
|
||||
@@ -24,10 +24,10 @@ jobs:
|
||||
- name: Run unit tests
|
||||
run: python3 -m unittest discover -s tests -v
|
||||
|
||||
- name: Run five short live cases
|
||||
- name: Run five large-world live cases
|
||||
run: python3 -m logicbench run tasks/short-smoke.json
|
||||
|
||||
- name: Show hidden systems, behavior skeletons, results, and summary
|
||||
- name: Show dynamic worlds, rendered histories, results, and summary
|
||||
if: always()
|
||||
run: |
|
||||
test -f artifacts/short-smoke/skeletons.jsonl && cat artifacts/short-smoke/skeletons.jsonl || true
|
||||
@@ -40,5 +40,5 @@ jobs:
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: short-abstraction-evaluation
|
||||
name: dynamic-world-evaluation
|
||||
path: artifacts/short-smoke/
|
||||
|
||||
@@ -1,78 +1,104 @@
|
||||
# Logic Skeleton Bench
|
||||
|
||||
Logic Skeleton Bench tests whether a model can infer an abstract mechanism from behavior rather than solve a rule statement with renamed nouns.
|
||||
Logic Skeleton Bench generates large, evolving fictional worlds and tests whether a model can recover a logical consequence from their natural-language histories.
|
||||
|
||||
The generator now creates an executable **hidden stochastic system**, runs controlled experiments against it, and builds a **behavior skeleton** containing only inputs, ordered events, and observed final outcomes. A renderer turns that evidence into a coherent natural-language world. The solver sees the rendered histories and predicts which outcome is more likely for a held-out event combination. It never sees the hidden rules or their probabilities.
|
||||
The benchmark has five stages:
|
||||
|
||||
The renderer and solver remain independent clients. Each can use a different model, endpoint, key, and OpenAI-compatible API style for future model-pair heatmaps.
|
||||
```text
|
||||
hidden typed program
|
||||
→ initial world with many entities
|
||||
→ seeded stochastic evolution
|
||||
→ deterministic realized event history and gold proof
|
||||
→ natural-language rendering and solving
|
||||
```
|
||||
|
||||
## Current task model
|
||||
Probability is used only to decide which candidate events actually occur. Once a seed has produced a history, the question and answer are deterministic. The solver never estimates a probability and never sees repeated-trial frequency tables.
|
||||
|
||||
Every trial starts from the same empty state. Hidden rules can:
|
||||
## Current world model
|
||||
|
||||
- trigger on an event;
|
||||
- test an input attribute or an internal state flag;
|
||||
- fire with a hidden probability;
|
||||
- add a record, permanently invalidate it, or set a flag used by a later event.
|
||||
The initial implementation mixes several logic domains in every world:
|
||||
|
||||
An invalidation is a persistent veto, so probabilistically triggered actions can conflict. Historical cases report aggregate `PRESENT`/`ABSENT` counts from repeated trials. The query uses a new composition of events and asks only which outcome is more likely—not for the exact probability.
|
||||
- directed spatial order (`ABOVE`/`BELOW`);
|
||||
- membership, subset inheritance, and disjointness;
|
||||
- locations and movement;
|
||||
- symmetric connectivity;
|
||||
- sequential numeric updates;
|
||||
- Boolean state changes.
|
||||
|
||||
The local evaluator computes the true query probability by exact enumeration of all stochastic branches. Random simulation is used only to create the finite observations shown to the model. Both are reproducible from the task seed.
|
||||
The hidden program contains typed relations, operators, quantifiers, inference laws, probabilistic transition rules, and their preconditions. A default short world contains 128 entities and runs for 1000 simulation steps, producing roughly 400 realized events with the checked-in seed.
|
||||
|
||||
The complete hidden-system language, execution semantics, public behavior format, and non-leakage boundary are specified in [the behavioral IR specification](docs/skeleton-language.md).
|
||||
After evolution, the generator searches the realized state for a query with a shortest proof at least as long as the configured `length`. It rejects directly stated answers and stores the exact proof event IDs internally. The renderer receives all public initial observations and all realized events, but not the program, probabilities, hidden ranks, proof, generator checks, or gold answer.
|
||||
|
||||
`length` is structural complexity, not a natural-language token target. It is the number of distinct ordered query events and hidden event rules. Presets are `short=4`, `medium=6`, and `long=8`; explicit integers from 4 through 10 are accepted.
|
||||
The full grammar and execution contract are defined in [the dynamic-world IR specification](docs/skeleton-language.md).
|
||||
|
||||
## Complexity controls
|
||||
|
||||
`length` controls minimum proof length, not entity count, event count, or natural-language tokens:
|
||||
|
||||
| Preset | Minimum proof events |
|
||||
| --- | ---: |
|
||||
| `short` | 4 |
|
||||
| `medium` | 6 |
|
||||
| `long` | 8 |
|
||||
|
||||
`entity_count` and `simulation_steps` are separate task fields. Explicit proof lengths from 3 through 10 are accepted.
|
||||
|
||||
## Local use
|
||||
|
||||
Python 3.11 or newer is sufficient; the package has no runtime dependencies.
|
||||
Python 3.11 or newer is sufficient; there are no runtime dependencies.
|
||||
|
||||
Generate three short, answer-bearing records without calling an LLM:
|
||||
Generate an answer-bearing world without calling an LLM:
|
||||
|
||||
```bash
|
||||
python3 -m logicbench generate --count 3 --length short --seed 20260922
|
||||
python3 -m logicbench generate \
|
||||
--count 1 \
|
||||
--length short \
|
||||
--entity-count 128 \
|
||||
--simulation-steps 1000 \
|
||||
--seed 20260922
|
||||
```
|
||||
|
||||
Run the checked-in live task:
|
||||
|
||||
```bash
|
||||
export RENDERER_API_KEY='...'
|
||||
export RENDERER_BASE_URL='your-renderer-compatible-base-url'
|
||||
export RENDERER_MODEL='your-renderer-model'
|
||||
export RENDERER_BASE_URL='...'
|
||||
export RENDERER_MODEL='...'
|
||||
export SOLVER_API_KEY='...'
|
||||
export SOLVER_BASE_URL='your-solver-compatible-base-url'
|
||||
export SOLVER_MODEL='your-solver-model'
|
||||
export SOLVER_BASE_URL='...'
|
||||
export SOLVER_MODEL='...'
|
||||
python3 -m logicbench run tasks/short-smoke.json
|
||||
```
|
||||
|
||||
| Variable | Meaning |
|
||||
| --- | --- |
|
||||
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials; values may be identical |
|
||||
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials |
|
||||
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Required independent OpenAI-compatible endpoints |
|
||||
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` (default) or `chat_completions` |
|
||||
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required, independently selected model IDs |
|
||||
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` or `chat_completions` |
|
||||
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required independent model IDs |
|
||||
| `RENDERER_TIMEOUT_SECONDS`, `SOLVER_TIMEOUT_SECONDS` | Optional per-request timeout; default `240` |
|
||||
| `RENDERER_ENABLE_THINKING`, `SOLVER_ENABLE_THINKING` | Optional `true`/`false` provider extension; omitted unless set |
|
||||
| `RENDERER_ENABLE_THINKING`, `SOLVER_ENABLE_THINKING` | Optional provider extension; omitted unless set |
|
||||
|
||||
Outputs under the task's `output_dir` are:
|
||||
The renderer and solver remain independent clients for model-pair heatmaps. The checked-in Qwen workflow explicitly disables solver thinking to keep non-streaming test latency bounded.
|
||||
|
||||
- `skeletons.jsonl`: full hidden programs, sampled behavior, exact probabilities, gold labels, and latent traces; written before any LLM call;
|
||||
- `rendered.jsonl`: each natural-language problem, checkpointed immediately after rendering;
|
||||
- `solver-outputs.jsonl`: each complete raw solver response, checkpointed before parsing;
|
||||
- `results.jsonl`: rendered problems, complete solver responses, parsed labels, parse errors, format compliance, gold probabilities, and scores;
|
||||
- `summary.json`: aggregate accuracy and structural metadata.
|
||||
## Artifacts
|
||||
|
||||
## Gitea Actions one-shot test
|
||||
Each run writes:
|
||||
|
||||
`.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when manually started. It targets the `minecraft-and-affine-server` runner label, runs unit tests and five short live cases, prints the auditable records, and uploads `short-abstraction-evaluation`.
|
||||
- `skeletons.jsonl`: hidden program, entities, hidden initial and final states, complete event trace, query, gold answer, proof, and anti-leakage checks;
|
||||
- `rendered.jsonl`: natural-language worlds, checkpointed immediately after rendering;
|
||||
- `solver-outputs.jsonl`: complete raw solver responses, checkpointed before parsing;
|
||||
- `results.jsonl`: parsed labels, parse errors, format compliance, and correctness;
|
||||
- `summary.json`: aggregate accuracy, parsing metrics, scale, and per-world event counts.
|
||||
|
||||
Configure repository Actions with:
|
||||
## Gitea Actions
|
||||
|
||||
1. Secrets `RENDERER_API_KEY` and `SOLVER_API_KEY`.
|
||||
2. Variables `RENDERER_BASE_URL`, `RENDERER_MODEL`, and `RENDERER_API_STYLE`.
|
||||
3. Variables `SOLVER_BASE_URL`, `SOLVER_MODEL`, and `SOLVER_API_STYLE`.
|
||||
`.gitea/workflows/short-eval.yml` is manual-only (`workflow_dispatch`) and targets `minecraft-and-affine-server`. It runs unit tests followed by five large-world cases. Because rendering hundreds of events is substantially slower than the former toy benchmark, the job timeout is 45 minutes.
|
||||
|
||||
For Qwen's OpenAI-compatible API, use the compatible-mode `/v1` endpoint and `chat_completions`. Keys, URLs, and models remain role-specific even when both roles currently use the same provider. The checked-in short workflow explicitly disables solver thinking to avoid long non-streaming requests; remove or change that workflow value when thinking-mode performance is itself under test.
|
||||
Repository Actions settings remain two secrets and six role-specific variables:
|
||||
|
||||
- secrets: `RENDERER_API_KEY`, `SOLVER_API_KEY`;
|
||||
- variables: each role's `BASE_URL`, `MODEL`, and `API_STYLE`.
|
||||
|
||||
## Tests
|
||||
|
||||
|
||||
+193
-154
@@ -1,207 +1,246 @@
|
||||
# Hidden System and Behavioral Skeleton Language
|
||||
# Dynamic World Generator and Intermediate Language
|
||||
|
||||
This document specifies version 0.2 of the benchmark. The central change is that natural-language rendering no longer receives a system's rules. It receives only behavior produced by executing those rules.
|
||||
This specification defines the answer-bearing world representation generated before natural-language rendering.
|
||||
|
||||
## Three-layer contract
|
||||
## Pipeline boundary
|
||||
|
||||
```text
|
||||
HiddenSystem --execute/sample--> BehaviorSkeleton --LLM render--> Natural-language problem
|
||||
| | |
|
||||
never public observations only solver input
|
||||
|
|
||||
exact enumeration --> gold probability --> majority label --> evaluator only
|
||||
HiddenProgram + seed
|
||||
↓
|
||||
Initial WorldState
|
||||
↓ stochastic transition sampling
|
||||
Realized Event Trace
|
||||
↓ deterministic closure and query selection
|
||||
Query + Gold Proof
|
||||
↓ redact program, hidden state, proof, and answer
|
||||
Renderer Payload
|
||||
↓
|
||||
Natural-language history → Solver
|
||||
```
|
||||
|
||||
Generation follows this order:
|
||||
Randomness affects which events occur. It never makes the final question probabilistic: after the trace is realized, the evaluator asks whether a proposition follows in that particular world.
|
||||
|
||||
1. A seeded local generator builds an executable `HiddenSystem`.
|
||||
2. The simulator runs controlled support cases repeatedly and records aggregate outcomes.
|
||||
3. A held-out query combines known events in a combination not present in the histories.
|
||||
4. An exact local enumerator computes the query's true outcome distribution and gold label.
|
||||
5. The full answer-bearing record is written before either LLM is called.
|
||||
6. `renderer_payload()` removes the program, exact probability, gold label, and latent trace.
|
||||
7. The renderer expresses only the public behavioral evidence in a surface domain.
|
||||
8. The solver infers a mechanism from those observations and selects the more likely outcome.
|
||||
|
||||
This is system identification followed by transfer, not direct deduction from stated laws.
|
||||
|
||||
## Hidden system grammar
|
||||
## Hidden program
|
||||
|
||||
```text
|
||||
HiddenSystem ::= {
|
||||
initial_record,
|
||||
conflict_policy,
|
||||
rules: HiddenRule+
|
||||
HiddenProgram ::= {
|
||||
types: TypeName+,
|
||||
relations: PredicateName+,
|
||||
operators: OperatorName+,
|
||||
quantifiers: QuantifierName+,
|
||||
structures: StructureName+,
|
||||
inference_rules: RuleExpression+,
|
||||
transition_rules: TransitionRule+
|
||||
}
|
||||
|
||||
HiddenRule ::= {
|
||||
TransitionRule ::= {
|
||||
id,
|
||||
on_event,
|
||||
conditions: Condition*,
|
||||
probability,
|
||||
action,
|
||||
target
|
||||
trigger,
|
||||
preconditions: Expression*,
|
||||
probability: Number[0,1],
|
||||
effects: Expression+
|
||||
}
|
||||
```
|
||||
|
||||
Condition ::= {
|
||||
source: "attribute" | "state",
|
||||
key,
|
||||
operator: "EQ" | "LE" | "GT" | "IS_SET",
|
||||
value
|
||||
The current finite vocabulary includes:
|
||||
|
||||
```text
|
||||
types:
|
||||
Entity, Set, Region, Number, Boolean, State
|
||||
|
||||
relations:
|
||||
EQUAL, NOT_EQUAL, ABOVE, BELOW,
|
||||
MEMBER, SUBSET, DISJOINT,
|
||||
ADJACENT, CONNECTED, CAUSES, LOCATED_IN
|
||||
|
||||
operators:
|
||||
NOT, AND, OR, IMPLIES,
|
||||
ADD, SUBTRACT, COMPARE,
|
||||
UNION, INTERSECTION,
|
||||
MOVE, SWAP, SET, COMPOSE
|
||||
|
||||
quantifiers:
|
||||
ALL, EXISTS
|
||||
|
||||
structures:
|
||||
CHAIN, BRANCH, MERGE, CYCLE, NEST, CONDITION, SEQUENCE
|
||||
```
|
||||
|
||||
Not every primitive must affect every query. The point of the shared world is that many logic domains evolve concurrently and become realistic distractors for one proof-bearing target.
|
||||
|
||||
## Entities and initial state
|
||||
|
||||
```text
|
||||
Entity ::= { id, type }
|
||||
|
||||
WorldState ::= {
|
||||
locations,
|
||||
directed_order_relations,
|
||||
memberships,
|
||||
subset_relations,
|
||||
disjoint_relations,
|
||||
connections,
|
||||
numeric_values,
|
||||
boolean_flags,
|
||||
hidden_generation_attributes
|
||||
}
|
||||
|
||||
action ::= "ADD_RECORD" | "INVALIDATE_RECORD" | "SET_FLAG"
|
||||
probability ::= real number in [0, 1]
|
||||
```
|
||||
|
||||
The grammar is finite, while different rule selections, parameters, event sequences, attributes, and stochastic outcomes generate an unbounded set of behavioral structures.
|
||||
The default 128-entity allocation includes ordinary entities, sets, regions, and numeric counters. Entity IDs are abstract symbols; the renderer must assign stable names within its chosen scenario.
|
||||
|
||||
### State and execution
|
||||
Some initial facts are public, such as locations and counter values. Each has an `I####` audit ID. Latent generation attributes, such as levels used to keep randomly generated order and subset graphs consistent, are stored for audit but removed from the renderer payload.
|
||||
|
||||
Each independent trial starts with:
|
||||
## Event language
|
||||
|
||||
```text
|
||||
record = ABSENT
|
||||
invalidated = false
|
||||
flags = {}
|
||||
WorldEvent ::= {
|
||||
id,
|
||||
time,
|
||||
kind,
|
||||
participants,
|
||||
arguments
|
||||
}
|
||||
```
|
||||
|
||||
Events execute from left to right. For each event, matching rules execute in stored order:
|
||||
|
||||
1. evaluate all conditions against immutable attributes and current state;
|
||||
2. if they hold, sample that rule's Bernoulli trigger;
|
||||
3. if triggered, apply its action immediately.
|
||||
|
||||
Actions mean:
|
||||
|
||||
- `SET_FLAG(x)`: add internal flag `x`; later rules can test it;
|
||||
- `ADD_RECORD`: make the record present unless it was previously invalidated;
|
||||
- `INVALIDATE_RECORD`: remove the record and permanently set `invalidated=true`.
|
||||
|
||||
The current `PERSISTENT_INVALIDATION_VETO` policy therefore implements a one-vote veto. A later add cannot restore an invalidated record. Event order matters because a flag can enable a later rule and an early veto persists.
|
||||
|
||||
Probabilities belong to hidden rule activation. They are not printed in the rendered problem. The task asks whether `PRESENT` or `ABSENT` has probability greater than one half, not for a numerical probability.
|
||||
|
||||
## Exact gold computation versus sampled evidence
|
||||
|
||||
These are deliberately separate:
|
||||
|
||||
- Support histories are finite empirical data. The seeded simulator samples each case 48 times for short tasks and 64 times for longer tasks.
|
||||
- The gold query probability is not estimated by Monte Carlo. The evaluator propagates a probability distribution over every reachable internal state, branching at every applicable stochastic rule.
|
||||
|
||||
For each state with mass `m` and applicable rule probability `p`, exact enumeration creates:
|
||||
Current realized event forms are:
|
||||
|
||||
```text
|
||||
inactive branch: same state, mass m(1-p)
|
||||
active branch: action(state), mass mp
|
||||
MOVE(entity, origin, destination)
|
||||
RELATE(ABOVE, higher, lower, region)
|
||||
RELATE(MEMBER, entity, set)
|
||||
RELATE(SUBSET, child_set, parent_set)
|
||||
RELATE(DISJOINT, left_set, right_set)
|
||||
RELATE(CONNECTED, left_entity, right_entity)
|
||||
UPDATE_NUMBER(counter, ADD|SUBTRACT, amount, before, after)
|
||||
SET_STATE(entity, flag, before, after)
|
||||
```
|
||||
|
||||
Equal resulting states are merged after each rule. At the end, the masses of all states whose record is present are summed. The answer is `PRESENT` when that sum exceeds `0.5`, otherwise `ABSENT`. Current generation rejects cases within `0.08` of the boundary to avoid fragile labels.
|
||||
Every simulator step selects a candidate transition type. Preconditions are evaluated, then the transition's probability determines whether an event actually happens. Only realized events enter the trace. A fixed seed makes the complete trace reproducible.
|
||||
|
||||
## Behavioral skeleton grammar
|
||||
The default short configuration runs 1000 candidate steps and currently realizes about 400 events. Thus probability controls world evolution, while the resulting event history is an ordinary deterministic collection of facts and changes.
|
||||
|
||||
## Current inference semantics
|
||||
|
||||
```text
|
||||
BehaviorSkeleton ::= {
|
||||
ABOVE(x,y) ∧ ABOVE(y,z) → ABOVE(x,z)
|
||||
ABOVE(x,y) → BELOW(y,x)
|
||||
|
||||
MEMBER(x,A) ∧ SUBSET(A,B) → MEMBER(x,B)
|
||||
MEMBER(x,A) ∧ DISJOINT(A,B) → ¬MEMBER(x,B)
|
||||
|
||||
CONNECTED(x,y) ↔ CONNECTED(y,x)
|
||||
|
||||
numeric updates execute in timestamp order
|
||||
later SET_STATE operations replace the prior value of the same flag
|
||||
MOVE replaces the current region of its participant
|
||||
```
|
||||
|
||||
`ABOVE` and `SUBSET` events are generated from adjacent hidden levels. This allows long chains while preventing contradictory cycles. The levels are generator machinery, not observable premises.
|
||||
|
||||
## Query generation
|
||||
|
||||
```text
|
||||
Query ::= {
|
||||
predicate,
|
||||
arguments,
|
||||
time
|
||||
}
|
||||
```
|
||||
|
||||
After simulation, the query builder searches the realized graphs rather than inserting a prefabricated puzzle. Current query families are:
|
||||
|
||||
- `spatial_order`: transitive `ABOVE` proof;
|
||||
- reverse `spatial_order`: a `NO` answer proved by an opposite-direction chain;
|
||||
- `membership_inheritance`: membership propagated through nested subsets;
|
||||
- `membership_disjointness`: inherited membership in one set plus disjointness from the queried set.
|
||||
|
||||
For a requested complexity `k`, the selected proof contains at least `k` realized event IDs. The queried proposition itself must not appear as a direct event. The full record stores both the event IDs and normalized proof facts.
|
||||
|
||||
Current generation checks record that:
|
||||
|
||||
- no event directly states the query;
|
||||
- proof length meets the requested minimum;
|
||||
- proof events are a small fraction of the full history;
|
||||
- simple direct-fact and too-short-proof baselines fail.
|
||||
|
||||
These checks are structural. They should be expanded with executable solver baselines as the benchmark grows.
|
||||
|
||||
## Full skeleton
|
||||
|
||||
```text
|
||||
WorldSkeleton ::= {
|
||||
id,
|
||||
family,
|
||||
complexity,
|
||||
event_vocabulary,
|
||||
attribute_vocabulary,
|
||||
observation_protocol,
|
||||
hidden_system,
|
||||
support_cases: BehaviorCase+,
|
||||
query_case: BehaviorCase,
|
||||
answer_options: ["PRESENT", "ABSENT"],
|
||||
gold_answer,
|
||||
gold_present_probability,
|
||||
latent_trace
|
||||
}
|
||||
|
||||
BehaviorCase ::= {
|
||||
id,
|
||||
attributes,
|
||||
entity_count,
|
||||
simulation_steps,
|
||||
hidden_program,
|
||||
entities,
|
||||
initial_state,
|
||||
events,
|
||||
trials,
|
||||
observed_present,
|
||||
observed_absent
|
||||
final_state,
|
||||
query,
|
||||
answer_options,
|
||||
gold_answer,
|
||||
proof_event_ids,
|
||||
proof_facts,
|
||||
generator_checks
|
||||
}
|
||||
```
|
||||
|
||||
Support cases have integer observation counts that sum to `trials`. The query has `null` observation counts because its result is what the solver must predict.
|
||||
`skeletons.jsonl` contains this complete record for reproducibility and auditing.
|
||||
|
||||
The public payload contains:
|
||||
## Renderer boundary
|
||||
|
||||
The renderer receives only:
|
||||
|
||||
```text
|
||||
family, complexity,
|
||||
event_vocabulary, attribute_vocabulary,
|
||||
observation_protocol, support_cases,
|
||||
query_case, answer_options
|
||||
family
|
||||
complexity
|
||||
entity_types
|
||||
initial_observations
|
||||
events
|
||||
query
|
||||
answer_options
|
||||
```
|
||||
|
||||
It never contains:
|
||||
It does not receive:
|
||||
|
||||
```text
|
||||
id, hidden_system, gold_answer,
|
||||
gold_present_probability, latent_trace
|
||||
world id
|
||||
hidden program or transition probabilities
|
||||
latent initial-state attributes
|
||||
complete final state
|
||||
gold answer
|
||||
proof event IDs or proof facts
|
||||
generator checks
|
||||
```
|
||||
|
||||
## Current generated structure
|
||||
The renderer must map every symbol to one stable surface name, preserve all events and their direction and ordering, and produce one chronological fictional history. Every rendered initial fact and event line retains its `I####` or `V#####` audit ID. The evaluator rejects a rendering when any expected ID is absent. The renderer may compress formatting, but it may not summarize away distractors, state hidden rules, expose the proof, or directly assert the queried proposition.
|
||||
|
||||
Every system has at least four interacting rules:
|
||||
## Complexity dimensions
|
||||
|
||||
- an unconditional probabilistic add event;
|
||||
- an attribute-gated probabilistic veto event;
|
||||
- an event that probabilistically sets an invisible flag;
|
||||
- a later event whose effect requires that flag.
|
||||
The axes are deliberately independent:
|
||||
|
||||
Longer configurations add attribute-gated add or veto rules. Support evidence includes:
|
||||
```text
|
||||
length minimum gold-proof events
|
||||
entity_count total typed entities
|
||||
simulation_steps candidate evolution steps
|
||||
event_count realized events; measured after simulation
|
||||
```
|
||||
|
||||
- every event in isolation;
|
||||
- pairwise event combinations;
|
||||
- both orders of the state-gated pair;
|
||||
- low/high attribute counterfactual controls;
|
||||
- repeated outcomes rather than rule annotations.
|
||||
Named proof presets are `short=4`, `medium=6`, and `long=8`. The task file separately controls world size and simulation duration.
|
||||
|
||||
The query applies all events in order. Its complete combination is held out from support evidence, so the model must compose effects learned from smaller interventions.
|
||||
## Planned extensions
|
||||
|
||||
Generated indices alternate between present-favored and absent-favored systems to prevent a one-label benchmark. The index is redacted from the renderer payload so this balancing mechanism cannot become a shortcut. It affects hidden parameter ranges, never the answer after generation; the exact enumerator remains authoritative.
|
||||
The IR is intended to grow without changing the visibility boundary. Useful next additions include:
|
||||
|
||||
## Complexity and determinism
|
||||
|
||||
`complexity` is the number of distinct events in the held-out sequence and the minimum number of corresponding hidden rules. It is independent of rendered token count.
|
||||
|
||||
| Preset | Ordered query events |
|
||||
| --- | ---: |
|
||||
| `short` | 4 |
|
||||
| `medium` | 6 |
|
||||
| `long` | 8 |
|
||||
|
||||
Explicit values from 4 through 10 are accepted. `generate_skeletons(count, length, seed)` uses a local pseudorandom generator; the same code version and arguments produce equivalent hidden systems, samples, queries, and gold answers.
|
||||
|
||||
## Renderer obligations
|
||||
|
||||
The renderer may rename event codes, attributes, states, and outcomes into a coherent scenario. It must preserve every numeric count, attribute value, event order, trial boundary, query input, and answer label.
|
||||
|
||||
It must not:
|
||||
|
||||
- receive or state hidden rules;
|
||||
- turn empirical correlations into declared laws;
|
||||
- expose trigger probabilities, internal flags, latent traces, or the answer;
|
||||
- collapse a repeated experiment into a deterministic statement;
|
||||
- add causal or real-world assumptions;
|
||||
- show the exact query combination as an already observed history.
|
||||
|
||||
Retaining the complete pre-render record makes rendering fidelity independently auditable.
|
||||
|
||||
## Extension points
|
||||
|
||||
The next DSL extensions can remain compatible with the three-layer boundary:
|
||||
|
||||
- multiple entities and relations between them;
|
||||
- delayed actions, expiry, and multiple clock phases;
|
||||
- priority, last-write, merge, and fallback conflict policies;
|
||||
- partial observation and noisy sensors distinct from stochastic behavior;
|
||||
- candidate-program enumeration to report identifiability;
|
||||
- counterfactual and intervention queries in addition to outcome prediction.
|
||||
|
||||
Any extension must keep gold computation local, expose behavior rather than source rules, preserve seeded reproducibility, and add tests for execution order, conflict semantics, and renderer redaction.
|
||||
- relation deletion and expiry;
|
||||
- explicit event causation chains;
|
||||
- multi-effect transactions and conflict policies;
|
||||
- union and intersection queries;
|
||||
- quantified `ALL` and `EXISTS` questions;
|
||||
- temporal queries about intermediate rather than final state;
|
||||
- counterfactual interventions;
|
||||
- executable nearest-neighbor, keyword, local-window, and shallow-proof baselines;
|
||||
- renderer fidelity verification against every source event.
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
"""Behavioral hidden-system benchmark."""
|
||||
|
||||
from .models import BehaviorCase, BehaviorSkeleton, Condition, HiddenRule, HiddenSystem
|
||||
from .models import Entity, HiddenProgram, Query, TransitionRule, WorldEvent, WorldSkeleton
|
||||
from .skeletons import generate_skeletons, resolve_length
|
||||
|
||||
__all__ = [
|
||||
"BehaviorCase",
|
||||
"BehaviorSkeleton",
|
||||
"Condition",
|
||||
"HiddenRule",
|
||||
"HiddenSystem",
|
||||
"Entity",
|
||||
"HiddenProgram",
|
||||
"Query",
|
||||
"TransitionRule",
|
||||
"WorldEvent",
|
||||
"WorldSkeleton",
|
||||
"generate_skeletons",
|
||||
"resolve_length",
|
||||
]
|
||||
|
||||
+11
-1
@@ -29,6 +29,8 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
generate.add_argument("--count", type=int, default=3)
|
||||
generate.add_argument("--length", default="short")
|
||||
generate.add_argument("--seed", type=int, default=0)
|
||||
generate.add_argument("--entity-count", type=int, default=128)
|
||||
generate.add_argument("--simulation-steps", type=int, default=1000)
|
||||
generate.add_argument("--output", type=Path)
|
||||
|
||||
run = subparsers.add_parser("run", help="run renderer and solver from a task JSON file")
|
||||
@@ -39,7 +41,13 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
if args.command == "generate":
|
||||
skeletons = generate_skeletons(args.count, args.length, args.seed)
|
||||
skeletons = generate_skeletons(
|
||||
args.count,
|
||||
args.length,
|
||||
args.seed,
|
||||
entity_count=args.entity_count,
|
||||
simulation_steps=args.simulation_steps,
|
||||
)
|
||||
lines = [json.dumps(item.to_dict(), ensure_ascii=False) for item in skeletons]
|
||||
text = "\n".join(lines) + "\n"
|
||||
if args.output:
|
||||
@@ -59,6 +67,8 @@ def main(argv: list[str] | None = None) -> int:
|
||||
length=task["length"],
|
||||
seed=int(task["seed"]),
|
||||
output_dir=Path(task["output_dir"]),
|
||||
entity_count=int(task.get("entity_count", 128)),
|
||||
simulation_steps=int(task.get("simulation_steps", 1000)),
|
||||
)
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
|
||||
@@ -81,7 +81,7 @@ class LLMClient:
|
||||
"schema": schema,
|
||||
}
|
||||
},
|
||||
"max_output_tokens": 2500,
|
||||
"max_output_tokens": 12000,
|
||||
}
|
||||
else:
|
||||
path = "/chat/completions"
|
||||
@@ -95,7 +95,7 @@ class LLMClient:
|
||||
"type": "json_schema",
|
||||
"json_schema": {"name": schema_name, "strict": True, "schema": schema},
|
||||
},
|
||||
"max_tokens": 2500,
|
||||
"max_tokens": 12000,
|
||||
}
|
||||
|
||||
if self.config.enable_thinking is not None:
|
||||
|
||||
+43
-44
@@ -5,78 +5,77 @@ from typing import Any
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Condition:
|
||||
"""A predicate over immutable inputs or mutable simulator state."""
|
||||
|
||||
source: str
|
||||
key: str
|
||||
operator: str
|
||||
value: Any
|
||||
class Entity:
|
||||
id: str
|
||||
type: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HiddenRule:
|
||||
"""One executable rule in the hidden system language."""
|
||||
|
||||
class TransitionRule:
|
||||
id: str
|
||||
on_event: str
|
||||
conditions: tuple[Condition, ...]
|
||||
trigger: str
|
||||
preconditions: tuple[str, ...]
|
||||
probability: float
|
||||
action: str
|
||||
target: str
|
||||
effects: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HiddenSystem:
|
||||
"""The latent program. This object must never be sent to the renderer."""
|
||||
|
||||
initial_record: str
|
||||
conflict_policy: str
|
||||
rules: tuple[HiddenRule, ...]
|
||||
class HiddenProgram:
|
||||
types: tuple[str, ...]
|
||||
relations: tuple[str, ...]
|
||||
operators: tuple[str, ...]
|
||||
quantifiers: tuple[str, ...]
|
||||
structures: tuple[str, ...]
|
||||
inference_rules: tuple[str, ...]
|
||||
transition_rules: tuple[TransitionRule, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BehaviorCase:
|
||||
"""Observable inputs and aggregate output from repeated independent trials."""
|
||||
|
||||
class WorldEvent:
|
||||
id: str
|
||||
attributes: dict[str, Any]
|
||||
events: tuple[str, ...]
|
||||
trials: int
|
||||
observed_present: int | None
|
||||
observed_absent: int | None
|
||||
time: int
|
||||
kind: str
|
||||
participants: tuple[str, ...]
|
||||
arguments: dict[str, Any]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BehaviorSkeleton:
|
||||
"""A hidden program plus its public behavioral evidence and answer-bearing query."""
|
||||
class Query:
|
||||
predicate: str
|
||||
arguments: tuple[str, ...]
|
||||
time: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WorldSkeleton:
|
||||
id: str
|
||||
family: str
|
||||
complexity: int
|
||||
event_vocabulary: tuple[str, ...]
|
||||
attribute_vocabulary: dict[str, str]
|
||||
observation_protocol: tuple[str, ...]
|
||||
hidden_system: HiddenSystem
|
||||
support_cases: tuple[BehaviorCase, ...]
|
||||
query_case: BehaviorCase
|
||||
entity_count: int
|
||||
simulation_steps: int
|
||||
hidden_program: HiddenProgram
|
||||
entities: tuple[Entity, ...]
|
||||
initial_state: dict[str, Any]
|
||||
events: tuple[WorldEvent, ...]
|
||||
final_state: dict[str, Any]
|
||||
query: Query
|
||||
answer_options: tuple[str, ...]
|
||||
gold_answer: str
|
||||
gold_present_probability: float
|
||||
latent_trace: tuple[str, ...]
|
||||
proof_event_ids: tuple[str, ...]
|
||||
proof_facts: tuple[str, ...]
|
||||
generator_checks: dict[str, Any]
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return asdict(self)
|
||||
|
||||
def renderer_payload(self) -> dict[str, Any]:
|
||||
"""Expose behavior only: no program, probability, trace, or answer."""
|
||||
"""Expose realized history, never the source program, proof, or answer."""
|
||||
return {
|
||||
"family": self.family,
|
||||
"complexity": self.complexity,
|
||||
"event_vocabulary": self.event_vocabulary,
|
||||
"attribute_vocabulary": self.attribute_vocabulary,
|
||||
"observation_protocol": self.observation_protocol,
|
||||
"support_cases": [asdict(case) for case in self.support_cases],
|
||||
"query_case": asdict(self.query_case),
|
||||
"entity_types": {entity.id: entity.type for entity in self.entities},
|
||||
"initial_observations": self.initial_state["public_observations"],
|
||||
"events": [asdict(event) for event in self.events],
|
||||
"query": asdict(self.query),
|
||||
"answer_options": self.answer_options,
|
||||
}
|
||||
|
||||
+38
-20
@@ -6,7 +6,7 @@ from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
from .models import BehaviorSkeleton
|
||||
from .models import WorldSkeleton
|
||||
from .skeletons import generate_skeletons
|
||||
|
||||
|
||||
@@ -31,7 +31,6 @@ class EvaluationResult:
|
||||
rendered_problem: str
|
||||
answer_options: tuple[str, ...]
|
||||
gold_answer: str
|
||||
gold_present_probability: float
|
||||
solver_output: str
|
||||
model_answer: str | None
|
||||
format_compliant: bool
|
||||
@@ -47,20 +46,20 @@ RENDER_SCHEMA = {
|
||||
}
|
||||
|
||||
|
||||
def render_skeleton(client: JsonModel, skeleton: BehaviorSkeleton) -> str:
|
||||
def render_skeleton(client: JsonModel, skeleton: WorldSkeleton) -> str:
|
||||
prompt = json.dumps(skeleton.renderer_payload(), ensure_ascii=False, indent=2)
|
||||
response = client.request_json(
|
||||
system=(
|
||||
"You turn behavioral observations from an unknown stochastic system into one self-contained "
|
||||
"natural-language inductive reasoning problem. Invent a coherent concrete or fictional domain. "
|
||||
"Give each event code and attribute a stable surface name, and preserve every case's attributes, "
|
||||
"event order, trial count, observed outcome counts, query, and exact answer labels. The histories "
|
||||
"are empirical observations, not explicit system laws: do not state or invent the hidden rules, "
|
||||
"trigger probabilities, intermediate flags, exact query probability, or answer. Make clear that "
|
||||
"all trials use one unchanged system and that the question asks which final outcome is more likely, "
|
||||
"not for an exact probability. Present repeated histories in a compact table rather than verbose "
|
||||
"paragraphs. State the supplied answer labels, but do not impose a response format or forbid an "
|
||||
"explanation; the evaluator supplies its own output protocol to the solver."
|
||||
"Turn a large formal history from one simulated world into a self-contained natural-language "
|
||||
"reasoning problem. Invent one coherent concrete or fictional setting and assign stable names to "
|
||||
"every symbol. Render every initial observation and every realized event exactly once, preserving "
|
||||
"time order, participants, relation direction, numeric values, and state changes. Render the query "
|
||||
"at the end. The events are facts that actually occurred, not repeated probability experiments. "
|
||||
"Never mention the generator, hidden program, probabilities, proof path, gold answer, or authoring "
|
||||
"instructions. Do not directly state the queried proposition. Begin each rendered fact or event line "
|
||||
"with its original I#### or V##### audit ID. Use compact chronological log lines rather than "
|
||||
"explanatory prose, but do not omit apparent distractors. State the answer labels; the "
|
||||
"evaluator separately supplies the solver response protocol."
|
||||
),
|
||||
user=f"Render this intermediate representation:\n{prompt}",
|
||||
schema_name="rendered_problem",
|
||||
@@ -69,7 +68,16 @@ def render_skeleton(client: JsonModel, skeleton: BehaviorSkeleton) -> str:
|
||||
problem = response.get("problem")
|
||||
if not isinstance(problem, str) or not problem.strip():
|
||||
raise RuntimeError("renderer returned an empty problem")
|
||||
return problem.strip()
|
||||
problem = problem.strip()
|
||||
expected_ids = [item["id"] for item in skeleton.initial_state["public_observations"]]
|
||||
expected_ids.extend(event.id for event in skeleton.events)
|
||||
missing = [identifier for identifier in expected_ids if identifier not in problem]
|
||||
if missing:
|
||||
preview = ", ".join(missing[:10])
|
||||
raise RuntimeError(
|
||||
f"renderer omitted {len(missing)} initial observations or events; first missing IDs: {preview}"
|
||||
)
|
||||
return problem
|
||||
|
||||
|
||||
def parse_marked_answer(output: str, answer_options: tuple[str, ...]) -> str:
|
||||
@@ -110,10 +118,10 @@ def solve_problem(client: JsonModel, problem: str, answer_options: tuple[str, ..
|
||||
def request_solver_output(client: JsonModel, problem: str, answer_options: tuple[str, ...]) -> str:
|
||||
return client.request_text(
|
||||
system=(
|
||||
"Infer the stable hidden mechanism from the repeated behavioral observations, then predict which "
|
||||
"query outcome is more likely. Event order, attributes, stochastic frequencies, interactions, and "
|
||||
"counterfactual controls may matter. Do not demand an exact probability. Keep the explanation under "
|
||||
"200 words, then put "
|
||||
"Solve the question from the described world history. Track entity identity, relation direction, "
|
||||
"temporal updates, set inheritance, ordering, and relevant compositions while ignoring distractors. "
|
||||
"Use only the stated observations and ordinary meanings explicitly established by the story. Keep "
|
||||
"the explanation under 200 words, then put "
|
||||
"the final answer on the last non-empty line in the exact form '#### <answer>', where <answer> "
|
||||
"is exactly one permitted answer label. Do not use unstated real-world assumptions."
|
||||
),
|
||||
@@ -135,9 +143,17 @@ def run_evaluation(
|
||||
length: str | int,
|
||||
seed: int,
|
||||
output_dir: Path,
|
||||
entity_count: int = 128,
|
||||
simulation_steps: int = 1000,
|
||||
) -> dict[str, Any]:
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
skeletons = generate_skeletons(count=count, length=length, seed=seed)
|
||||
skeletons = generate_skeletons(
|
||||
count=count,
|
||||
length=length,
|
||||
seed=seed,
|
||||
entity_count=entity_count,
|
||||
simulation_steps=simulation_steps,
|
||||
)
|
||||
_write_jsonl(output_dir / "skeletons.jsonl", [item.to_dict() for item in skeletons])
|
||||
|
||||
results: list[EvaluationResult] = []
|
||||
@@ -165,7 +181,6 @@ def run_evaluation(
|
||||
rendered_problem=problem,
|
||||
answer_options=skeleton.answer_options,
|
||||
gold_answer=skeleton.gold_answer,
|
||||
gold_present_probability=skeleton.gold_present_probability,
|
||||
solver_output=solver_output,
|
||||
model_answer=answer,
|
||||
format_compliant=format_compliant,
|
||||
@@ -189,6 +204,9 @@ def run_evaluation(
|
||||
"length": length,
|
||||
"complexity": skeletons[0].complexity,
|
||||
"seed": seed,
|
||||
"entity_count": entity_count,
|
||||
"simulation_steps": simulation_steps,
|
||||
"event_counts": [len(item.events) for item in skeletons],
|
||||
}
|
||||
(output_dir / "summary.json").write_text(
|
||||
json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
|
||||
|
||||
+579
-216
@@ -1,26 +1,33 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
from dataclasses import dataclass
|
||||
from itertools import combinations
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from .models import BehaviorCase, BehaviorSkeleton, Condition, HiddenRule, HiddenSystem
|
||||
from .models import Entity, HiddenProgram, Query, TransitionRule, WorldEvent, WorldSkeleton
|
||||
|
||||
|
||||
LENGTH_PRESETS = {"short": 4, "medium": 6, "long": 8}
|
||||
ANSWER_OPTIONS = ("PRESENT", "ABSENT")
|
||||
ANSWER_OPTIONS = ("YES", "NO")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _State:
|
||||
present: bool = False
|
||||
invalidated: bool = False
|
||||
flags: frozenset[str] = frozenset()
|
||||
@dataclass
|
||||
class _World:
|
||||
locations: dict[str, str]
|
||||
order_level: dict[str, int]
|
||||
set_level: dict[str, int]
|
||||
above: dict[tuple[str, str], str] = field(default_factory=dict)
|
||||
subsets: dict[tuple[str, str], str] = field(default_factory=dict)
|
||||
members: dict[tuple[str, str], str] = field(default_factory=dict)
|
||||
disjoint: dict[frozenset[str], str] = field(default_factory=dict)
|
||||
connected: set[frozenset[str]] = field(default_factory=set)
|
||||
numbers: dict[str, int] = field(default_factory=dict)
|
||||
flags: dict[tuple[str, str], bool] = field(default_factory=dict)
|
||||
|
||||
|
||||
def resolve_length(value: str | int) -> int:
|
||||
"""Resolve the number of distinct events in the held-out query."""
|
||||
"""Resolve required proof length, independently of event-log or text length."""
|
||||
if isinstance(value, int):
|
||||
result = value
|
||||
elif value in LENGTH_PRESETS:
|
||||
@@ -29,246 +36,602 @@ def resolve_length(value: str | int) -> int:
|
||||
try:
|
||||
result = int(value)
|
||||
except ValueError as exc:
|
||||
choices = ", ".join(LENGTH_PRESETS)
|
||||
raise ValueError(f"length must be an integer or one of: {choices}") from exc
|
||||
if not 4 <= result <= 10:
|
||||
raise ValueError("length must be between 4 and 10")
|
||||
raise ValueError(f"length must be an integer or one of: {', '.join(LENGTH_PRESETS)}") from exc
|
||||
if not 3 <= result <= 10:
|
||||
raise ValueError("length must be between 3 and 10")
|
||||
return result
|
||||
|
||||
|
||||
def _condition_holds(condition: Condition, attributes: dict[str, Any], state: _State) -> bool:
|
||||
if condition.source == "attribute":
|
||||
actual = attributes.get(condition.key)
|
||||
elif condition.source == "state":
|
||||
actual = condition.key in state.flags
|
||||
else:
|
||||
raise ValueError(f"unknown condition source: {condition.source}")
|
||||
def _program() -> HiddenProgram:
|
||||
return HiddenProgram(
|
||||
types=("Entity", "Set", "Region", "Number", "Boolean", "State"),
|
||||
relations=(
|
||||
"EQUAL",
|
||||
"NOT_EQUAL",
|
||||
"ABOVE",
|
||||
"BELOW",
|
||||
"MEMBER",
|
||||
"SUBSET",
|
||||
"DISJOINT",
|
||||
"ADJACENT",
|
||||
"CONNECTED",
|
||||
"CAUSES",
|
||||
"LOCATED_IN",
|
||||
),
|
||||
operators=(
|
||||
"NOT",
|
||||
"AND",
|
||||
"OR",
|
||||
"IMPLIES",
|
||||
"ADD",
|
||||
"SUBTRACT",
|
||||
"COMPARE",
|
||||
"UNION",
|
||||
"INTERSECTION",
|
||||
"MOVE",
|
||||
"SWAP",
|
||||
"SET",
|
||||
"COMPOSE",
|
||||
),
|
||||
quantifiers=("ALL", "EXISTS"),
|
||||
structures=("CHAIN", "BRANCH", "MERGE", "CYCLE", "NEST", "CONDITION", "SEQUENCE"),
|
||||
inference_rules=(
|
||||
"ABOVE(x,y) AND ABOVE(y,z) IMPLIES ABOVE(x,z)",
|
||||
"ABOVE(x,y) IMPLIES BELOW(y,x)",
|
||||
"MEMBER(x,A) AND SUBSET(A,B) IMPLIES MEMBER(x,B)",
|
||||
"MEMBER(x,A) AND DISJOINT(A,B) IMPLIES NOT MEMBER(x,B)",
|
||||
"CONNECTED is symmetric",
|
||||
"Numeric updates compose in temporal order",
|
||||
),
|
||||
transition_rules=(
|
||||
TransitionRule(
|
||||
"move",
|
||||
"SELECT_ENTITY",
|
||||
("target region differs from current region",),
|
||||
0.72,
|
||||
("MOVE(entity, region)",),
|
||||
),
|
||||
TransitionRule(
|
||||
"encounter-order",
|
||||
"COLOCATED_PAIR",
|
||||
("entities share a region", "latent order levels are adjacent"),
|
||||
0.78,
|
||||
("SET ABOVE(higher, lower)", "SET BELOW(lower, higher)"),
|
||||
),
|
||||
TransitionRule(
|
||||
"membership",
|
||||
"ENTITY_SET_CONTACT",
|
||||
("set is at base level",),
|
||||
0.68,
|
||||
("SET MEMBER(entity, set)",),
|
||||
),
|
||||
TransitionRule(
|
||||
"subset",
|
||||
"SET_PAIR_CONTACT",
|
||||
("set levels are adjacent",),
|
||||
0.76,
|
||||
("SET SUBSET(lower, higher)",),
|
||||
),
|
||||
TransitionRule(
|
||||
"disjoint",
|
||||
"SET_PAIR_CONTACT",
|
||||
("sets have equal level", "sets are distinct"),
|
||||
0.64,
|
||||
("SET DISJOINT(left, right)",),
|
||||
),
|
||||
TransitionRule(
|
||||
"numeric-update",
|
||||
"COUNTER_PULSE",
|
||||
(),
|
||||
0.82,
|
||||
("ADD or SUBTRACT(counter, amount)",),
|
||||
),
|
||||
TransitionRule(
|
||||
"connect",
|
||||
"ENTITY_PAIR_CONTACT",
|
||||
(),
|
||||
0.55,
|
||||
("SET CONNECTED(left, right)",),
|
||||
),
|
||||
TransitionRule(
|
||||
"state-change",
|
||||
"ENTITY_SIGNAL",
|
||||
(),
|
||||
0.58,
|
||||
("SET Boolean state",),
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
operations = {
|
||||
"EQ": lambda left, right: left == right,
|
||||
"LE": lambda left, right: left <= right,
|
||||
"GT": lambda left, right: left > right,
|
||||
"IS_SET": lambda left, right: left is bool(right),
|
||||
|
||||
def _entities(entity_count: int) -> tuple[list[Entity], dict[str, list[str]]]:
|
||||
if entity_count < 48:
|
||||
raise ValueError("entity_count must be at least 48")
|
||||
region_count = max(4, entity_count // 32)
|
||||
set_count = max(24, entity_count // 5)
|
||||
counter_count = max(8, entity_count // 16)
|
||||
agent_count = entity_count - region_count - set_count - counter_count
|
||||
groups = {
|
||||
"Entity": [f"E{i:03d}" for i in range(agent_count)],
|
||||
"Set": [f"S{i:03d}" for i in range(set_count)],
|
||||
"Region": [f"R{i:03d}" for i in range(region_count)],
|
||||
"Number": [f"N{i:03d}" for i in range(counter_count)],
|
||||
}
|
||||
try:
|
||||
return operations[condition.operator](actual, condition.value)
|
||||
except KeyError as exc:
|
||||
raise ValueError(f"unknown condition operator: {condition.operator}") from exc
|
||||
entities = [Entity(identifier, kind) for kind, values in groups.items() for identifier in values]
|
||||
return entities, groups
|
||||
|
||||
|
||||
def _apply_action(state: _State, rule: HiddenRule) -> _State:
|
||||
if rule.action == "SET_FLAG":
|
||||
return _State(state.present, state.invalidated, state.flags | {rule.target})
|
||||
if rule.action == "ADD_RECORD":
|
||||
if state.invalidated:
|
||||
return state
|
||||
return _State(True, False, state.flags)
|
||||
if rule.action == "INVALIDATE_RECORD":
|
||||
return _State(False, True, state.flags)
|
||||
raise ValueError(f"unknown action: {rule.action}")
|
||||
|
||||
|
||||
def exact_present_probability(
|
||||
system: HiddenSystem, attributes: dict[str, Any], events: tuple[str, ...]
|
||||
) -> float:
|
||||
"""Enumerate all stochastic branches while respecting event and state order."""
|
||||
initial = _State(present=system.initial_record == "PRESENT")
|
||||
distribution: dict[_State, float] = {initial: 1.0}
|
||||
for event in events:
|
||||
for rule in (item for item in system.rules if item.on_event == event):
|
||||
next_distribution: dict[_State, float] = {}
|
||||
for state, mass in distribution.items():
|
||||
if not all(_condition_holds(item, attributes, state) for item in rule.conditions):
|
||||
next_distribution[state] = next_distribution.get(state, 0.0) + mass
|
||||
continue
|
||||
inactive_mass = mass * (1.0 - rule.probability)
|
||||
active_state = _apply_action(state, rule)
|
||||
next_distribution[state] = next_distribution.get(state, 0.0) + inactive_mass
|
||||
next_distribution[active_state] = (
|
||||
next_distribution.get(active_state, 0.0) + mass * rule.probability
|
||||
)
|
||||
distribution = next_distribution
|
||||
return sum(mass for state, mass in distribution.items() if state.present)
|
||||
|
||||
|
||||
def _simulate_once(
|
||||
system: HiddenSystem,
|
||||
attributes: dict[str, Any],
|
||||
events: tuple[str, ...],
|
||||
rng: random.Random,
|
||||
) -> bool:
|
||||
state = _State(present=system.initial_record == "PRESENT")
|
||||
for event in events:
|
||||
for rule in (item for item in system.rules if item.on_event == event):
|
||||
if all(_condition_holds(item, attributes, state) for item in rule.conditions):
|
||||
if rng.random() < rule.probability:
|
||||
state = _apply_action(state, rule)
|
||||
return state.present
|
||||
|
||||
|
||||
def _observe_case(
|
||||
case_id: str,
|
||||
system: HiddenSystem,
|
||||
attributes: dict[str, Any],
|
||||
events: tuple[str, ...],
|
||||
trials: int,
|
||||
rng: random.Random,
|
||||
) -> BehaviorCase:
|
||||
present = sum(_simulate_once(system, attributes, events, rng) for _ in range(trials))
|
||||
return BehaviorCase(
|
||||
id=case_id,
|
||||
attributes=attributes,
|
||||
events=events,
|
||||
trials=trials,
|
||||
observed_present=present,
|
||||
observed_absent=trials - present,
|
||||
def _initial_world(
|
||||
groups: dict[str, list[str]], rng: random.Random, proof_length: int
|
||||
) -> tuple[_World, dict[str, Any]]:
|
||||
agents, sets, regions, counters = (
|
||||
groups["Entity"],
|
||||
groups["Set"],
|
||||
groups["Region"],
|
||||
groups["Number"],
|
||||
)
|
||||
|
||||
|
||||
def _build_system(index: int, complexity: int, rng: random.Random) -> HiddenSystem:
|
||||
"""Build a finite stochastic program with a gated rule and veto conflicts."""
|
||||
present_favored = index % 2 == 0
|
||||
add_probability = rng.choice((0.85, 0.9, 0.95)) if present_favored else rng.choice((0.65, 0.75))
|
||||
veto_probability = rng.choice((0.15, 0.25)) if present_favored else rng.choice((0.7, 0.8, 0.9))
|
||||
marker_probability = rng.choice((0.6, 0.75, 0.9))
|
||||
gated_action = "ADD_RECORD" if present_favored else "INVALIDATE_RECORD"
|
||||
gated_probability = rng.choice((0.6, 0.75, 0.85))
|
||||
|
||||
rules = [
|
||||
HiddenRule("R0", "E0", (), add_probability, "ADD_RECORD", "record"),
|
||||
HiddenRule(
|
||||
"R1",
|
||||
"E1",
|
||||
(Condition("attribute", "load", "LE", 4),),
|
||||
veto_probability,
|
||||
"INVALIDATE_RECORD",
|
||||
"record",
|
||||
),
|
||||
HiddenRule("R2", "E2", (), marker_probability, "SET_FLAG", "marked"),
|
||||
HiddenRule(
|
||||
"R3",
|
||||
"E3",
|
||||
(Condition("state", "marked", "IS_SET", True),),
|
||||
gated_probability,
|
||||
gated_action,
|
||||
"record",
|
||||
),
|
||||
order_levels = max(proof_length + 2, 8)
|
||||
set_levels = max(proof_length + 1, 8)
|
||||
locations = {agent: rng.choice(regions) for agent in agents}
|
||||
world = _World(
|
||||
locations=locations,
|
||||
order_level={agent: index % order_levels for index, agent in enumerate(agents)},
|
||||
set_level={item: index % set_levels for index, item in enumerate(sets)},
|
||||
numbers={counter: rng.randint(10, 90) for counter in counters},
|
||||
)
|
||||
public = [
|
||||
{"id": f"I{index:04d}", "predicate": "LOCATED_IN", "arguments": [agent, region]}
|
||||
for index, (agent, region) in enumerate(locations.items())
|
||||
]
|
||||
for number in range(4, complexity):
|
||||
action = "ADD_RECORD" if (number + index) % 3 else "INVALIDATE_RECORD"
|
||||
probability = rng.choice((0.2, 0.35, 0.5))
|
||||
condition = Condition("attribute", "class", "EQ", "priority")
|
||||
rules.append(
|
||||
HiddenRule(f"R{number}", f"E{number}", (condition,), probability, action, "record")
|
||||
)
|
||||
return HiddenSystem(
|
||||
initial_record="ABSENT",
|
||||
conflict_policy="PERSISTENT_INVALIDATION_VETO",
|
||||
rules=tuple(rules),
|
||||
offset = len(public)
|
||||
public.extend(
|
||||
{
|
||||
"id": f"I{offset + index:04d}",
|
||||
"predicate": "INITIAL_VALUE",
|
||||
"arguments": [counter, value],
|
||||
}
|
||||
for index, (counter, value) in enumerate(world.numbers.items())
|
||||
)
|
||||
hidden = {
|
||||
"order_level": world.order_level,
|
||||
"set_level": world.set_level,
|
||||
"initial_locations": dict(locations),
|
||||
"initial_numbers": dict(world.numbers),
|
||||
}
|
||||
return world, {"public_observations": public, "hidden_state": hidden}
|
||||
|
||||
|
||||
def _support_event_sets(complexity: int) -> list[tuple[str, ...]]:
|
||||
events = [f"E{number}" for number in range(complexity)]
|
||||
cases: list[tuple[str, ...]] = [(event,) for event in events]
|
||||
cases.extend((("E2", "E3"), ("E3", "E2"), ("E0", "E1")))
|
||||
cases.extend(tuple(pair) for pair in combinations(events, 2) if tuple(pair) not in cases)
|
||||
return cases
|
||||
def _event(
|
||||
events: list[WorldEvent],
|
||||
time: int,
|
||||
kind: str,
|
||||
participants: tuple[str, ...],
|
||||
**arguments: Any,
|
||||
) -> WorldEvent:
|
||||
item = WorldEvent(f"V{len(events):05d}", time, kind, participants, arguments)
|
||||
events.append(item)
|
||||
return item
|
||||
|
||||
|
||||
def _generate_one(index: int, complexity: int, rng: random.Random) -> BehaviorSkeleton:
|
||||
system = _build_system(index, complexity, rng)
|
||||
low = {"load": 3, "class": "priority"}
|
||||
high = {"load": 8, "class": "standard"}
|
||||
query_attributes = dict(low)
|
||||
query_events = tuple(f"E{number}" for number in range(complexity))
|
||||
probability = exact_present_probability(system, query_attributes, query_events)
|
||||
if abs(probability - 0.5) < 0.08:
|
||||
raise RuntimeError("generated query has an insufficient probability margin")
|
||||
gold = "PRESENT" if probability > 0.5 else "ABSENT"
|
||||
def _simulate(
|
||||
world: _World,
|
||||
groups: dict[str, list[str]],
|
||||
steps: int,
|
||||
rng: random.Random,
|
||||
) -> list[WorldEvent]:
|
||||
agents, sets, regions, counters = (
|
||||
groups["Entity"],
|
||||
groups["Set"],
|
||||
groups["Region"],
|
||||
groups["Number"],
|
||||
)
|
||||
events: list[WorldEvent] = []
|
||||
kinds = ("ENCOUNTER", "MOVE", "MEMBER", "SUBSET", "DISJOINT", "NUMBER", "CONNECT", "STATE")
|
||||
weights = (30, 12, 11, 15, 7, 11, 8, 6)
|
||||
probabilities = {
|
||||
"ENCOUNTER": 0.78,
|
||||
"MOVE": 0.72,
|
||||
"MEMBER": 0.68,
|
||||
"SUBSET": 0.76,
|
||||
"DISJOINT": 0.64,
|
||||
"NUMBER": 0.82,
|
||||
"CONNECT": 0.55,
|
||||
"STATE": 0.58,
|
||||
}
|
||||
|
||||
trials = 48 if complexity <= 4 else 64
|
||||
support: list[BehaviorCase] = []
|
||||
for case_number, events in enumerate(_support_event_sets(complexity)):
|
||||
attributes = low if case_number % 3 else high
|
||||
support.append(
|
||||
_observe_case(
|
||||
f"history-{case_number:03d}", system, dict(attributes), events, trials, rng
|
||||
for time in range(1, steps + 1):
|
||||
kind = rng.choices(kinds, weights=weights, k=1)[0]
|
||||
if rng.random() >= probabilities[kind]:
|
||||
continue
|
||||
|
||||
if kind == "MOVE":
|
||||
agent = rng.choice(agents)
|
||||
choices = [region for region in regions if region != world.locations[agent]]
|
||||
destination = rng.choice(choices)
|
||||
origin = world.locations[agent]
|
||||
world.locations[agent] = destination
|
||||
_event(events, time, "MOVE", (agent, origin, destination), origin=origin, destination=destination)
|
||||
|
||||
elif kind == "ENCOUNTER":
|
||||
lower_level = rng.randrange(max(world.order_level.values()))
|
||||
lower = [item for item in agents if world.order_level[item] == lower_level]
|
||||
higher = [item for item in agents if world.order_level[item] == lower_level + 1]
|
||||
if not lower or not higher:
|
||||
continue
|
||||
low, high = rng.choice(lower), rng.choice(higher)
|
||||
if world.locations[low] != world.locations[high] or (high, low) in world.above:
|
||||
continue
|
||||
item = _event(
|
||||
events,
|
||||
time,
|
||||
"RELATE",
|
||||
(high, low),
|
||||
predicate="ABOVE",
|
||||
subject=high,
|
||||
object=low,
|
||||
region=world.locations[high],
|
||||
)
|
||||
world.above[(high, low)] = item.id
|
||||
|
||||
elif kind == "MEMBER":
|
||||
agent = rng.choice(agents)
|
||||
base_sets = [item for item in sets if world.set_level[item] == 0]
|
||||
container = rng.choice(base_sets)
|
||||
if (agent, container) in world.members:
|
||||
continue
|
||||
item = _event(
|
||||
events,
|
||||
time,
|
||||
"RELATE",
|
||||
(agent, container),
|
||||
predicate="MEMBER",
|
||||
subject=agent,
|
||||
object=container,
|
||||
)
|
||||
world.members[(agent, container)] = item.id
|
||||
|
||||
elif kind == "SUBSET":
|
||||
lower_level = rng.randrange(max(world.set_level.values()))
|
||||
lower = [item for item in sets if world.set_level[item] == lower_level]
|
||||
higher = [item for item in sets if world.set_level[item] == lower_level + 1]
|
||||
child, parent = rng.choice(lower), rng.choice(higher)
|
||||
if (child, parent) in world.subsets or any(left == child for left, _right in world.subsets):
|
||||
continue
|
||||
item = _event(
|
||||
events,
|
||||
time,
|
||||
"RELATE",
|
||||
(child, parent),
|
||||
predicate="SUBSET",
|
||||
subject=child,
|
||||
object=parent,
|
||||
)
|
||||
world.subsets[(child, parent)] = item.id
|
||||
|
||||
elif kind == "DISJOINT":
|
||||
level = rng.randrange(max(world.set_level.values()) + 1)
|
||||
same_level = [item for item in sets if world.set_level[item] == level]
|
||||
if len(same_level) < 2:
|
||||
continue
|
||||
left, right = rng.sample(same_level, 2)
|
||||
key = frozenset((left, right))
|
||||
if key in world.disjoint:
|
||||
continue
|
||||
item = _event(
|
||||
events,
|
||||
time,
|
||||
"RELATE",
|
||||
(left, right),
|
||||
predicate="DISJOINT",
|
||||
subject=left,
|
||||
object=right,
|
||||
)
|
||||
world.disjoint[key] = item.id
|
||||
|
||||
elif kind == "NUMBER":
|
||||
counter = rng.choice(counters)
|
||||
delta = rng.choice((-7, -5, -3, 2, 4, 6))
|
||||
before = world.numbers[counter]
|
||||
after = max(0, before + delta)
|
||||
world.numbers[counter] = after
|
||||
_event(
|
||||
events,
|
||||
time,
|
||||
"UPDATE_NUMBER",
|
||||
(counter,),
|
||||
operation="ADD" if delta >= 0 else "SUBTRACT",
|
||||
amount=abs(delta),
|
||||
before=before,
|
||||
after=after,
|
||||
)
|
||||
|
||||
elif kind == "CONNECT":
|
||||
left, right = rng.sample(agents, 2)
|
||||
key = frozenset((left, right))
|
||||
if key in world.connected:
|
||||
continue
|
||||
world.connected.add(key)
|
||||
_event(
|
||||
events,
|
||||
time,
|
||||
"RELATE",
|
||||
(left, right),
|
||||
predicate="CONNECTED",
|
||||
subject=left,
|
||||
object=right,
|
||||
)
|
||||
|
||||
else:
|
||||
agent = rng.choice(agents)
|
||||
flag = rng.choice(("ACTIVE", "SEALED", "VERIFIED"))
|
||||
before = world.flags.get((agent, flag), False)
|
||||
after = not before
|
||||
world.flags[(agent, flag)] = after
|
||||
_event(
|
||||
events,
|
||||
time,
|
||||
"SET_STATE",
|
||||
(agent,),
|
||||
state=flag,
|
||||
before=before,
|
||||
after=after,
|
||||
)
|
||||
return events
|
||||
|
||||
|
||||
def _shortest_path(
|
||||
edges: dict[tuple[str, str], str], minimum: int
|
||||
) -> tuple[str, str, tuple[str, ...], tuple[str, ...]] | None:
|
||||
adjacency: dict[str, list[tuple[str, str]]] = {}
|
||||
for (left, right), event_id in edges.items():
|
||||
adjacency.setdefault(left, []).append((right, event_id))
|
||||
candidates: list[tuple[str, str, tuple[str, ...], tuple[str, ...]]] = []
|
||||
for source in adjacency:
|
||||
queue = deque([(source, (source,), ())])
|
||||
seen = {source}
|
||||
while queue:
|
||||
node, nodes, event_ids = queue.popleft()
|
||||
if len(event_ids) >= minimum:
|
||||
candidates.append((source, node, nodes, event_ids))
|
||||
continue
|
||||
for target, event_id in adjacency.get(node, []):
|
||||
if target not in seen:
|
||||
seen.add(target)
|
||||
queue.append((target, nodes + (target,), event_ids + (event_id,)))
|
||||
if not candidates:
|
||||
return None
|
||||
candidates.sort(key=lambda item: (len(item[3]), item[0], item[1]))
|
||||
return candidates[0]
|
||||
|
||||
|
||||
def _membership_path(
|
||||
world: _World, minimum: int
|
||||
) -> tuple[str, str, tuple[str, ...], tuple[str, ...]] | None:
|
||||
subset_path = _shortest_path(world.subsets, minimum - 1)
|
||||
if subset_path is None:
|
||||
return None
|
||||
candidates = []
|
||||
for (entity, base_set), member_event in world.members.items():
|
||||
path = _shortest_path_from(world.subsets, base_set, minimum - 1)
|
||||
if path:
|
||||
target, nodes, event_ids = path
|
||||
candidates.append((entity, target, (member_event,) + event_ids, (entity,) + nodes))
|
||||
if not candidates:
|
||||
return None
|
||||
candidates.sort(key=lambda item: (len(item[2]), item[0], item[1]))
|
||||
entity, target, event_ids, nodes = candidates[0]
|
||||
return entity, target, nodes, event_ids
|
||||
|
||||
|
||||
def _membership_negative(
|
||||
world: _World, minimum: int
|
||||
) -> tuple[str, str, tuple[str, ...], tuple[str, ...]] | None:
|
||||
adjacency: dict[str, list[tuple[str, str]]] = {}
|
||||
for (left, right), event_id in world.subsets.items():
|
||||
adjacency.setdefault(left, []).append((right, event_id))
|
||||
for (entity, base_set), member_event in sorted(world.members.items()):
|
||||
reachable = {base_set}
|
||||
queue = deque([(base_set, (base_set,), ())])
|
||||
paths: list[tuple[str, tuple[str, ...], tuple[str, ...]]] = []
|
||||
while queue:
|
||||
node, nodes, event_ids = queue.popleft()
|
||||
paths.append((node, nodes, event_ids))
|
||||
for target, event_id in adjacency.get(node, []):
|
||||
if target not in reachable:
|
||||
reachable.add(target)
|
||||
queue.append((target, nodes + (target,), event_ids + (event_id,)))
|
||||
for known_set, nodes, subset_events in paths:
|
||||
if 1 + len(subset_events) < minimum - 1:
|
||||
continue
|
||||
for pair, disjoint_event in world.disjoint.items():
|
||||
if known_set not in pair:
|
||||
continue
|
||||
other = next(item for item in pair if item != known_set)
|
||||
if other in reachable:
|
||||
continue
|
||||
proof_ids = (member_event,) + subset_events + (disjoint_event,)
|
||||
proof_nodes = (entity,) + nodes + (other,)
|
||||
return entity, other, proof_nodes, proof_ids
|
||||
return None
|
||||
|
||||
|
||||
def _shortest_path_from(
|
||||
edges: dict[tuple[str, str], str], source: str, minimum: int
|
||||
) -> tuple[str, tuple[str, ...], tuple[str, ...]] | None:
|
||||
adjacency: dict[str, list[tuple[str, str]]] = {}
|
||||
for (left, right), event_id in edges.items():
|
||||
adjacency.setdefault(left, []).append((right, event_id))
|
||||
queue = deque([(source, (source,), ())])
|
||||
seen = {source}
|
||||
while queue:
|
||||
node, nodes, event_ids = queue.popleft()
|
||||
if len(event_ids) >= minimum:
|
||||
return node, nodes, event_ids
|
||||
for target, event_id in adjacency.get(node, []):
|
||||
if target not in seen:
|
||||
seen.add(target)
|
||||
queue.append((target, nodes + (target,), event_ids + (event_id,)))
|
||||
return None
|
||||
|
||||
|
||||
def _choose_query(
|
||||
index: int, complexity: int, world: _World, events: list[WorldEvent]
|
||||
) -> tuple[Query, str, tuple[str, ...], tuple[str, ...], str] | None:
|
||||
if index % 2 == 0:
|
||||
path = _shortest_path(world.above, complexity)
|
||||
if path is None:
|
||||
return None
|
||||
source, target, nodes, proof_ids = path
|
||||
positive = index % 4 == 0
|
||||
arguments = (source, target) if positive else (target, source)
|
||||
gold = "YES" if positive else "NO"
|
||||
facts = tuple(
|
||||
f"ABOVE({nodes[position]},{nodes[position + 1]})"
|
||||
for position in range(len(nodes) - 1)
|
||||
)
|
||||
if not positive:
|
||||
facts += (f"ABOVE({source},{target}) therefore NOT ABOVE({target},{source})",)
|
||||
return Query("ABOVE", arguments, events[-1].time), gold, proof_ids, facts, "spatial_order"
|
||||
|
||||
if index % 4 == 1:
|
||||
negative = _membership_negative(world, complexity)
|
||||
if negative is None:
|
||||
return None
|
||||
entity, target, nodes, proof_ids = negative
|
||||
known_set, disjoint_set = nodes[-2], nodes[-1]
|
||||
facts = [f"MEMBER({entity},{nodes[1]})"]
|
||||
facts.extend(
|
||||
f"SUBSET({nodes[position]},{nodes[position + 1]})"
|
||||
for position in range(1, len(nodes) - 2)
|
||||
)
|
||||
facts.extend(
|
||||
(
|
||||
f"DISJOINT({known_set},{disjoint_set})",
|
||||
f"therefore NOT MEMBER({entity},{target})",
|
||||
)
|
||||
)
|
||||
# Keeping events fixed while changing load makes the veto guard observable.
|
||||
next_id = len(support)
|
||||
support.append(
|
||||
_observe_case(f"history-{next_id:03d}", system, dict(low), ("E0", "E1"), trials, rng)
|
||||
)
|
||||
support.append(
|
||||
_observe_case(
|
||||
f"history-{next_id + 1:03d}", system, dict(high), ("E0", "E1"), trials, rng
|
||||
return (
|
||||
Query("MEMBER", (entity, target), events[-1].time),
|
||||
"NO",
|
||||
proof_ids,
|
||||
tuple(facts),
|
||||
"membership_disjointness",
|
||||
)
|
||||
|
||||
membership = _membership_path(world, complexity)
|
||||
if membership is None:
|
||||
return None
|
||||
entity, target, nodes, proof_ids = membership
|
||||
facts = [f"MEMBER({entity},{nodes[1]})"]
|
||||
facts.extend(
|
||||
f"SUBSET({nodes[position]},{nodes[position + 1]})"
|
||||
for position in range(1, len(nodes) - 1)
|
||||
)
|
||||
support.append(
|
||||
_observe_case(
|
||||
f"history-{next_id + 2:03d}",
|
||||
system,
|
||||
dict(low),
|
||||
("E0", "E2", "E3"),
|
||||
trials,
|
||||
rng,
|
||||
)
|
||||
return (
|
||||
Query("MEMBER", (entity, target), events[-1].time),
|
||||
"YES",
|
||||
proof_ids,
|
||||
tuple(facts),
|
||||
"membership_inheritance",
|
||||
)
|
||||
|
||||
query = BehaviorCase(
|
||||
id="query",
|
||||
attributes=query_attributes,
|
||||
events=query_events,
|
||||
trials=1,
|
||||
observed_present=None,
|
||||
observed_absent=None,
|
||||
|
||||
def _final_state(world: _World) -> dict[str, Any]:
|
||||
return {
|
||||
"locations": dict(world.locations),
|
||||
"above": [[left, right] for left, right in sorted(world.above)],
|
||||
"subsets": [[left, right] for left, right in sorted(world.subsets)],
|
||||
"members": [[entity, container] for entity, container in sorted(world.members)],
|
||||
"disjoint": [sorted(pair) for pair in sorted(world.disjoint, key=lambda item: sorted(item))],
|
||||
"connected": [sorted(pair) for pair in sorted(world.connected, key=lambda item: sorted(item))],
|
||||
"numbers": dict(world.numbers),
|
||||
"flags": [
|
||||
{"entity": entity, "state": state, "value": value}
|
||||
for (entity, state), value in sorted(world.flags.items())
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _generate_one(
|
||||
index: int,
|
||||
complexity: int,
|
||||
entity_count: int,
|
||||
simulation_steps: int,
|
||||
rng: random.Random,
|
||||
) -> WorldSkeleton:
|
||||
entities, groups = _entities(entity_count)
|
||||
world, initial_state = _initial_world(groups, rng, complexity)
|
||||
events = _simulate(world, groups, simulation_steps, rng)
|
||||
if not events:
|
||||
raise RuntimeError("simulation produced no events")
|
||||
selected = _choose_query(index, complexity, world, events)
|
||||
if selected is None:
|
||||
raise RuntimeError("simulation did not produce a sufficiently deep proof")
|
||||
query, gold, proof_ids, proof_facts, query_family = selected
|
||||
direct = any(
|
||||
event.kind == "RELATE"
|
||||
and event.arguments.get("predicate") == query.predicate
|
||||
and (event.arguments.get("subject"), event.arguments.get("object")) == query.arguments
|
||||
for event in events
|
||||
)
|
||||
trace = tuple(
|
||||
[f"query events: {' -> '.join(query_events)}"]
|
||||
+ [
|
||||
f"{rule.id}: on {rule.on_event}, p={rule.probability}, {rule.action}({rule.target})"
|
||||
for rule in system.rules
|
||||
]
|
||||
+ [f"exact P(PRESENT)={probability:.8f}", f"gold={gold}"]
|
||||
)
|
||||
return BehaviorSkeleton(
|
||||
id=f"dynamic-{index:04d}",
|
||||
family="stochastic_behavioral_system",
|
||||
if direct:
|
||||
raise RuntimeError("query was directly stated in the event history")
|
||||
checks = {
|
||||
"direct_query_event_absent": True,
|
||||
"proof_length": len(proof_ids),
|
||||
"minimum_proof_length": complexity,
|
||||
"event_count": len(events),
|
||||
"proof_event_ratio": len(proof_ids) / len(events),
|
||||
"query_family": query_family,
|
||||
"weak_baselines": {
|
||||
"direct_fact": False,
|
||||
"proof_shorter_than_requested": len(proof_ids) < complexity,
|
||||
"proof_is_large_fraction_of_history": len(proof_ids) / len(events) >= 0.05,
|
||||
},
|
||||
}
|
||||
return WorldSkeleton(
|
||||
id=f"world-{index:04d}",
|
||||
family="dynamic_multi_logic_world",
|
||||
complexity=complexity,
|
||||
event_vocabulary=tuple(f"E{number}" for number in range(complexity)),
|
||||
attribute_vocabulary={"load": "integer", "class": "category"},
|
||||
observation_protocol=(
|
||||
"Each trial begins with an absent record and no internal flags.",
|
||||
"Events occur from left to right; the same hidden system is used in every trial.",
|
||||
"PRESENT and ABSENT counts are final observations over independent repeated trials.",
|
||||
"For the query, choose which final outcome is more likely; an exact probability is not required.",
|
||||
),
|
||||
hidden_system=system,
|
||||
support_cases=tuple(support),
|
||||
query_case=query,
|
||||
entity_count=entity_count,
|
||||
simulation_steps=simulation_steps,
|
||||
hidden_program=_program(),
|
||||
entities=tuple(entities),
|
||||
initial_state=initial_state,
|
||||
events=tuple(events),
|
||||
final_state=_final_state(world),
|
||||
query=query,
|
||||
answer_options=ANSWER_OPTIONS,
|
||||
gold_answer=gold,
|
||||
gold_present_probability=probability,
|
||||
latent_trace=trace,
|
||||
proof_event_ids=proof_ids,
|
||||
proof_facts=proof_facts,
|
||||
generator_checks=checks,
|
||||
)
|
||||
|
||||
|
||||
def generate_skeletons(count: int, length: str | int, seed: int = 0) -> list[BehaviorSkeleton]:
|
||||
def generate_skeletons(
|
||||
count: int,
|
||||
length: str | int,
|
||||
seed: int = 0,
|
||||
entity_count: int = 128,
|
||||
simulation_steps: int = 1000,
|
||||
) -> list[WorldSkeleton]:
|
||||
if count < 1:
|
||||
raise ValueError("count must be positive")
|
||||
if simulation_steps < 100:
|
||||
raise ValueError("simulation_steps must be at least 100")
|
||||
complexity = resolve_length(length)
|
||||
rng = random.Random(seed)
|
||||
generated: list[BehaviorSkeleton] = []
|
||||
generated: list[WorldSkeleton] = []
|
||||
for index in range(count):
|
||||
for _attempt in range(100):
|
||||
for _attempt in range(60):
|
||||
try:
|
||||
generated.append(_generate_one(index, complexity, rng))
|
||||
generated.append(
|
||||
_generate_one(index, complexity, entity_count, simulation_steps, rng)
|
||||
)
|
||||
break
|
||||
except RuntimeError as exc:
|
||||
if "insufficient probability margin" not in str(exc):
|
||||
raise
|
||||
except RuntimeError:
|
||||
continue
|
||||
else:
|
||||
raise RuntimeError("could not generate a query with a stable probability margin")
|
||||
raise RuntimeError("could not generate a world with the requested proof complexity")
|
||||
return generated
|
||||
|
||||
+2
-2
@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "logic-skeleton-bench"
|
||||
version = "0.2.0"
|
||||
description = "Infer stochastic hidden systems from rendered behavioral observations."
|
||||
version = "0.3.0"
|
||||
description = "Generate and solve proof-bearing histories from large evolving worlds."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
license = { text = "MIT" }
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
{
|
||||
"name": "short-smoke",
|
||||
"description": "Infer five short stochastic hidden systems from behavioral histories.",
|
||||
"description": "Solve five proof-bearing queries from large simulated world histories.",
|
||||
"count": 5,
|
||||
"length": "short",
|
||||
"entity_count": 128,
|
||||
"simulation_steps": 1000,
|
||||
"seed": 20260922,
|
||||
"output_dir": "artifacts/short-smoke"
|
||||
}
|
||||
|
||||
+30
-14
@@ -1,3 +1,4 @@
|
||||
import json
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
@@ -21,21 +22,23 @@ class FailingSolver:
|
||||
raise RuntimeError("solver failed")
|
||||
|
||||
|
||||
class AuditRenderer:
|
||||
def request_json(self, **kwargs):
|
||||
payload = json.loads(kwargs["user"][kwargs["user"].find("{") :])
|
||||
identifiers = [item["id"] for item in payload["initial_observations"]]
|
||||
identifiers.extend(item["id"] for item in payload["events"])
|
||||
return {"problem": "A compact behavioral problem.\n" + "\n".join(identifiers)}
|
||||
|
||||
|
||||
class PipelineTests(unittest.TestCase):
|
||||
def test_end_to_end_writes_skeletons_before_scored_results(self):
|
||||
renderer = QueueClient([
|
||||
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
|
||||
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
|
||||
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
|
||||
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
|
||||
{"problem": "Infer from histories; choose PRESENT or ABSENT."},
|
||||
])
|
||||
renderer = AuditRenderer()
|
||||
solver = QueueClient([
|
||||
"The positive behavior dominates.\n#### PRESENT",
|
||||
"The veto behavior dominates.\n#### ABSENT",
|
||||
"The positive behavior dominates.\n#### PRESENT",
|
||||
"The veto behavior dominates.\n#### ABSENT",
|
||||
"The positive behavior dominates.\n#### PRESENT",
|
||||
"The relation follows through the event chain.\n#### YES",
|
||||
"The target set is disjoint from the inherited membership.\n#### NO",
|
||||
"The reverse ordering is false.\n#### NO",
|
||||
"Membership propagates through the nested sets.\n#### YES",
|
||||
"The relation follows through the event chain.\n#### YES",
|
||||
])
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
output = Path(directory)
|
||||
@@ -70,7 +73,7 @@ class PipelineTests(unittest.TestCase):
|
||||
self.assertEqual(parse_solver_answer("Reasoning.\n#### YES", ("YES", "NO")), ("YES", True))
|
||||
|
||||
def test_rendered_problem_is_checkpointed_before_solver_failure(self):
|
||||
renderer = QueueClient([{"problem": "A compact behavioral problem."}])
|
||||
renderer = AuditRenderer()
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
output = Path(directory)
|
||||
with self.assertRaisesRegex(RuntimeError, "solver failed"):
|
||||
@@ -85,7 +88,7 @@ class PipelineTests(unittest.TestCase):
|
||||
self.assertIn("A compact behavioral problem.", (output / "rendered.jsonl").read_text())
|
||||
|
||||
def test_unparseable_solver_output_is_scored_without_stopping_batch(self):
|
||||
renderer = QueueClient([{"problem": "A compact behavioral problem."}])
|
||||
renderer = AuditRenderer()
|
||||
solver = QueueClient(["The response ended before its final label."])
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
output = Path(directory)
|
||||
@@ -101,6 +104,19 @@ class PipelineTests(unittest.TestCase):
|
||||
self.assertEqual(summary["correct"], 0)
|
||||
self.assertIn('"parse_error":', (output / "results.jsonl").read_text())
|
||||
|
||||
def test_renderer_omission_is_rejected_before_solving(self):
|
||||
renderer = QueueClient([{"problem": "This rendering omitted its audit IDs."}])
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
with self.assertRaisesRegex(RuntimeError, "renderer omitted"):
|
||||
run_evaluation(
|
||||
renderer=renderer,
|
||||
solver=FailingSolver(),
|
||||
count=1,
|
||||
length="short",
|
||||
seed=1,
|
||||
output_dir=Path(directory),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
+47
-48
@@ -1,75 +1,74 @@
|
||||
import unittest
|
||||
|
||||
from logicbench.models import Condition, HiddenRule, HiddenSystem
|
||||
from logicbench.skeletons import exact_present_probability, generate_skeletons, resolve_length
|
||||
from logicbench.skeletons import generate_skeletons, resolve_length
|
||||
|
||||
|
||||
class SkeletonTests(unittest.TestCase):
|
||||
def test_short_means_four_query_events(self):
|
||||
skeletons = generate_skeletons(count=3, length="short", seed=42)
|
||||
self.assertEqual([item.complexity for item in skeletons], [4, 4, 4])
|
||||
self.assertTrue(all(len(item.query_case.events) == 4 for item in skeletons))
|
||||
self.assertEqual(
|
||||
[item.gold_answer for item in skeletons], ["PRESENT", "ABSENT", "PRESENT"]
|
||||
)
|
||||
def test_short_world_has_many_entities_and_realized_events(self):
|
||||
skeleton = generate_skeletons(1, "short", seed=42)[0]
|
||||
self.assertEqual(skeleton.entity_count, 128)
|
||||
self.assertEqual(skeleton.simulation_steps, 1000)
|
||||
self.assertGreater(len(skeleton.events), 300)
|
||||
self.assertGreaterEqual(len(skeleton.proof_event_ids), 4)
|
||||
self.assertLess(skeleton.generator_checks["proof_event_ratio"], 0.05)
|
||||
|
||||
def test_generation_is_deterministic(self):
|
||||
first = [item.to_dict() for item in generate_skeletons(6, 4, seed=7)]
|
||||
second = [item.to_dict() for item in generate_skeletons(6, 4, seed=7)]
|
||||
first = [item.to_dict() for item in generate_skeletons(4, 4, seed=7)]
|
||||
second = [item.to_dict() for item in generate_skeletons(4, 4, seed=7)]
|
||||
self.assertEqual(first, second)
|
||||
|
||||
def test_renderer_receives_behavior_but_no_latent_information(self):
|
||||
def test_renderer_receives_realized_history_but_no_program_or_answer(self):
|
||||
skeleton = generate_skeletons(1, "short", seed=1)[0]
|
||||
payload = skeleton.renderer_payload()
|
||||
for secret in (
|
||||
"id",
|
||||
"hidden_system",
|
||||
"hidden_program",
|
||||
"hidden_state",
|
||||
"final_state",
|
||||
"gold_answer",
|
||||
"gold_present_probability",
|
||||
"latent_trace",
|
||||
"proof_event_ids",
|
||||
"proof_facts",
|
||||
"generator_checks",
|
||||
):
|
||||
self.assertNotIn(secret, payload)
|
||||
self.assertIn("support_cases", payload)
|
||||
self.assertIsNone(payload["query_case"]["observed_present"])
|
||||
self.assertEqual(len(payload["events"]), len(skeleton.events))
|
||||
self.assertEqual(len(payload["entity_types"]), 128)
|
||||
|
||||
def test_exact_enumerator_applies_persistent_veto(self):
|
||||
system = HiddenSystem(
|
||||
initial_record="ABSENT",
|
||||
conflict_policy="PERSISTENT_INVALIDATION_VETO",
|
||||
rules=(
|
||||
HiddenRule("add", "A", (), 0.8, "ADD_RECORD", "record"),
|
||||
HiddenRule("veto", "B", (), 0.25, "INVALIDATE_RECORD", "record"),
|
||||
),
|
||||
)
|
||||
self.assertAlmostEqual(exact_present_probability(system, {}, ("A", "B")), 0.6)
|
||||
self.assertAlmostEqual(exact_present_probability(system, {}, ("B", "A")), 0.6)
|
||||
def test_query_is_not_directly_stated_and_proof_events_exist(self):
|
||||
for skeleton in generate_skeletons(5, "short", seed=20260922):
|
||||
query_pair = skeleton.query.arguments
|
||||
direct = [
|
||||
event
|
||||
for event in skeleton.events
|
||||
if event.kind == "RELATE"
|
||||
and event.arguments.get("predicate") == skeleton.query.predicate
|
||||
and (event.arguments.get("subject"), event.arguments.get("object")) == query_pair
|
||||
]
|
||||
self.assertFalse(direct)
|
||||
event_ids = {event.id for event in skeleton.events}
|
||||
self.assertTrue(set(skeleton.proof_event_ids) <= event_ids)
|
||||
self.assertGreaterEqual(len(skeleton.proof_event_ids), skeleton.complexity)
|
||||
|
||||
def test_exact_enumerator_respects_stateful_event_order(self):
|
||||
system = HiddenSystem(
|
||||
initial_record="ABSENT",
|
||||
conflict_policy="PERSISTENT_INVALIDATION_VETO",
|
||||
rules=(
|
||||
HiddenRule("mark", "A", (), 0.5, "SET_FLAG", "ready"),
|
||||
HiddenRule(
|
||||
"add",
|
||||
"B",
|
||||
(Condition("state", "ready", "IS_SET", True),),
|
||||
0.8,
|
||||
"ADD_RECORD",
|
||||
"record",
|
||||
),
|
||||
),
|
||||
)
|
||||
self.assertAlmostEqual(exact_present_probability(system, {}, ("A", "B")), 0.4)
|
||||
self.assertEqual(exact_present_probability(system, {}, ("B", "A")), 0.0)
|
||||
def test_world_contains_multiple_logic_domains(self):
|
||||
skeleton = generate_skeletons(1, "short", seed=3)[0]
|
||||
predicates = {
|
||||
event.arguments.get("predicate")
|
||||
for event in skeleton.events
|
||||
if event.kind == "RELATE"
|
||||
}
|
||||
self.assertTrue({"ABOVE", "MEMBER", "SUBSET", "CONNECTED"} <= predicates)
|
||||
kinds = {event.kind for event in skeleton.events}
|
||||
self.assertTrue({"MOVE", "UPDATE_NUMBER", "SET_STATE"} <= kinds)
|
||||
|
||||
def test_length_validation(self):
|
||||
def test_length_and_scale_validation(self):
|
||||
self.assertEqual(resolve_length("medium"), 6)
|
||||
self.assertEqual(resolve_length("5"), 5)
|
||||
with self.assertRaises(ValueError):
|
||||
resolve_length("huge")
|
||||
with self.assertRaises(ValueError):
|
||||
resolve_length(3)
|
||||
resolve_length(2)
|
||||
with self.assertRaises(ValueError):
|
||||
generate_skeletons(1, "short", entity_count=20)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Reference in New Issue
Block a user