logic-skeleton-bench/docs/skeleton-language.md

Hidden System and Behavioral Skeleton Language

This document specifies version 0.2 of the benchmark. The central change is that natural-language rendering no longer receives a system's rules. It receives only behavior produced by executing those rules.

Three-layer contract

HiddenSystem --execute/sample--> BehaviorSkeleton --LLM render--> Natural-language problem
      |                              |                                  |
 never public                 observations only                    solver input
      |
 exact enumeration --> gold probability --> majority label --> evaluator only

Generation follows this order:

1. A seeded local generator builds an executable HiddenSystem.
2. The simulator runs controlled support cases repeatedly and records aggregate outcomes.
3. A held-out query combines known events in a combination not present in the histories.
4. An exact local enumerator computes the query's true outcome distribution and gold label.
5. The full answer-bearing record is written before either LLM is called.
6. renderer_payload() removes the program, exact probability, gold label, and latent trace.
7. The renderer expresses only the public behavioral evidence in a surface domain.
8. The solver infers a mechanism from those observations and selects the more likely outcome.

This is system identification followed by transfer, not direct deduction from stated laws.

Hidden system grammar

HiddenSystem ::= {
  initial_record,
  conflict_policy,
  rules: HiddenRule+
}

HiddenRule ::= {
  id,
  on_event,
  conditions: Condition*,
  probability,
  action,
  target
}

Condition ::= {
  source: "attribute" | "state",
  key,
  operator: "EQ" | "LE" | "GT" | "IS_SET",
  value
}

action ::= "ADD_RECORD" | "INVALIDATE_RECORD" | "SET_FLAG"
probability ::= real number in [0, 1]

The grammar is finite, while different rule selections, parameters, event sequences, attributes, and stochastic outcomes generate an unbounded set of behavioral structures.

State and execution

Each independent trial starts with:

record = ABSENT
invalidated = false
flags = {}

Events execute from left to right. For each event, matching rules execute in stored order:

1. evaluate all conditions against immutable attributes and current state;
2. if they hold, sample that rule's Bernoulli trigger;
3. if triggered, apply its action immediately.

Actions mean:

• SET_FLAG(x): add internal flag x; later rules can test it;
• ADD_RECORD: make the record present unless it was previously invalidated;
• INVALIDATE_RECORD: remove the record and permanently set invalidated=true.

The current PERSISTENT_INVALIDATION_VETO policy therefore implements a one-vote veto. A later add cannot restore an invalidated record. Event order matters because a flag can enable a later rule and an early veto persists.

Probabilities belong to hidden rule activation. They are not printed in the rendered problem. The task asks whether PRESENT or ABSENT has probability greater than one half, not for a numerical probability.

Exact gold computation versus sampled evidence

These are deliberately separate:

• Support histories are finite empirical data. The seeded simulator samples each case 48 times for short tasks and 64 times for longer tasks.
• The gold query probability is not estimated by Monte Carlo. The evaluator propagates a probability distribution over every reachable internal state, branching at every applicable stochastic rule.

For each state with mass m and applicable rule probability p, exact enumeration creates:

inactive branch: same state, mass m(1-p)
active branch:   action(state), mass mp

Equal resulting states are merged after each rule. At the end, the masses of all states whose record is present are summed. The answer is PRESENT when that sum exceeds 0.5, otherwise ABSENT. Current generation rejects cases within 0.08 of the boundary to avoid fragile labels.

Behavioral skeleton grammar

BehaviorSkeleton ::= {
  id,
  family,
  complexity,
  event_vocabulary,
  attribute_vocabulary,
  observation_protocol,
  hidden_system,
  support_cases: BehaviorCase+,
  query_case: BehaviorCase,
  answer_options: ["PRESENT", "ABSENT"],
  gold_answer,
  gold_present_probability,
  latent_trace
}

BehaviorCase ::= {
  id,
  attributes,
  events,
  trials,
  observed_present,
  observed_absent
}

Support cases have integer observation counts that sum to trials. The query has null observation counts because its result is what the solver must predict.

The public payload contains:

family, complexity,
event_vocabulary, attribute_vocabulary,
observation_protocol, support_cases,
query_case, answer_options

It never contains:

id, hidden_system, gold_answer,
gold_present_probability, latent_trace

Current generated structure

Every system has at least four interacting rules:

• an unconditional probabilistic add event;
• an attribute-gated probabilistic veto event;
• an event that probabilistically sets an invisible flag;
• a later event whose effect requires that flag.

Longer configurations add attribute-gated add or veto rules. Support evidence includes:

• every event in isolation;
• pairwise event combinations;
• both orders of the state-gated pair;
• low/high attribute counterfactual controls;
• repeated outcomes rather than rule annotations.

The query applies all events in order. Its complete combination is held out from support evidence, so the model must compose effects learned from smaller interventions.

Generated indices alternate between present-favored and absent-favored systems to prevent a one-label benchmark. The index is redacted from the renderer payload so this balancing mechanism cannot become a shortcut. It affects hidden parameter ranges, never the answer after generation; the exact enumerator remains authoritative.

Complexity and determinism

complexity is the number of distinct events in the held-out sequence and the minimum number of corresponding hidden rules. It is independent of rendered token count.

Preset	Ordered query events
short	4
medium	6
long	8

Explicit values from 4 through 10 are accepted. generate_skeletons(count, length, seed) uses a local pseudorandom generator; the same code version and arguments produce equivalent hidden systems, samples, queries, and gold answers.

Renderer obligations

The renderer may rename event codes, attributes, states, and outcomes into a coherent scenario. It must preserve every numeric count, attribute value, event order, trial boundary, query input, and answer label.

It must not:

• receive or state hidden rules;
• turn empirical correlations into declared laws;
• expose trigger probabilities, internal flags, latent traces, or the answer;
• collapse a repeated experiment into a deterministic statement;
• add causal or real-world assumptions;
• show the exact query combination as an already observed history.

Retaining the complete pre-render record makes rendering fidelity independently auditable.

Extension points

The next DSL extensions can remain compatible with the three-layer boundary:

• multiple entities and relations between them;
• delayed actions, expiry, and multiple clock phases;
• priority, last-write, merge, and fallback conflict policies;
• partial observation and noisy sensors distinct from stochastic behavior;
• candidate-program enumeration to report identifiability;
• counterfactual and intervention queries in addition to outcome prediction.

Any extension must keep gold computation local, expose behavior rather than source rules, preserve seeded reproducibility, and add tests for execution order, conflict semantics, and renderer redaction.