Replace explicit rules with behavioral hidden systems

This commit is contained in:
2026-09-23 00:10:37 +08:00
parent 3cf3db7b12
commit 9f54f8f5a3
12 changed files with 598 additions and 380 deletions
+25 -26
View File
@@ -1,28 +1,33 @@
# Logic Skeleton Bench
Logic Skeleton Bench tests whether a solver transfers the same abstract structure across different surface stories. It first creates a typed intermediate representation with a deterministic gold answer, asks one LLM to render that representation as a natural-language problem, asks another LLM to solve only the rendered problem, and then scores the answer.
Logic Skeleton Bench tests whether a model can infer an abstract mechanism from behavior rather than solve a rule statement with renamed nouns.
The renderer and solver are intentionally separate clients. Each can use a different model, API endpoint, key, and OpenAI-compatible API style, so model-pair heatmaps can be added without changing the skeleton generator.
The generator now creates an executable **hidden stochastic system**, runs controlled experiments against it, and builds a **behavior skeleton** containing only inputs, ordered events, and observed final outcomes. A renderer turns that evidence into a coherent natural-language world. The solver sees the rendered histories and predicts which outcome is more likely for a held-out event combination. It never sees the hidden rules or their probabilities.
## What the first version contains
The renderer and solver remain independent clients. Each can use a different model, endpoint, key, and OpenAI-compatible API style for future model-pair heatmaps.
Three finite skeleton families are included:
## Current task model
- `transitive_order`: a chain of directional relations;
- `membership_inheritance`: membership propagated through subset relations;
- `sequential_state_update`: ordered arithmetic state changes.
Every trial starts from the same empty state. Hidden rules can:
Every generated skeleton contains its facts, rules, query, permitted answer labels, deterministic `gold_answer`, and proof trace. The renderer receives a copy with `gold_answer` and `proof` removed.
- trigger on an event;
- test an input attribute or an internal state flag;
- fire with a hidden probability;
- add a record, permanently invalidate it, or set a flag used by a later event.
The full field grammar, predicate signatures, generator invariants, and renderer boundary are defined in [the Skeleton IR specification](docs/skeleton-language.md).
An invalidation is a persistent veto, so probabilistically triggered actions can conflict. Historical cases report aggregate `PRESENT`/`ABSENT` counts from repeated trials. The query uses a new composition of events and asks only which outcome is more likely—not for the exact probability.
`length` means logical complexity, not natural-language tokens. It is the number of proof-bearing facts or state updates. Presets are `short=2`, `medium=4`, and `long=6`; an integer from 1 through 20 is also accepted.
The local evaluator computes the true query probability by exact enumeration of all stochastic branches. Random simulation is used only to create the finite observations shown to the model. Both are reproducible from the task seed.
The complete hidden-system language, execution semantics, public behavior format, and non-leakage boundary are specified in [the behavioral IR specification](docs/skeleton-language.md).
`length` is structural complexity, not a natural-language token target. It is the number of distinct ordered query events and hidden event rules. Presets are `short=4`, `medium=6`, and `long=8`; explicit integers from 4 through 10 are accepted.
## Local use
Python 3.11 or newer is sufficient; the package has no runtime dependencies.
Generate three short skeletons without calling an LLM:
Generate three short, answer-bearing records without calling an LLM:
```bash
python3 -m logicbench generate --count 3 --length short --seed 20260922
@@ -40,36 +45,30 @@ export SOLVER_MODEL='your-solver-model'
python3 -m logicbench run tasks/short-smoke.json
```
Role-specific values override the shared fallback:
| Variable | Meaning |
| --- | --- |
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials; values may currently be identical |
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Required independent OpenAI-compatible endpoints; values may currently be identical |
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Required independent credentials; values may be identical |
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Required independent OpenAI-compatible endpoints |
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` (default) or `chat_completions` |
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required, independently selected model IDs |
Outputs are written under the task's `output_dir`:
Outputs under the task's `output_dir` are:
- `skeletons.jsonl` is written before any LLM call and includes gold answers;
- `results.jsonl` contains rendered problems, predictions, and per-case scores;
- `summary.json` contains aggregate accuracy and complexity metadata.
- `skeletons.jsonl`: full hidden programs, sampled behavior, exact probabilities, gold labels, and latent traces; written before any LLM call;
- `results.jsonl`: rendered problems, complete solver responses, parsed labels, gold probabilities, and scores;
- `summary.json`: aggregate accuracy and structural metadata.
## Gitea Actions one-shot test
The workflow `.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when someone manually clicks **Run workflow**. It runs five cases with `length: short` and has a 15-minute timeout.
`.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when manually started. It targets the `minecraft-and-affine-server` runner label, runs unit tests and five short live cases, prints the auditable records, and uploads `short-abstraction-evaluation`.
The run log prints every raw answer-bearing skeleton followed by each rendered question, the solver's complete text, its parsed `#### <answer>` label, the gold answer, and correctness. The same records and summary are retained in the `short-abstraction-evaluation` Actions artifact for inspection.
Create these repository Actions settings:
Configure repository Actions with:
1. Secrets `RENDERER_API_KEY` and `SOLVER_API_KEY`.
2. Variables `RENDERER_BASE_URL`, `RENDERER_MODEL`, and `RENDERER_API_STYLE`.
3. Variables `SOLVER_BASE_URL`, `SOLVER_MODEL`, and `SOLVER_API_STYLE`.
The workflow targets the existing `minecraft-and-affine-server` runner label. Verify that a runner advertising this label is online, then manually dispatch **Short abstraction evaluation** from the repository's Actions page.
For a current all-Qwen run, put the Model Studio key into both role-specific secrets and its API Host into both role-specific base-URL variables. The values may be identical today, but the two client boundaries remain explicit. Use `chat_completions` for both API-style variables. For example, a fast renderer and stronger solver can use `qwen3.8-flash` and `qwen3.8-max`; both currently support JSON Schema structured output. Do not hard-code keys or endpoints in the workflow.
For Qwen's OpenAI-compatible API, use the compatible-mode `/v1` endpoint and `chat_completions`. Keys, URLs, and models remain role-specific even when both roles currently use the same provider.
## Tests