Zayden-Jung/logic-skeleton-bench
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Archive first live benchmark as curated report

Browse Source
This commit is contained in:
Codex Bot
2026-09-22 23:22:57 +08:00
parent fab07f80b0
commit 3cf3db7b12
2 changed files with 172 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
reports/2026-09-22-run-90/README.md
+154
View File
@@ -0,0 +1,154 @@
# Short Smoke Evaluation — Gitea Actions Run #90
## Run metadata
| Field | Value |
| --- | --- |
| Gitea run | [#90](https://gitea.entroia.com/Zayden-Jung/logic-skeleton-bench/actions/runs/90) |
| Job | `98` on `minecraft-and-affine-server` |
| Source commit | `fab07f80b00cbbc1dd4cd02d1a9e14609f3f9ffa` |
| Task | `short-smoke` |
| Seed | `20260922` |
| Length preset | `short` |
| Structural complexity | `2` |
| Cases | `5` |
| Correct | `5` |
| Accuracy | `100%` |
| Renderer model | Not captured by run #90 |
| Solver model | Not captured by run #90 |
The protected API keys are deliberately absent. The model identifiers were repository variables but were not printed by this workflow, so they are recorded as unknown rather than guessed.
## Case results
### 1. `order-0000`
Original skeleton:
```json
{"id":"order-0000","family":"transitive_order","complexity":2,"entities":{"E0":"Entity","E1":"Entity","E2":"Entity"},"facts":[{"predicate":"LEFT_OF","arguments":["E0","E1"]},{"predicate":"LEFT_OF","arguments":["E1","E2"]}],"rules":["LEFT_OF is transitive","RIGHT_OF(x,y) iff LEFT_OF(y,x)"],"query":{"predicate":"RELATION","arguments":["E0","E2"]},"answer_options":["LEFT_OF","RIGHT_OF","UNKNOWN"],"gold_answer":"LEFT_OF","proof":["LEFT_OF(E0,E1)","LEFT_OF(E1,E2)"]}
```
Rendered problem:
> On a long straight shelf running east to west, three mysterious artifacts — the Gilded Skull, the Iron Compass, and the Emerald Vial — are arranged in a row. The Gilded Skull is located directly to the left of the Iron Compass. The Iron Compass is located directly to the left of the Emerald Vial. Given that 'to the left of' is a transitive relationship, and 'to the right of' is the inverse relationship, determine the positional relationship between the Gilded Skull and the Emerald Vial. Your answer must be exactly one of: LEFT_OF, RIGHT_OF, or UNKNOWN.
Solver output:
```text
The Gilded Skull is left of the Iron Compass, and the Iron Compass is left of the Emerald Vial. Since “left of” is transitive, the Gilded Skull is left of the Emerald Vial.
#### LEFT_OF
```
Parsed `LEFT_OF`; gold `LEFT_OF`; **correct**.
### 2. `set-0001`
Original skeleton:
```json
{"id":"set-0001","family":"membership_inheritance","complexity":2,"entities":{"E0":"Entity","S0":"Set","S1":"Set"},"facts":[{"predicate":"SUBSET_OF","arguments":["S0","S1"]},{"predicate":"MEMBER_OF","arguments":["E0","S0"]}],"rules":["MEMBER_OF(x,A) and SUBSET_OF(A,B) imply MEMBER_OF(x,B)"],"query":{"predicate":"MEMBER_OF","arguments":["E0","S1"]},"answer_options":["YES","NO","UNKNOWN"],"gold_answer":"YES","proof":["MEMBER_OF(E0,S0)","SUBSET_OF(S0,S1)"]}
```
Rendered problem:
> A certain creature is a member of the Forest Council. The Forest Council is a subset of the Grand Animal Assembly. Based on the rule that membership in a subset implies membership in the superset, does this creature belong to the Grand Animal Assembly? If the answer is YES, return 1; if NO, return 0; if UNKNOWN, return 2. Return the single numerical answer. No extra text.
Solver output:
```text
Membership in a subset implies membership in its superset.
#### YES
```
Parsed `YES`; gold `YES`; **correct**.
### 3. `state-0002`
Original skeleton:
```json
{"id":"state-0002","family":"sequential_state_update","complexity":2,"entities":{"E0":"Counter"},"facts":[{"predicate":"INITIAL_VALUE","arguments":["E0","7"]},{"predicate":"INCREMENT","arguments":["E0","1"]},{"predicate":"INCREMENT","arguments":["E0","1"]}],"rules":["Apply updates in the listed order"],"query":{"predicate":"FINAL_VALUE","arguments":["E0"]},"answer_options":["9","10","8"],"gold_answer":"9","proof":["INCREMENT(1)","INCREMENT(1)"]}
```
Rendered problem:
> A digital scoreboard in a fictional space station arena starts the match with a displayed score of 7. During the first quarter, the scoreboard registers a single point update, increasing the displayed number by 1. During the second quarter, it registers another single point update, increasing the displayed number by 1 again. Apply the updates in the order listed. What is the final number displayed on the scoreboard?
>
> Options: 9, 10, 8
>
> Return exactly one answer label from the options above, and nothing else.
Solver output:
```text
Starting at 7, add 1 and then add 1: 7 + 1 + 1 = 9.
#### 9
```
Parsed `9`; gold `9`; **correct**.
### 4. `order-0003`
Original skeleton:
```json
{"id":"order-0003","family":"transitive_order","complexity":2,"entities":{"E0":"Entity","E1":"Entity","E2":"Entity"},"facts":[{"predicate":"LEFT_OF","arguments":["E0","E1"]},{"predicate":"LEFT_OF","arguments":["E1","E2"]}],"rules":["LEFT_OF is transitive","RIGHT_OF(x,y) iff LEFT_OF(y,x)"],"query":{"predicate":"RELATION","arguments":["E0","E2"]},"answer_options":["LEFT_OF","RIGHT_OF","UNKNOWN"],"gold_answer":"LEFT_OF","proof":["LEFT_OF(E0,E1)","LEFT_OF(E1,E2)"]}
```
Rendered problem:
> Three fictional planets, Alpha, Beta, and Gamma, are arranged in a straight line in a solar system. It is observed that Alpha is to the LEFT_OF Beta, and Beta is to the LEFT_OF Gamma. Using the rule that LEFT_OF is transitive, and that RIGHT_OF(x, y) if and only if LEFT_OF(y, x), determine the relation between Alpha and Gamma. Provide the answer using exactly one of the following options: 'LEFT_OF', 'RIGHT_OF', or 'UNKNOWN'. Provide your final answer as a JSON object with the key 'answer'.
Solver output:
```text
Alpha is to the left of Beta, and Beta is to the left of Gamma, so by transitivity Alpha is to the left of Gamma.
#### LEFT_OF
```
Parsed `LEFT_OF`; gold `LEFT_OF`; **correct**.
### 5. `set-0004`
Original skeleton:
```json
{"id":"set-0004","family":"membership_inheritance","complexity":2,"entities":{"E0":"Entity","S0":"Set","S1":"Set"},"facts":[{"predicate":"MEMBER_OF","arguments":["E0","S0"]},{"predicate":"SUBSET_OF","arguments":["S0","S1"]}],"rules":["MEMBER_OF(x,A) and SUBSET_OF(A,B) imply MEMBER_OF(x,B)"],"query":{"predicate":"MEMBER_OF","arguments":["E0","S1"]},"answer_options":["YES","NO","UNKNOWN"],"gold_answer":"YES","proof":["MEMBER_OF(E0,S0)","SUBSET_OF(S0,S1)"]}
```
Rendered problem:
> In the magical archives of the Crystal Library, a special glowing book named `Asterisk` has been cataloged in the `Forbidden` section. Furthermore, all books in the `Forbidden` section are also part of the restricted `Rare` collection, meaning the `Forbidden` section is a subset of the `Rare` collection. Based on these facts, determine whether the `Asterisk` book is a member of the `Rare` collection. Is the statement 'The Asterisk book is in the Rare collection' true?
>
> Answer with exactly one of the following: YES, NO, or UNKNOWN.
Solver output:
```text
Since `Asterisk` is in the `Forbidden` section and every book in the `Forbidden` section is also in the `Rare` collection, `Asterisk` must be in the `Rare` collection.
#### YES
```
Parsed `YES`; gold `YES`; **correct**.
## Interpretation and anomalies
The solver answered all five rendered problems correctly. This is solver accuracy only; it does not establish that all renderings faithfully preserve their source skeletons.
Observed renderer issues:
1. `order-0000` strengthened `LEFT_OF` to “directly to the left,” which suggests adjacency even though adjacency is absent from the skeleton.
2. `set-0001` replaced the required labels with a numerical `1/0/2` response protocol. The solver still returned `YES` because the evaluator appended the permitted labels to its prompt.
3. `order-0003` requested a JSON object even though the benchmark solver protocol requires a final `#### <answer>` line.
These cases motivate a separate renderer-fidelity check. Until such a check exists, aggregate solver accuracy must not be interpreted as end-to-end benchmark validity.
## Raw artifact
The original `skeletons.jsonl`, `results.jsonl`, and `summary.json` remain attached to [Gitea Actions run #90](https://gitea.entroia.com/Zayden-Jung/logic-skeleton-bench/actions/runs/90) as `short-abstraction-evaluation`. They are deliberately not duplicated in Git.
reports/README.md
+18
View File
@@ -0,0 +1,18 @@
# Evaluation Records
This directory contains curated, immutable milestone reports rather than every benchmark run. Each committed report contains the human-readable evidence needed to understand the result:
- original answer-bearing skeletons;
- rendered problems;
- complete solver outputs and parsed answers;
- scores, run metadata, and qualitative anomalies.
Raw JSONL from ordinary runs belongs in the Gitea Actions artifact, not Git. Promote a run into this directory only when it establishes a baseline, changes the benchmark protocol, exposes an important failure mode, or is explicitly selected for long-term comparison. This keeps repository growth proportional to meaningful experimental milestones instead of CI volume.
## Runs
| Date | Run | Task | Cases | Accuracy | Commit |
| --- | --- | --- | ---: | ---: | --- |
| 2026-09-22 | [Gitea Actions #90](2026-09-22-run-90/README.md) | `short-smoke` | 5 | 100% | `fab07f8` |
Model identifiers were not emitted by run #90 and therefore are intentionally recorded as unknown rather than inferred. Future artifacts should capture the renderer and solver model IDs as non-secret metadata.