Files
logic-skeleton-bench/reports/2026-09-22-run-90

Short Smoke Evaluation — Gitea Actions Run #90

Run metadata

Field Value
Gitea run #90
Job 98 on minecraft-and-affine-server
Source commit fab07f80b00cbbc1dd4cd02d1a9e14609f3f9ffa
Task short-smoke
Seed 20260922
Length preset short
Structural complexity 2
Cases 5
Correct 5
Accuracy 100%
Renderer model Not captured by run #90
Solver model Not captured by run #90

The protected API keys are deliberately absent. The model identifiers were repository variables but were not printed by this workflow, so they are recorded as unknown rather than guessed.

Case results

1. order-0000

Original skeleton:

{"id":"order-0000","family":"transitive_order","complexity":2,"entities":{"E0":"Entity","E1":"Entity","E2":"Entity"},"facts":[{"predicate":"LEFT_OF","arguments":["E0","E1"]},{"predicate":"LEFT_OF","arguments":["E1","E2"]}],"rules":["LEFT_OF is transitive","RIGHT_OF(x,y) iff LEFT_OF(y,x)"],"query":{"predicate":"RELATION","arguments":["E0","E2"]},"answer_options":["LEFT_OF","RIGHT_OF","UNKNOWN"],"gold_answer":"LEFT_OF","proof":["LEFT_OF(E0,E1)","LEFT_OF(E1,E2)"]}

Rendered problem:

On a long straight shelf running east to west, three mysterious artifacts — the Gilded Skull, the Iron Compass, and the Emerald Vial — are arranged in a row. The Gilded Skull is located directly to the left of the Iron Compass. The Iron Compass is located directly to the left of the Emerald Vial. Given that 'to the left of' is a transitive relationship, and 'to the right of' is the inverse relationship, determine the positional relationship between the Gilded Skull and the Emerald Vial. Your answer must be exactly one of: LEFT_OF, RIGHT_OF, or UNKNOWN.

Solver output:

The Gilded Skull is left of the Iron Compass, and the Iron Compass is left of the Emerald Vial. Since “left of” is transitive, the Gilded Skull is left of the Emerald Vial.

#### LEFT_OF

Parsed LEFT_OF; gold LEFT_OF; correct.

2. set-0001

Original skeleton:

{"id":"set-0001","family":"membership_inheritance","complexity":2,"entities":{"E0":"Entity","S0":"Set","S1":"Set"},"facts":[{"predicate":"SUBSET_OF","arguments":["S0","S1"]},{"predicate":"MEMBER_OF","arguments":["E0","S0"]}],"rules":["MEMBER_OF(x,A) and SUBSET_OF(A,B) imply MEMBER_OF(x,B)"],"query":{"predicate":"MEMBER_OF","arguments":["E0","S1"]},"answer_options":["YES","NO","UNKNOWN"],"gold_answer":"YES","proof":["MEMBER_OF(E0,S0)","SUBSET_OF(S0,S1)"]}

Rendered problem:

A certain creature is a member of the Forest Council. The Forest Council is a subset of the Grand Animal Assembly. Based on the rule that membership in a subset implies membership in the superset, does this creature belong to the Grand Animal Assembly? If the answer is YES, return 1; if NO, return 0; if UNKNOWN, return 2. Return the single numerical answer. No extra text.

Solver output:

Membership in a subset implies membership in its superset.

#### YES

Parsed YES; gold YES; correct.

3. state-0002

Original skeleton:

{"id":"state-0002","family":"sequential_state_update","complexity":2,"entities":{"E0":"Counter"},"facts":[{"predicate":"INITIAL_VALUE","arguments":["E0","7"]},{"predicate":"INCREMENT","arguments":["E0","1"]},{"predicate":"INCREMENT","arguments":["E0","1"]}],"rules":["Apply updates in the listed order"],"query":{"predicate":"FINAL_VALUE","arguments":["E0"]},"answer_options":["9","10","8"],"gold_answer":"9","proof":["INCREMENT(1)","INCREMENT(1)"]}

Rendered problem:

A digital scoreboard in a fictional space station arena starts the match with a displayed score of 7. During the first quarter, the scoreboard registers a single point update, increasing the displayed number by 1. During the second quarter, it registers another single point update, increasing the displayed number by 1 again. Apply the updates in the order listed. What is the final number displayed on the scoreboard?

Options: 9, 10, 8

Return exactly one answer label from the options above, and nothing else.

Solver output:

Starting at 7, add 1 and then add 1: 7 + 1 + 1 = 9.

#### 9

Parsed 9; gold 9; correct.

4. order-0003

Original skeleton:

{"id":"order-0003","family":"transitive_order","complexity":2,"entities":{"E0":"Entity","E1":"Entity","E2":"Entity"},"facts":[{"predicate":"LEFT_OF","arguments":["E0","E1"]},{"predicate":"LEFT_OF","arguments":["E1","E2"]}],"rules":["LEFT_OF is transitive","RIGHT_OF(x,y) iff LEFT_OF(y,x)"],"query":{"predicate":"RELATION","arguments":["E0","E2"]},"answer_options":["LEFT_OF","RIGHT_OF","UNKNOWN"],"gold_answer":"LEFT_OF","proof":["LEFT_OF(E0,E1)","LEFT_OF(E1,E2)"]}

Rendered problem:

Three fictional planets, Alpha, Beta, and Gamma, are arranged in a straight line in a solar system. It is observed that Alpha is to the LEFT_OF Beta, and Beta is to the LEFT_OF Gamma. Using the rule that LEFT_OF is transitive, and that RIGHT_OF(x, y) if and only if LEFT_OF(y, x), determine the relation between Alpha and Gamma. Provide the answer using exactly one of the following options: 'LEFT_OF', 'RIGHT_OF', or 'UNKNOWN'. Provide your final answer as a JSON object with the key 'answer'.

Solver output:

Alpha is to the left of Beta, and Beta is to the left of Gamma, so by transitivity Alpha is to the left of Gamma.

#### LEFT_OF

Parsed LEFT_OF; gold LEFT_OF; correct.

5. set-0004

Original skeleton:

{"id":"set-0004","family":"membership_inheritance","complexity":2,"entities":{"E0":"Entity","S0":"Set","S1":"Set"},"facts":[{"predicate":"MEMBER_OF","arguments":["E0","S0"]},{"predicate":"SUBSET_OF","arguments":["S0","S1"]}],"rules":["MEMBER_OF(x,A) and SUBSET_OF(A,B) imply MEMBER_OF(x,B)"],"query":{"predicate":"MEMBER_OF","arguments":["E0","S1"]},"answer_options":["YES","NO","UNKNOWN"],"gold_answer":"YES","proof":["MEMBER_OF(E0,S0)","SUBSET_OF(S0,S1)"]}

Rendered problem:

In the magical archives of the Crystal Library, a special glowing book named Asterisk has been cataloged in the Forbidden section. Furthermore, all books in the Forbidden section are also part of the restricted Rare collection, meaning the Forbidden section is a subset of the Rare collection. Based on these facts, determine whether the Asterisk book is a member of the Rare collection. Is the statement 'The Asterisk book is in the Rare collection' true?

Answer with exactly one of the following: YES, NO, or UNKNOWN.

Solver output:

Since `Asterisk` is in the `Forbidden` section and every book in the `Forbidden` section is also in the `Rare` collection, `Asterisk` must be in the `Rare` collection.

#### YES

Parsed YES; gold YES; correct.

Interpretation and anomalies

The solver answered all five rendered problems correctly. This is solver accuracy only; it does not establish that all renderings faithfully preserve their source skeletons.

Observed renderer issues:

  1. order-0000 strengthened LEFT_OF to “directly to the left,” which suggests adjacency even though adjacency is absent from the skeleton.
  2. set-0001 replaced the required labels with a numerical 1/0/2 response protocol. The solver still returned YES because the evaluator appended the permitted labels to its prompt.
  3. order-0003 requested a JSON object even though the benchmark solver protocol requires a final #### <answer> line.

These cases motivate a separate renderer-fidelity check. Until such a check exists, aggregate solver accuracy must not be interpreted as end-to-end benchmark validity.

Raw artifact

The original skeletons.jsonl, results.jsonl, and summary.json remain attached to Gitea Actions run #90 as short-abstraction-evaluation. They are deliberately not duplicated in Git.