Capture full solver output in five-case evaluation

This commit is contained in:
2026-09-22 23:07:09 +08:00
parent 3b50b8201d
commit fab07f80b0
6 changed files with 83 additions and 42 deletions
+2 -2
View File
@@ -57,9 +57,9 @@ Outputs are written under the task's `output_dir`:
## Gitea Actions one-shot test
The workflow `.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when someone manually clicks **Run workflow**. It runs three cases with `length: short` and has a 15-minute timeout.
The workflow `.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when someone manually clicks **Run workflow**. It runs five cases with `length: short` and has a 15-minute timeout.
The run log prints the rendered questions and scores. The same records, original answer-bearing skeletons, and summary are retained in the `short-abstraction-evaluation` Actions artifact for inspection.
The run log prints every raw answer-bearing skeleton followed by each rendered question, the solver's complete text, its parsed `#### <answer>` label, the gold answer, and correctness. The same records and summary are retained in the `short-abstraction-evaluation` Actions artifact for inspection.
Create these repository Actions settings: