Capture full solver output in five-case evaluation
This commit is contained in:
@@ -57,9 +57,9 @@ Outputs are written under the task's `output_dir`:
|
||||
|
||||
## Gitea Actions one-shot test
|
||||
|
||||
The workflow `.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when someone manually clicks **Run workflow**. It runs three cases with `length: short` and has a 15-minute timeout.
|
||||
The workflow `.gitea/workflows/short-eval.yml` has only `workflow_dispatch`, so it consumes model quota only when someone manually clicks **Run workflow**. It runs five cases with `length: short` and has a 15-minute timeout.
|
||||
|
||||
The run log prints the rendered questions and scores. The same records, original answer-bearing skeletons, and summary are retained in the `short-abstraction-evaluation` Actions artifact for inspection.
|
||||
The run log prints every raw answer-bearing skeleton followed by each rendered question, the solver's complete text, its parsed `#### <answer>` label, the gold answer, and correctness. The same records and summary are retained in the `short-abstraction-evaluation` Actions artifact for inspection.
|
||||
|
||||
Create these repository Actions settings:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user