Restore portable evaluation workflow for configured runner
This commit is contained in:
@@ -31,7 +31,8 @@ python3 -m logicbench generate --count 3 --length short --seed 20260922
|
||||
Run the checked-in live task:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY='...'
|
||||
export LLM_API_KEY='...'
|
||||
export LLM_BASE_URL='your-provider-compatible-base-url'
|
||||
export RENDERER_MODEL='your-renderer-model'
|
||||
export SOLVER_MODEL='your-solver-model'
|
||||
python3 -m logicbench run tasks/short-smoke.json
|
||||
@@ -41,8 +42,10 @@ Role-specific values override the shared fallback:
|
||||
|
||||
| Variable | Meaning |
|
||||
| --- | --- |
|
||||
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Optional separate keys; each falls back to `OPENAI_API_KEY` |
|
||||
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Optional endpoints; each falls back to `OPENAI_BASE_URL`, then `https://api.openai.com/v1` |
|
||||
| `LLM_API_KEY` | Optional shared key when both roles use the same account/provider |
|
||||
| `RENDERER_API_KEY`, `SOLVER_API_KEY` | Optional role-specific keys; each takes precedence over `LLM_API_KEY` |
|
||||
| `LLM_BASE_URL` | Optional shared OpenAI-compatible endpoint |
|
||||
| `RENDERER_BASE_URL`, `SOLVER_BASE_URL` | Optional role-specific endpoints; each takes precedence over `LLM_BASE_URL` |
|
||||
| `RENDERER_API_STYLE`, `SOLVER_API_STYLE` | `responses` (default) or `chat_completions` |
|
||||
| `RENDERER_MODEL`, `SOLVER_MODEL` | Required, independently selected model IDs |
|
||||
|
||||
@@ -60,25 +63,14 @@ The run log prints the rendered questions and scores. The same records, original
|
||||
|
||||
Create these repository Actions settings:
|
||||
|
||||
1. Secret `OPENAI_API_KEY`, or separate secrets `RENDERER_API_KEY` and `SOLVER_API_KEY`.
|
||||
1. Secret `LLM_API_KEY` when both roles share credentials, or role-specific secrets `RENDERER_API_KEY` and `SOLVER_API_KEY` when they do not.
|
||||
2. Variables `RENDERER_MODEL` and `SOLVER_MODEL`.
|
||||
3. Optional variables `RENDERER_BASE_URL`, `SOLVER_BASE_URL`, `RENDERER_API_STYLE`, and `SOLVER_API_STYLE`. Leave base URLs empty for OpenAI; use `responses` as the API style unless a compatible provider only supports chat completions.
|
||||
3. Variable `LLM_BASE_URL` for a shared provider endpoint, or role-specific variables `RENDERER_BASE_URL` and `SOLVER_BASE_URL` for different providers.
|
||||
4. Optional variables `RENDERER_API_STYLE` and `SOLVER_API_STYLE`, each set to `responses` or `chat_completions` according to that provider/model.
|
||||
|
||||
The workflow targets `runs-on: logic-bench`. Add this label to the existing `act_runner` using one of these methods:
|
||||
The workflow targets the existing `minecraft-and-affine-server` runner label. Verify that a runner advertising this label is online, then manually dispatch **Short abstraction evaluation** from the repository's Actions page.
|
||||
|
||||
- In the Gitea UI, open the runner settings, edit its labels, and add `logic-bench:docker://docker.gitea.com/runner-images:ubuntu-latest`.
|
||||
- Or add the same string to `runner.labels` in the runner's `config.yaml`, preserving existing labels, then restart `act_runner` with that config.
|
||||
- Or start the daemon with `--labels` / `GITEA_RUNNER_LABELS`; this has higher precedence than the config file and must include every label the runner should continue advertising.
|
||||
|
||||
Example config fragment:
|
||||
|
||||
```yaml
|
||||
runner:
|
||||
labels:
|
||||
- "logic-bench:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
```
|
||||
|
||||
After restarting the runner, verify that `logic-bench` appears in its Gitea settings, then manually dispatch **Short abstraction evaluation** from the repository's Actions page.
|
||||
For a current all-Qwen run, create only the shared `LLM_API_KEY` secret with the Model Studio key. Set `LLM_BASE_URL` to the API Host shown for that key and region, set the two model variables independently, and use `chat_completions` for both API-style variables. For example, a fast renderer and stronger solver can use `qwen3.8-flash` and `qwen3.8-max`; both currently support JSON Schema structured output. Do not hard-code the DashScope key or endpoint in the workflow.
|
||||
|
||||
## Tests
|
||||
|
||||
|
||||
Reference in New Issue
Block a user