This commit is contained in:
@@ -5,7 +5,7 @@ on:
|
||||
|
||||
jobs:
|
||||
extractor:
|
||||
runs-on: gpu
|
||||
runs-on: ces-gpu-formal
|
||||
timeout-minutes: 1440
|
||||
env:
|
||||
TOKENIZERS_PARALLELISM: "false"
|
||||
|
||||
@@ -153,7 +153,7 @@ CI 的单 epoch smoke run 会执行 benchmark 来验证整条评测管线,但
|
||||
|
||||
1. 调用 AutoDL Pro API 创建一张 `PRO6000-96G`;
|
||||
2. 等待 SSH 就绪,将模型数据和经过 SHA-256 固定的 `act_runner` 传入实例;
|
||||
3. 注册带 `ubuntu-latest:host`、`ces-ci:host`、`gpu:host` 标签的一次性 runner;
|
||||
3. 注册只带 `ces-gpu-formal:host` 专用标签的一次性 runner;
|
||||
4. 可选地通过 Gitea API dispatch `gpu-train.yaml`,也可保留为手动触发;
|
||||
5. 等待 runner 完成一项任务并退出,将 checkpoint、benchmark 和 runner 日志回传阿里云;
|
||||
6. 无论成功或失败都调用 API 关机;默认不释放实例,避免未经确认删除系统盘。
|
||||
@@ -187,3 +187,7 @@ python3 ops/autodl_pro_controller.py run \
|
||||
`run` 会真实创建按量计费实例。`release_after_run` 默认是 `false`;只有确认回传产物完整并愿意
|
||||
删除该 Pro 实例时才应改为 `true`。控制器只负责调度与数据搬运,训练步骤仍完全由
|
||||
`.gitea/workflows/gpu-train.yaml` 定义。
|
||||
|
||||
正式 Runner 故意不注册 `ubuntu-latest`、`ces-ci` 或通用 `gpu` 标签。ephemeral runner 只会
|
||||
执行一个任务;使用通用标签可能让它先领取普通 CI 或 smoke job,随后退出,导致正式训练继续
|
||||
排队。普通测试 Runner 与临时正式训练 Runner 因而使用互不重叠的标签。
|
||||
|
||||
@@ -24,9 +24,7 @@
|
||||
"registration_token_file": "/root/.config/ces-controller/secrets/gitea-runner-token",
|
||||
"name_prefix": "ces-autodl-pro6000",
|
||||
"labels": [
|
||||
"ubuntu-latest:host",
|
||||
"ces-ci:host",
|
||||
"gpu:host"
|
||||
"ces-gpu-formal:host"
|
||||
],
|
||||
"job_timeout": "24h",
|
||||
"cache_path": "/var/cache/ces-controller/act_runner-0.2.13-linux-amd64",
|
||||
|
||||
@@ -40,11 +40,9 @@ def test_snapshot_to_ssh_target(tmp_path):
|
||||
|
||||
|
||||
def test_runner_config_disables_docker_and_keeps_host_labels():
|
||||
rendered = controller.runner_config(
|
||||
["ubuntu-latest:host", "ces-ci:host", "gpu:host"], "24h"
|
||||
)
|
||||
rendered = controller.runner_config(["ces-gpu-formal:host"], "24h")
|
||||
assert 'docker_host: "-"' in rendered
|
||||
assert '"gpu:host"' in rendered
|
||||
assert '"ces-gpu-formal:host"' in rendered
|
||||
assert "capacity: 1" in rendered
|
||||
|
||||
|
||||
@@ -61,9 +59,5 @@ def test_secret_permissions_are_enforced(tmp_path):
|
||||
def test_example_config_is_valid_json():
|
||||
path = Path(__file__).parents[1] / "ops" / "autodl-pro.example.json"
|
||||
config = json.loads(path.read_text(encoding="utf-8"))
|
||||
assert config["runner"]["labels"] == [
|
||||
"ubuntu-latest:host",
|
||||
"ces-ci:host",
|
||||
"gpu:host",
|
||||
]
|
||||
assert config["runner"]["labels"] == ["ces-gpu-formal:host"]
|
||||
assert stat.S_IMODE(MODULE_PATH.stat().st_mode) in {0o644, 0o755}
|
||||
|
||||
Reference in New Issue
Block a user