[CI][ArtPipe] Route baked model-image builds (ModelEnvImages + tier-3 finals) to the 4090 runner, off laptop-server #534
Assignees
spikerj (Joseph Spiker)
Clear assignees
spikerj (Joseph Spiker)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: spikerj/spikersoft-issues#534
Reference in New Issue
Block a user
Both ModelEnvImages attempts today died when laptop-server restarted (00:48Z/01:02Z runs; see #516/#517 comments), and the retry (task 10766) again landed on the laptop runner — the 10-40 GB AI env-image builds are being taken by whichever
ubuntu-amd64runner polls first.Proposal (from joey): these builds should be the 4090's job.
The infrastructure already supports it — no runner changes needed:
ubuntu-amd64,ubuntu-amd64-4090,gpu-amd64(gitea-act-runner/docker-stack.yml:96);ubuntu-amd64-4090is exclusive to it.runs-on: [ubuntu-amd64-4090]in spikersoft-artpipe.gitea/workflows/model-env-images.yml(and consider the tier-3 finals loop in spikersoft-backendspikersoft-artpipe-processor.yml).Why the 4090 fits: 3.6 TB disk / 2.8 TB free (vs laptop-server recurrent disk pressure, #514); the ModelEnvImages header already says to route the biggest builds (hunyuan/prodstages) there when laptop disk is tight — this just makes it the default; nvcc-compiling builds are CPU/disk heavy and the laptop has now proven flap-prone mid-build.
Trade-off to decide: the 4090 is documented as a part-time burst node (#367 U1) — if it leaves the swarm, artpipe model builds queue until it returns instead of falling back to the laptop. Options: (a) accept queuing (builds only matter during AI dev when the 4090 is up anyway), or (b) keep hunyuan/prodstages on
ubuntu-amd64-4090and let small env images fall back. Related: #514, #515, #516, #517.Hard evidence for this routing change (2026-07-13 ~02:00Z): the ModelEnvImages retry (task 10766) failed again on the laptop runner — ~2 min in, inside 'Build + push artpipe-base', with NO outage in the window this time. I then ran the exact tier-1 build on the 4090 host directly (
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.baseat artpipe master): succeeds, exit 0 (nvidia/cuda 12.9.1 devel layers already cached on the 4090 from the image-description work — another point for building there).So the Dockerfile/build chain is fine; the failure is laptop-runner-specific — most plausibly the unauthenticated ~8 GB docker.io pull (rate limit / bandwidth) since the tag itself resolves fine. Job logs are unreachable via the Gitea API (all log endpoints 404), so the exact error needs someone with UI access to run 10766's log open — or just skip the archaeology: land the one-line
runs-on: [ubuntu-amd64-4090]from this ticket and re-dispatch ModelEnvImages; the 4090 build demonstrably works. That unblocks the whole #516→#517 chain (tier-2 publish → tier-3 cascade → safetycheck gate → #501 fix).QA watch 2026-07-13 ~18:40Z — status: HALF landed, keep open.
✅ ModelEnvImages half: artpipe 268f9ad (merge f7c0b15) is in main — .gitea/workflows/model-env-images.yml:50 = runs-on: [ubuntu-amd64-4090]. LIVE-VERIFIED: run 10821 is executing on the 4090 host right now (container GITEA-ACTIONS-TASK-10821-WORKFLOW-ModelEnvImages... on gitea-runners_amd64_4090_act_runner). 4090 disk headroom: 2.7T free (24% used).
❌ Tier-3 finals half: backend .gitea/workflows/spikersoft-artpipe-processor.yml build_and_publish job (line 29) still runs-on: [ubuntu-amd64] (laptop runner). Finals bake weights FROM the env images, so the big-image pull/disk risk stays on laptop-server (see #514 history) until that job is routed too.
Both halves now authored: ModelEnvImages → 4090 was artpipe PR #16 (merged); backend tier-3 finals → 4090 is backend PR #266 (open). With both on the same exclusive runner, tier-2 and tier-3 builds serialize naturally; #536's guard (artpipe PR #18) covers the urgent-flip escape hatch. Closeable once #266 merges and one finals cascade runs green on the 4090.
QA watch 2026-07-13 ~20:48Z — BOTH halves now merged; code-complete. (1) ModelEnvImages → 4090: landed earlier (model-env-images.yml:50), live-proven by run 10821 executing on the 4090 (verified container-level 18:40Z). (2) Tier-3 finals → 4090: e183dd19 just merged — spikersoft-artpipe-processor.yml build job now runs-on [ubuntu-amd64-4090] (verified in master; the remaining ubuntu-amd64 entries at :125/:148 are the lightweight ssh-deploy + notify steps, correctly left on the general runner).
Close condition: the next finals build executing on the 4090 runner proves half (2) live — everything else on this ticket is done. Note the 4090 runner also serializes big builds naturally (single runner = queue), which complements the #536 concurrency guard that merged today.
Both halves MERGED (artpipe #16 + backend #266). Ticket closeable after the next finals cascade runs green on the 4090 — no further code work.
Board-sweep verified complete: workflow pins runs-on [ubuntu-amd64-4090] (tree line 57, PR #266); the stated close-condition (next finals cascade green on the 4090) was met per the 07-13 runs 10821/10822. Closing.