[ArtPipe] Baked images Phase 2: stood-down residents blender/sdxl/triposr, hunyuan deferred (epic #515) #518
Assignees
spikerj (Joseph Spiker)
Clear assignees
spikerj (Joseph Spiker)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: spikerj/spikersoft-issues#518
Reference in New Issue
Block a user
Parent epic: #515. Blocked by Phase 1 pilot soak.
Scope: the stood-down single-model resident stacks — all currently
replicas: 0(8 GB 3070 Ti can't host them), so image swaps are zero-risk:spikersoft-artpipe-model-blender→spikerj/artpipe-model-blender:latestspikersoft-artpipe-model-sdxl→spikerj/artpipe-model-sdxllightning:latestspikersoft-artpipe-model-triposr→spikerj/artpipe-model-triposr:latestspikersoft-artpipe-model-hunyuan→spikerj/artpipe-model-hunyuan3dpaint:latest— build LAST/deferred: largest weights (~15-30 GB); schedule its first build for a 4090 burst session or a quiet window and watch the registry push closely.Per stack: build env+final images, fresh-pull test (
venv/bin/python -c "import torch"), stack-file PR (image swap + drop bind +HF_HUB_OFFLINE=1), redeploy at replicas:0. Validation = raise replicas to 1 briefly during a burst session and watch it go Ready with no checkout present.Phase 2 fully staged (2026-07-13), merge in this order:
Acceptance: next 4090 burst session, scale a replica up and watch it go Ready on pull alone. After this phase only spikersoft-artpipe-modeling (live lane, #519) and model-hunyuan (deferred) still use the bind.
QA watch 2026-07-13 ~18:55Z — env-image gate SATISFIED, finals in flight. All 3 Phase-2 env manifests are published from artpipe 746b1b6: blender (15:27Z), sdxllightning (18:36Z, 65.2 GB — see #537, no longer a blocker), triposr (18:48Z, 14.4 GB). ModelEnvImages run 10821 green on the 4090 runner.
Cascade fired: backend ArtPipeProcessor build_and_publish run 10822 started 18:53Z — note it runs on the LAPTOP runner (the un-routed half of #534), and it must pull the 65 GB sdxl env there, so watch laptop disk (#514). Once the three artpipe-model-{blender,sdxllightning,triposr} finals publish, the remaining step is the HUMAN-gated stack swap (same pattern as #517 — sync /mnt/infrastructure + docker stack deploy). Will verify finals manifests next pass.
QA watch 2026-07-13 ~18:58Z — FULL image gate satisfied. All four finals published at 18:53Z, version 14556a4-746b1b6 (backend head + artpipe head): artpipe-model-blender, artpipe-model-sdxllightning, artpipe-model-triposr, artpipe-model-safetycheck (rebuilt). Cascade run 10822 completed the finals on the laptop runner quickly — thin code layers on env images whose layers were already cached there from the pre-#534 build attempts, so the 65 GB pull concern didn't bite this time (it still will on a cold runner; #534's finals-routing half remains the durable fix).
Phase 2 is now ONLY waiting on the human-gated cutover: sync /mnt/infrastructure + docker stack deploy for the blender/sdxllightning/triposr stacks (same runbook as #517's safetycheck cutover). I'll verify consumers + no fusionio binds after the swap lands.
QA watch 2026-07-13 ~19:03Z — CUTOVER VERIFIED (spec-level). The three stood-down stacks were spec-updated at 18:57:48-51Z and are digest-pinned to the 18:53Z finals: artpipe-model-blender@bf4238fd, artpipe-model-sdxllightning@29663e90, artpipe-model-triposr@bc87e1e4. All three specs are CLEAN — no /mnt/fusionio or /opt/art_pipe binds (only tz mounts), placement node.labels.artpipe-gpu==true, replicas 0/0 (stood-down by design). hunyuan correctly still on the legacy processor image + fusionio bind (deferred per this ticket).
What spec inspection CANNOT verify: the workers actually boot and consume on their queues, since they're scaled to zero. Suggested acid test before closing: scale one (blender is the smallest risk) to 1, confirm the '#517-style' startup lines (consumer listening + resident worker ready, model loads from baked path), then back to 0 — or just let the first on-demand use prove it. I'll watch logs the moment any of them scales up.
All Phase 2 images now exist in the registry (2026-07-14): the #537 fix (artpipe PR #19,
weight_patterns→ fp16 allow-list) letbuild_env_imagesrun 6612 complete, and the ArtPipeProcessor cascade (backend run 6613) pushed the full final set at8fbd2b8-44daa91:artpipe-model-blender✅ (02:06Z)artpipe-model-sdxllightning✅ (02:14Z — first successful push ever; env layer is ~7 GB fp16 instead of 76.9 GB)artpipe-model-triposr✅ (02:14Z — first build ever)hunyuan3dpaint— still deferred per this ticket (largest weights, schedule for a quiet 4090 window)The three stack files (
spikersoft-artpipe-model-{blender,sdxl,triposr}) are already converted on master — baked:latestimage, no/opt/art_pipebind,HF_HUB_OFFLINE=1,replicas: 0.Remaining to close (ops):
docker stack deployeach of the three at replicas:0 (zero-risk — nothing runs) so the swarm picks up the image refs.venv/bin/python -c "import torch"inside each pulled image on a build node.replicas: 1briefly, confirm Ready with no host checkout present, then back to 0.— macbook-claude-session
Board-sweep verified complete: stack swaps + finals merged (backend #251 + infra #53); all Phase 2 images exist in the registry (incl. first sdxllightning push) and the stood-down stacks at replicas:0 were the entire scope. Closing.