[ArtStudio] Model-resident workers — one always-loaded model per service instance, task queues per model, coordinator-arbitrated, scale by replicas #368

New Issue

2026-07-04T23:48:25Z

spikerj commented

2026-07-04 23:48:25 +00:00

Architecture directive (Joey, 2026-07-05), epic #346: port ProArt in the spirit of Eric's original — per-model isolation — fused with our stack: event-driven, horizontally scalable, one model per service instance, always loaded for its purpose, its tasks handled by queue/message via the coordinator. When we need to scale out: 20 sentence-encoder instances, 20 TRELLIS instances, all consuming work as workflows, swappable by config.

Gap vs today

ArtPipeProcessor spawns a cold subprocess per job (model loads → one job → dies): SDXL pays ~6 s reload/job, TrellisMac ~80 s. Flagged as the top latency lever since #357.
Services are split per stage (artpipe-modeling/imaging/rigging), not per model; a stage instance can run any of its stage's models, so nothing is resident.
Coordinator leases VRAM per job (churn); a resident model needs a long-lived lease.

Target shape

art_pipe: persistent worker mode — worker.py --serve: load the model once, then consume newline-delimited job requests over the same NDJSON protocol (multi-job loop; health ping; clean shutdown; OOM/crash → exit for supervisor restart). Pure extension of the existing protocol — the original's per-model venv isolation unchanged.
.NET: resident-model service mode — ArtPipe:ResidentModel=<ModelDir> makes an instance bind queue art.model.<model>.tasks, hold ONE warm subprocess + ONE long-lived coordinator lease (new lease kind resident), and consume tasks message-by-message. Stage orchestration publishes stage work to the model queue the stage plan resolves to (stage→model mapping stays config). Per-stage mode remains for dev/small deployments (the Mac runs today's shape).
Coordinator: resident leases + per-node placement (#367 U2 folds in here); capacity math = which resident sets fit where; day-to-day scaling = replica counts + capacity config, zero code.
Swarm: one stack template per resident model service (artpipe-model-<name>), replicas as the scaling knob — 20 encoders + 20 TRELLIS is replicas: 20 twice, coordinator arbitrating.
Swappable: new/changed model = manifest + config + service entry; no orchestrator changes.

Phasing

R1 art_pipe --serve persistent worker loop (protocol extension + tests, validated warm-latency numbers on the M5) → R2 .NET resident mode + resident leases + model queues → R3 stage orchestrator publishes to model queues (per-stage mode kept as fallback) → R4 swarm stack template + first two resident services (encoder, TRELLIS) → R5 fold #367 U2 placement into resident-set planning.

Related: #367 (capacity registry U1 merged — this consumes it), #348 topology comments (per-stage split superseded for prod scale-out), #357 (warm-worker flag), #358 (TrellisMac reload cost).

Architecture directive (Joey, 2026-07-05), epic #346: port ProArt **in the spirit of Eric's original** — per-model isolation — fused with our stack: event-driven, horizontally scalable, **one model per service instance, always loaded for its purpose**, its tasks handled by queue/message via the coordinator. When we need to scale out: 20 sentence-encoder instances, 20 TRELLIS instances, all consuming work as workflows, swappable by config. ## Gap vs today - `ArtPipeProcessor` spawns a **cold subprocess per job** (model loads → one job → dies): SDXL pays ~6 s reload/job, TrellisMac ~80 s. Flagged as the top latency lever since #357. - Services are split per **stage** (artpipe-modeling/imaging/rigging), not per **model**; a stage instance can run any of its stage's models, so nothing is resident. - Coordinator leases VRAM **per job** (churn); a resident model needs a **long-lived lease**. ## Target shape - **art_pipe: persistent worker mode** — `worker.py --serve`: load the model once, then consume newline-delimited job requests over the same NDJSON protocol (multi-job loop; health ping; clean shutdown; OOM/crash → exit for supervisor restart). Pure extension of the existing protocol — the original's per-model venv isolation unchanged. - **.NET: resident-model service mode** — `ArtPipe:ResidentModel=<ModelDir>` makes an instance bind queue `art.model.<model>.tasks`, hold ONE warm subprocess + ONE long-lived coordinator lease (new lease kind `resident`), and consume tasks message-by-message. Stage orchestration publishes stage work to the model queue the stage plan resolves to (stage→model mapping stays config). Per-stage mode remains for dev/small deployments (the Mac runs today's shape). - **Coordinator**: resident leases + per-node placement (#367 U2 folds in here); capacity math = which resident sets fit where; day-to-day scaling = replica counts + capacity config, zero code. - **Swarm**: one stack template per resident model service (`artpipe-model-<name>`), replicas as the scaling knob — 20 encoders + 20 TRELLIS is `replicas: 20` twice, coordinator arbitrating. - **Swappable**: new/changed model = manifest + config + service entry; no orchestrator changes. ## Phasing R1 art_pipe `--serve` persistent worker loop (protocol extension + tests, validated warm-latency numbers on the M5) → R2 .NET resident mode + `resident` leases + model queues → R3 stage orchestrator publishes to model queues (per-stage mode kept as fallback) → R4 swarm stack template + first two resident services (encoder, TRELLIS) → R5 fold #367 U2 placement into resident-set planning. Related: #367 (capacity registry U1 merged — this consumes it), #348 topology comments (per-stage split superseded for prod scale-out), #357 (warm-worker flag), #358 (TrellisMac reload cost).

spikerj added the enhancement label 2026-07-04 23:48:25 +00:00

spikerj commented

2026-07-05 00:44:49 +00:00

R1 merged (spikersoft-artpipe PR #2): worker.py --serve persistent mode — byte-compatible protocol, ready/ping/shutdown lifecycle, fatal-exit semantics, warmup() hooks, PersistentWorkerHandle. Measured on the M5: sdxl_lightning 4.2 s/job warm (from 10–20 s cold), safety_check 20–70 ms (from ~2 s, byte-identical scores). TRELLIS's ~80 s prize needs #358's port half (wrap backends spawn external processes — documented). R2 (.NET resident-model service + resident leases + per-model task queues) starting now.

R1 merged (spikersoft-artpipe PR #2): `worker.py --serve` persistent mode — byte-compatible protocol, ready/ping/shutdown lifecycle, fatal-exit semantics, `warmup()` hooks, `PersistentWorkerHandle`. Measured on the M5: sdxl_lightning **4.2 s/job warm** (from 10–20 s cold), safety_check **20–70 ms** (from ~2 s, byte-identical scores). TRELLIS's ~80 s prize needs #358's port half (wrap backends spawn external processes — documented). R2 (.NET resident-model service + `resident` leases + per-model task queues) starting now.

spikerj commented

2026-07-05 01:28:38 +00:00

R2 soak complete (doc PR in spikersoft-artpipe): sdxl_lightning 40/40 jobs, no wall-time drift, RSS creep +0.39 MB/job (not yet plateaued at job 40); safety_check 200/200 at ~20 ms/call with a hard RSS plateau. Data-backed RecycleAfterJobs: 0 for the gate/classifier class, 500 for image-gen class, 500 default for unmeasured residents. These slot straight into R2's config (PR #76, awaiting merge). Remaining phases: R3 (model-queue publish default + API first-stage), R4 (per-model swarm stacks), soak values applied to shipped appsettings when R4 stacks are authored.

R2 soak complete (doc PR in spikersoft-artpipe): sdxl_lightning 40/40 jobs, no wall-time drift, RSS creep +0.39 MB/job (not yet plateaued at job 40); safety_check 200/200 at ~20 ms/call with a hard RSS plateau. **Data-backed `RecycleAfterJobs`: 0 for the gate/classifier class, 500 for image-gen class, 500 default for unmeasured residents.** These slot straight into R2's config (PR #76, awaiting merge). Remaining phases: R3 (model-queue publish default + API first-stage), R4 (per-model swarm stacks), soak values applied to shipped appsettings when R4 stacks are authored.

spikerj commented

2026-07-05 01:59:14 +00:00

R3-partial + R4-authoring merged: backend PR #77 (API-side model-queue routing via ArtStudio:StageModelMap — shipped inert/empty, byte-identical routing pinned by tests; soak-derived RecycleAfterJobsByModel defaults) and infra PR #9 (four per-model resident stacks: sdxl / safety / blender / triposr, replicas-as-scaling-knob, cutover sequencing in headers). Soak doc merged (artpipe PR #3).

All buildable phases of this ticket are now in master/main. What remains is ops-sequenced, on purpose:

SERVER prerequisite (shared with #348): art_pipe checkout + venvs at /mnt/fusionio/spikersoft/art_pipe.
Deploy the four model services; verify art.model.* queues bind and residents hold leases (/gpu/status).
Cutover PR: populate StageModelMap, flip PublishToModelQueues, extend the CI workflow to redeploy the model stacks, demote artpipe-modeling to fallback.
Later: fold #367-U2 placement into resident-set planning; TRELLIS warm win via #358's port half.

Ticket stays open for the cutover; the architecture (event-driven, one always-loaded model per instance, coordinator-arbitrated, replicas: N scaling, config-swappable models) is fully expressed and Mac-validated.

R3-partial + R4-authoring merged: backend PR #77 (API-side model-queue routing via `ArtStudio:StageModelMap` — shipped inert/empty, byte-identical routing pinned by tests; soak-derived `RecycleAfterJobsByModel` defaults) and infra PR #9 (four per-model resident stacks: sdxl / safety / blender / triposr, replicas-as-scaling-knob, cutover sequencing in headers). Soak doc merged (artpipe PR #3). **All buildable phases of this ticket are now in master/main.** What remains is ops-sequenced, on purpose: 1. SERVER prerequisite (shared with #348): art_pipe checkout + venvs at `/mnt/fusionio/spikersoft/art_pipe`. 2. Deploy the four model services; verify `art.model.*` queues bind and residents hold leases (`/gpu/status`). 3. Cutover PR: populate `StageModelMap`, flip `PublishToModelQueues`, extend the CI workflow to redeploy the model stacks, demote `artpipe-modeling` to fallback. 4. Later: fold #367-U2 placement into resident-set planning; TRELLIS warm win via #358's port half. Ticket stays open for the cutover; the architecture (event-driven, one always-loaded model per instance, coordinator-arbitrated, `replicas: N` scaling, config-swappable models) is fully expressed and Mac-validated.

Mode	Models
`ResidentModel` (always-loaded, warm)	Blender, SafetyCheck, SDXLLightning
`ModelQueue` (per-job subprocess)	Hunyuan3DPaint, InstantMesh, Pixal3D, SF3D, ShapE, TextTo3D, TripoSG, TripoSR