[ArtStudio P2] ProArt GPU worker — RabbitMQ stage consumer wrapping art_pipe's worker protocol (modeling stage first) #348

New Issue

2026-07-04T05:10:37Z

spikerj commented

2026-07-04 05:10:37 +00:00

Sub-ticket of epic #346. Depends on #347 (contracts + domain). Informed by the model-fit spike.

What: the GPU worker that executes pipeline stages by driving ProArt's per-model subprocess workers, replacing art_pipe's Flask/SQLite orchestrator with our RabbitMQ + Mongo orchestration. Scaffold with the add-event-handler-worker skill (project, appsettings, Dockerfile, infra stack file, Gitea CI workflow).

Scope:

SpikerSoft.EventHandlers.ArtPipeProcessor consuming art.asset.stage.requested; single-replica, pinned to the GPU host (infra template: spikersoft-embeddings/docker-stack-gpu.yml — node.hostname == SERVER, nvidia device reservation)
GpuCoordinator integration: request/renew/release VRAM lease per stage job using the model's vram_mb from its manifest (art_pipe/models/*/artpipe.json); queue while lease unavailable
Drive art_pipe backends via the worker protocol (art_pipe/src/artpipe/worker.py: stdin JSON → stdout NDJSON started|progress|result|error), per-model venvs preserved; map NDJSON progress events to art.asset.stage.progress messages and the stage workflow doc
Docker image: CUDA base, art_pipe checkout + bootstrapped model venvs + HF weight cache volume; document image size / volume strategy (weights are tens of GB — volume-mount, don't bake)
Modeling stage end-to-end first: image→3D (TRELLIS — parity with today's SpikerSoft.EventHandlers.Trellis3D.Python) and text→3D (per spike outcome, e.g. Hunyuan3D-2 or TripoSR), then decimation + mesh-health gate, GLB artifact → GridFS
Stage completion publishes art.asset.stage.completed with artifact refs; failure → terminal error state on the workflow doc (no partial artifacts exposed), retry honors RestartFromStageCommand
Serilog→Seq + OTLP tracing with correlation from the originating command (telemetry-correlation rule); AOT-viability per repo skill where applicable
Decommission plan for SpikerSoft.EventHandlers.Trellis3D.Python once parity is proven (its .gitea/workflows/spikersoft-trellis3d.yml CI too)

Note: art_pipe assumes nvidia-smi/CUDA at startup (main.py::_check_gpu_compat) — the worker container must pass the GPU through. Keep the .NET consumer thin: process supervision + protocol translation only; no model code in .NET.

Acceptance: submitting an asset from P1 produces a real GLB in GridFS via a VRAM-leased art_pipe subprocess on the GPU host, with stage progress visible in the workflow doc and traces in Jaeger/Seq; a second concurrent submission queues on the lease rather than OOMing the card.

Sub-ticket of epic #346. Depends on #347 (contracts + domain). Informed by the model-fit spike. **What:** the GPU worker that executes pipeline stages by driving ProArt's per-model subprocess workers, replacing art_pipe's Flask/SQLite orchestrator with our RabbitMQ + Mongo orchestration. Scaffold with the `add-event-handler-worker` skill (project, appsettings, Dockerfile, infra stack file, Gitea CI workflow). **Scope:** - [ ] `SpikerSoft.EventHandlers.ArtPipeProcessor` consuming `art.asset.stage.requested`; single-replica, pinned to the GPU host (infra template: `spikersoft-embeddings/docker-stack-gpu.yml` — `node.hostname == SERVER`, nvidia device reservation) - [ ] GpuCoordinator integration: request/renew/release VRAM lease per stage job using the model's `vram_mb` from its manifest (`art_pipe/models/*/artpipe.json`); queue while lease unavailable - [ ] Drive art_pipe backends via the worker protocol (`art_pipe/src/artpipe/worker.py`: stdin JSON → stdout NDJSON `started|progress|result|error`), per-model venvs preserved; map NDJSON progress events to `art.asset.stage.progress` messages and the stage workflow doc - [ ] Docker image: CUDA base, art_pipe checkout + bootstrapped model venvs + HF weight cache volume; document image size / volume strategy (weights are tens of GB — volume-mount, don't bake) - [ ] Modeling stage end-to-end first: image→3D (TRELLIS — parity with today's `SpikerSoft.EventHandlers.Trellis3D.Python`) and text→3D (per spike outcome, e.g. Hunyuan3D-2 or TripoSR), then decimation + mesh-health gate, GLB artifact → GridFS - [ ] Stage completion publishes `art.asset.stage.completed` with artifact refs; failure → terminal error state on the workflow doc (no partial artifacts exposed), retry honors `RestartFromStageCommand` - [ ] Serilog→Seq + OTLP tracing with correlation from the originating command (telemetry-correlation rule); AOT-viability per repo skill where applicable - [ ] Decommission plan for `SpikerSoft.EventHandlers.Trellis3D.Python` once parity is proven (its `.gitea/workflows/spikersoft-trellis3d.yml` CI too) **Note:** art_pipe assumes `nvidia-smi`/CUDA at startup (`main.py::_check_gpu_compat`) — the worker container must pass the GPU through. Keep the .NET consumer thin: process supervision + protocol translation only; no model code in .NET. **Acceptance:** submitting an asset from P1 produces a real GLB in GridFS via a VRAM-leased art_pipe subprocess on the GPU host, with stage progress visible in the workflow doc and traces in Jaeger/Seq; a second concurrent submission queues on the lease rather than OOMing the card.

spikerj added the enhancement label 2026-07-04 05:10:37 +00:00