spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

ProArt Art Studio: full local pipeline on Windows dev machines #425

New Issue
Closed
opened 2026-07-05 23:16:11 +00:00 by spikerj · 1 comment
spikerj commented 2026-07-05 23:16:11 +00:00
Owner
Copy Link
Copy Source

Problem

The ProArt art_pipe generation worker (SpikerSoft.EventHandlers.ArtPipeProcessor) had no local path on Windows. It existed only for:

  • Mac dev — scripts/mac-dev-up.sh (native worker), and
  • Production — per-stage swarm stacks in spikersoft-infrastructure (spikersoft-artpipe-modeling / -sdxl / -blender / …), built by CI.

On Windows, the API published art.asset.stage.requested.<stage> messages to RabbitMQ but nothing consumed them, so an art "generate" task sat in its first stage forever ("stage 'concept' was never picked up by the worker").

Direction

Make the pipeline run end-to-end on Windows dev machines in Docker, alongside the stack Visual Studio already brings up — the Windows counterpart to the Mac script and the prod swarm stacks (one image, ArtPipe__Stages selects stages).

Done (PR: spikersoft-backend windows-full-local-composes)

  • docker-compose.yml / .override.yml — new artpipe-processor service: CUDA + runtime: nvidia, mounts the sibling ../spikersoft-artpipe checkout at /opt/art_pipe, consumes all 7 stages so nothing sits unconsumed. Uses the container-correct default appsettings.json (rabbitmq / mongo-router / seq / jaeger, GpuScheduling shared via gpu-coordinator); intentionally not ASPNETCORE_ENVIRONMENT=Development (that profile is Mac-native only).
  • ArtPipeProcessor Dockerfile — install blender + headless GL/X libs so the Blender-backed stages (rigging/animation/export/enrichment) run. The prod spikersoft-artpipe-model-blender stack needs this too.
  • scripts/windows-artpipe-bootstrap.ps1 — one-command bootstrap of the per-model Linux venvs + weights inside the running container.
  • docs/windows-local-artpipe.md — setup + troubleshooting runbook.

First-run prerequisite (documented)

Model venvs/weights are not vendored in the image (#348); each dev runs the bootstrap script once. Verified on an RTX 4090 box: concept (SDXLLightning) generation reached the pipeline after bootstrapping the venv.

Follow-ups / open questions

  • Bootstrap is multi-GB per model; consider a shared prebuilt art_pipe cache for devs.
  • Confirm blender apt package + listed GL libs are sufficient for headless blender --background on the CUDA base (image rebuild + a Blender-stage smoke test).
  • Decide whether the single all-stages local worker stays, or we mirror prod's per-stage split locally.

Related: ProArt epic #346, worker #348, safety gate #349, resident mode #368.

## Problem The ProArt `art_pipe` generation worker (`SpikerSoft.EventHandlers.ArtPipeProcessor`) had **no local path on Windows**. It existed only for: - **Mac dev** — `scripts/mac-dev-up.sh` (native worker), and - **Production** — per-stage swarm stacks in `spikersoft-infrastructure` (`spikersoft-artpipe-modeling` / `-sdxl` / `-blender` / …), built by CI. On Windows, the API published `art.asset.stage.requested.<stage>` messages to RabbitMQ but nothing consumed them, so an art "generate" task sat in its first stage forever ("stage 'concept' was never picked up by the worker"). ## Direction Make the pipeline run end-to-end on Windows dev machines in Docker, alongside the stack Visual Studio already brings up — the Windows counterpart to the Mac script and the prod swarm stacks (one image, `ArtPipe__Stages` selects stages). ## Done (PR: spikersoft-backend `windows-full-local-composes`) - **`docker-compose.yml` / `.override.yml`** — new `artpipe-processor` service: CUDA + `runtime: nvidia`, mounts the sibling `../spikersoft-artpipe` checkout at `/opt/art_pipe`, consumes all 7 stages so nothing sits unconsumed. Uses the container-correct default `appsettings.json` (rabbitmq / mongo-router / seq / jaeger, GpuScheduling `shared` via gpu-coordinator); intentionally **not** `ASPNETCORE_ENVIRONMENT=Development` (that profile is Mac-native only). - **ArtPipeProcessor `Dockerfile`** — install `blender` + headless GL/X libs so the Blender-backed stages (rigging/animation/export/enrichment) run. The prod `spikersoft-artpipe-model-blender` stack needs this too. - **`scripts/windows-artpipe-bootstrap.ps1`** — one-command bootstrap of the per-model Linux venvs + weights inside the running container. - **`docs/windows-local-artpipe.md`** — setup + troubleshooting runbook. ## First-run prerequisite (documented) Model venvs/weights are not vendored in the image (#348); each dev runs the bootstrap script once. Verified on an RTX 4090 box: `concept` (SDXLLightning) generation reached the pipeline after bootstrapping the venv. ## Follow-ups / open questions - Bootstrap is multi-GB per model; consider a shared prebuilt art_pipe cache for devs. - Confirm `blender` apt package + listed GL libs are sufficient for headless `blender --background` on the CUDA base (image rebuild + a Blender-stage smoke test). - Decide whether the single all-stages local worker stays, or we mirror prod's per-stage split locally. Related: ProArt epic #346, worker #348, safety gate #349, resident mode #368.
spikerj commented 2026-07-22 02:57:49 +00:00
Author
Owner
Copy Link
Copy Source

Board-sweep verified complete: docker-compose artpipe-processor service + scripts/windows-artpipe-bootstrap.ps1 both in tree; the ticket body's own Done section covers the full scope with only soft follow-up questions remaining. Closing.

Board-sweep verified complete: docker-compose artpipe-processor service + scripts/windows-artpipe-bootstrap.ps1 both in tree; the ticket body's own Done section covers the full scope with only soft follow-up questions remaining. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#425