spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Epic][ArtPipe][Infra] Per-model baked Docker images — venvs + weights built at image build time, no /opt/art_pipe bind #515

New Issue
Open
opened 2026-07-13 00:40:01 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-13 00:40:01 +00:00
Owner
Copy Link
Copy Source

Goal

Convert the art_pipe GPU worker from a bind-mounted checkout (/mnt/fusionio/spikersoft/art_pipe → /opt/art_pipe, six stacks, one shared image) to per-model pre-built Docker images with venvs AND weights fully baked at build time, so a fresh GPU node runs any model with docker pull alone. No /opt/art_pipe bind anywhere; runtime gets HF_HUB_OFFLINE=1.

This eliminates the entire provisioning-gap outage class: #455 (TripoSG not vendored on prod), #486 (SafetyCheck venv missing torch), #501 (label move broke on missing bind source), #432 (venv bootstrap gaps), and deletes the 4090 runbook's "clone + bootstrap venvs" step (#425-adjacent). Stage artifacts already flow via GridFS — the bind exists ONLY for code+venvs+weights.

Architecture (two-tier, so backend commits don't re-push 10-40 GB layers)

nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04
 └─ spikerj/artpipe-base            (artpipe repo: apt/blender/deadsnakes py3.10+3.11/.NET 10 runtime)
   └─ spikerj/artpipe-model-env-<m> (artpipe repo: /opt/art_pipe src + venv + BAKED weights)
     └─ spikerj/artpipe-model-<m>   (backend repo: dotnet publish layer on top)
  • Images only for models a stack deploys: safetycheck, sdxllightning, triposr, hunyuan3dpaint, blender, prodstages (SDXLLightning+TripoSR+SDTurboTex+Blender for the all-stages lane). FluxSchnell/gated/non-prod deferred.
  • Executor contract preserved ({PipelineRoot}/models/{ModelDir}/venv/bin/python, .venv fallback, HF cache at {PipelineRoot}/.cache) — tree built at /opt/art_pipe verbatim, zero .NET changes.
  • Dockerfiles reuse python -m artpipe setup --model X (venv_setup.py) — no duplicated bootstrap logic; bootstrap.py stays the local-dev path.
  • Tags: env :<artpipe_sha7> + :latest; final :<backend_sha7>-<artpipe_sha7> (immutable backout pin) + :latest. Gitea cleanup keep-last-3.
  • Backout: legacy spikersoft-artpipe-processor:latest keeps building until all six stacks cut over; SERVER checkout stays intact ≥1 month.

Risks (tracked in phase tickets)

Registry blob growth 80-150 GB in gitea-packages on /mnt/fusionio/minio (hard disk go/no-go gate in Phase 0); large-blob pushes through Traefik→Gitea→MinIO; runner disk (#514 — sequential builds + prune); HF_HUB_OFFLINE=1 surfacing straggler downloads (fix = extend manifest weights entries); prodstages model list must track appsettings StageSettings.

Phases

  • Phase 0: build scaffolding + disk go/no-go
  • Phase 1: SafetyCheck pilot (0.34 GB weights, always-on resident, currently broken per #486)
  • Phase 2: stood-down residents (blender, sdxl, triposr; hunyuan deferred to a burst session)
  • Phase 3: prodstages + live spikersoft-artpipe-modeling lane cutover
  • Phase 4: cleanup (retire legacy image + bind path, runbook/docs)

Acid test for done: move artpipe-gpu label to the 4090 with NO checkout at /mnt/fusionio/spikersoft/art_pipe — every artpipe stack goes healthy on pull alone.

Related: #455 #486 #501 #432 #425 #368 #413 #514.

## Goal Convert the art_pipe GPU worker from a bind-mounted checkout (`/mnt/fusionio/spikersoft/art_pipe → /opt/art_pipe`, six stacks, one shared image) to **per-model pre-built Docker images with venvs AND weights fully baked at build time**, so a fresh GPU node runs any model with `docker pull` alone. No `/opt/art_pipe` bind anywhere; runtime gets `HF_HUB_OFFLINE=1`. This eliminates the entire provisioning-gap outage class: #455 (TripoSG not vendored on prod), #486 (SafetyCheck venv missing torch), #501 (label move broke on missing bind source), #432 (venv bootstrap gaps), and deletes the 4090 runbook's "clone + bootstrap venvs" step (#425-adjacent). Stage artifacts already flow via GridFS — the bind exists ONLY for code+venvs+weights. ## Architecture (two-tier, so backend commits don't re-push 10-40 GB layers) ``` nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04 └─ spikerj/artpipe-base (artpipe repo: apt/blender/deadsnakes py3.10+3.11/.NET 10 runtime) └─ spikerj/artpipe-model-env-<m> (artpipe repo: /opt/art_pipe src + venv + BAKED weights) └─ spikerj/artpipe-model-<m> (backend repo: dotnet publish layer on top) ``` - Images only for models a stack deploys: safetycheck, sdxllightning, triposr, hunyuan3dpaint, blender, prodstages (SDXLLightning+TripoSR+SDTurboTex+Blender for the all-stages lane). FluxSchnell/gated/non-prod deferred. - Executor contract preserved (`{PipelineRoot}/models/{ModelDir}/venv/bin/python`, `.venv` fallback, HF cache at `{PipelineRoot}/.cache`) — tree built at `/opt/art_pipe` verbatim, **zero .NET changes**. - Dockerfiles reuse `python -m artpipe setup --model X` (venv_setup.py) — no duplicated bootstrap logic; bootstrap.py stays the local-dev path. - Tags: env `:<artpipe_sha7>` + `:latest`; final `:<backend_sha7>-<artpipe_sha7>` (immutable backout pin) + `:latest`. Gitea cleanup keep-last-3. - Backout: legacy `spikersoft-artpipe-processor:latest` keeps building until all six stacks cut over; SERVER checkout stays intact ≥1 month. ## Risks (tracked in phase tickets) Registry blob growth 80-150 GB in `gitea-packages` on `/mnt/fusionio/minio` (**hard disk go/no-go gate in Phase 0**); large-blob pushes through Traefik→Gitea→MinIO; runner disk (#514 — sequential builds + prune); `HF_HUB_OFFLINE=1` surfacing straggler downloads (fix = extend manifest weights entries); prodstages model list must track appsettings StageSettings. ## Phases - Phase 0: build scaffolding + disk go/no-go - Phase 1: SafetyCheck pilot (0.34 GB weights, always-on resident, currently broken per #486) - Phase 2: stood-down residents (blender, sdxl, triposr; hunyuan deferred to a burst session) - Phase 3: prodstages + live `spikersoft-artpipe-modeling` lane cutover - Phase 4: cleanup (retire legacy image + bind path, runbook/docs) **Acid test for done:** move `artpipe-gpu` label to the 4090 with NO checkout at `/mnt/fusionio/spikersoft/art_pipe` — every artpipe stack goes healthy on pull alone. Related: #455 #486 #501 #432 #425 #368 #413 #514.
spikerj commented 2026-07-14 05:13:33 +00:00
Author
Owner
Copy Link
Copy Source

Proposed endgame for this epic: split the worker from the model (Joey's architecture call, 2026-07-14)

The complaint that prompted this: "making a change to anything in .NET never seemed to make us rebuild the huge models and their little interaction harness... I don't want multi-gigabyte rebuilds just because a few lines changed in a shared library."

He is right, and the mechanism is worse than the symptom suggests.

What actually happens today

SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile.model is:

FROM ${ENV_IMAGE}                      # venvs + weights: 10-40 GB
COPY --from=publish /app/publish .     # the .NET worker

The worker and the models are ONE image. So any change to SpikerSoft.Common/** (most backend merges) rebuilds every model image. The push is small (weights layers are cross-repo mounted), but the build must first docker pull each 10-40 GB env image onto the runner. That is not theoretical: run 6691 died with unexpected EOF 13 minutes into pulling artpipe-model-env-triposr — triggered by a quiz-generation fix. A few lines of C# dragged tens of GB across the network and failed CI.

The architecture Joey wants is already 80% built here

  • RemoteArtifactSafetyGate.cs — RabbitMQ RPC to a separate SafetyCheck service (auto-delete reply queue, CorrelationId matching, W3C trace propagation). Live in production (SafetyGate:Remote=true).
  • python -m artpipe.worker --serve — long-lived stdin/stdout JSON job protocol (ServeWorkerProcess.cs).
  • ArtPipe__ResidentModel + PublishToModelQueues — the per-model queue topology from #368.

The #368 rollback conflated two things

  • (a) the model runs in its own container, invoked by message — architecture, costs nothing
  • (b) the model stays RESIDENT in VRAM for the container's lifetime — policy, needed ~34 GB of leases and could not fit SERVER's 8 GB card

We abandoned (a) because (b) did not fit. (a) works fine with per-task load/unload — exactly what the current subprocess lane already does — and VRAM behavior is unchanged because the gpu-coordinator brokers leases either way.

The one genuinely missing piece

Even the resident path bakes .NET into the multi-GB image (the resident IS the .NET worker wrapping python). For a real split, the model-side service must contain no .NET: artpipe needs its own RabbitMQ consumer — artpipe.service — wrapping the same job-spec protocol --serve already speaks.

Target

image contains rebuilt when size
artpipe-runner python + venvs + weights + harness models/python change 10-40 GB, rarely
spikersoft-artpipe-processor .NET worker only every .NET commit ~200 MB, seconds

The artpipe-model-* tier-3 images disappear (this epic's #520 cleanup, but for real). The .NET worker becomes GPU-free and volume-free — it can run on any node, which is exactly what #413 is chasing.

Phases (proposed)

  1. artpipe: artpipe.service — a RabbitMQ consumer implementing the --serve job spec; artifacts uploaded to a MinIO bucket, returning keys instead of leaving files under /opt/art_pipe/output. New image artpipe-runner (python only, NO .NET).
  2. backend: RemoteArtPipeStageExecutor — the RPC shape RemoteArtifactSafetyGate already proves, for all stages; fetch artifacts from the bucket. ArtPipeProcessor drops every model dependency → thin image.
  3. infra: spikersoft-artpipe-runner stack (GPU, artpipe-gpu label, the big image) + spikersoft-artpipe-processor as a thin CPU service; retire the tier-3 images.

MinIO is the enabler: the moment the two containers stop sharing a filesystem, artifact hand-off must go through object storage. #413 and this epic converge.

Wiring this up needs Joey's go-ahead on the phasing — happy to open the child tickets.

— macbook-claude-session

# Proposed endgame for this epic: split the worker from the model (Joey's architecture call, 2026-07-14) **The complaint that prompted this:** *"making a change to anything in .NET never seemed to make us rebuild the huge models and their little interaction harness... I don't want multi-gigabyte rebuilds just because a few lines changed in a shared library."* He is right, and the mechanism is worse than the symptom suggests. ## What actually happens today `SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile.model` is: ``` FROM ${ENV_IMAGE} # venvs + weights: 10-40 GB COPY --from=publish /app/publish . # the .NET worker ``` The worker and the models are ONE image. So any change to `SpikerSoft.Common/**` (most backend merges) rebuilds every model image. The *push* is small (weights layers are cross-repo mounted), but the **build must first `docker pull` each 10-40 GB env image onto the runner**. That is not theoretical: run 6691 died with `unexpected EOF` 13 minutes into pulling `artpipe-model-env-triposr` — triggered by a quiz-generation fix. A few lines of C# dragged tens of GB across the network and failed CI. ## The architecture Joey wants is already 80% built here - `RemoteArtifactSafetyGate.cs` — RabbitMQ RPC to a **separate** SafetyCheck service (auto-delete reply queue, CorrelationId matching, W3C trace propagation). **Live in production** (`SafetyGate:Remote=true`). - `python -m artpipe.worker --serve` — long-lived stdin/stdout JSON job protocol (`ServeWorkerProcess.cs`). - `ArtPipe__ResidentModel` + `PublishToModelQueues` — the per-model queue topology from #368. ## The #368 rollback conflated two things - **(a)** the model runs in its own container, invoked by message — *architecture*, costs nothing - **(b)** the model stays RESIDENT in VRAM for the container's lifetime — *policy*, needed ~34 GB of leases and could not fit SERVER's 8 GB card We abandoned (a) because (b) did not fit. **(a) works fine with per-task load/unload** — exactly what the current subprocess lane already does — and VRAM behavior is unchanged because the gpu-coordinator brokers leases either way. ## The one genuinely missing piece Even the resident path bakes .NET into the multi-GB image (the resident IS the .NET worker wrapping python). For a real split, the model-side service must contain **no .NET**: artpipe needs its own RabbitMQ consumer — `artpipe.service` — wrapping the same job-spec protocol `--serve` already speaks. ## Target | image | contains | rebuilt when | size | |---|---|---|---| | `artpipe-runner` | python + venvs + weights + harness | models/python change | 10-40 GB, rarely | | `spikersoft-artpipe-processor` | .NET worker only | every .NET commit | ~200 MB, seconds | The `artpipe-model-*` tier-3 images disappear (this epic's #520 cleanup, but for real). The .NET worker becomes GPU-free and **volume-free** — it can run on any node, which is exactly what #413 is chasing. ## Phases (proposed) 1. **artpipe**: `artpipe.service` — a RabbitMQ consumer implementing the `--serve` job spec; artifacts uploaded to a MinIO bucket, returning keys instead of leaving files under `/opt/art_pipe/output`. New image `artpipe-runner` (python only, NO .NET). 2. **backend**: `RemoteArtPipeStageExecutor` — the RPC shape `RemoteArtifactSafetyGate` already proves, for all stages; fetch artifacts from the bucket. ArtPipeProcessor drops every model dependency → thin image. 3. **infra**: `spikersoft-artpipe-runner` stack (GPU, `artpipe-gpu` label, the big image) + `spikersoft-artpipe-processor` as a thin CPU service; retire the tier-3 images. **MinIO is the enabler**: the moment the two containers stop sharing a filesystem, artifact hand-off must go through object storage. #413 and this epic converge. Wiring this up needs Joey's go-ahead on the phasing — happy to open the child tickets. — macbook-claude-session
spikerj commented 2026-07-29 18:26:03 +00:00
Author
Owner
Copy Link
Copy Source

Epic status roll-up — audited against origin/master. The acid test is met in configuration across all 15 artpipe stacks. This epic is substantially delivered and could plausibly close after one live check.

The bind is gone everywhere. Checking for actual host binds rather than the env var: scanning every spikersoft-artpipe* stack for a volumes: entry mounting /mnt/fusionio/spikersoft/art_pipe returns zero hits across all 15. The only remaining art_pipe references are ArtPipe__PipelineRoot=/opt/art_pipe — which is the preserved executor contract the epic explicitly requires ("tree built at /opt/art_pipe verbatim, zero .NET changes") — plus prose in the documented backout blocks.

HF_HUB_OFFLINE=1 is on 15 of 15 stacks.

The legacy image is referenced only in backout comments (spikersoft-artpipe-model-blender/docker-stack.yml:66, -safety:57, -sdxl:61), not as a live image: anywhere. That matches the stated backout plan — legacy kept buildable, not deployed.

Phase 3's blocker cleared today. Per my audit of #519: spikerj/artpipe-model-prodstages now has four versions, the newest built 2026-07-29T15:43Z, and the modeling stack points at it with the fusionio bind removed from the service spec. The tier-3 image that was the named blocker in the 2026-07-22 sweep exists and is building repeatedly, not just once.

What's left is verification, not construction:

  1. The acid test itself — move the artpipe-gpu label to the 4090 with no checkout at /mnt/fusionio/spikersoft/art_pipe and confirm every stack goes healthy on pull alone. Config says it should; only the live run proves it.
  2. Phase 4 cleanup — retiring the legacy image build and the runbook's "clone + bootstrap venvs" step.

Two cautions before running the acid test, both from other audits today:

  • #857: the registry still issues anonymous pull tokens publicly (re-verified live today). This epic's whole model is "a fresh node runs any model with docker pull alone" — which is exactly the capability currently available to anyone on the internet, for ~40 GB model-env images.
  • #775 / #821: blob-integrity hardening is unimplemented, and #821 records three corrupt giant layers from env run 17775 — including 15.75 GB and 17.25 GB, which match #858's two mega-layer RUN loops exactly. A pull-only architecture depends entirely on blob integrity, so those two are worth settling before declaring this done.

Neither blocks closing the epic on its own terms, but both are load-bearing for the property it delivers. Worth recording the linkage here so #515 doesn't close as "pull-only works" while the pull path itself has open integrity and access tickets.

**Epic status roll-up** — audited against `origin/master`. **The acid test is met in configuration across all 15 artpipe stacks. This epic is substantially delivered and could plausibly close after one live check.** **The bind is gone everywhere.** Checking for actual host binds rather than the env var: scanning every `spikersoft-artpipe*` stack for a `volumes:` entry mounting `/mnt/fusionio/spikersoft/art_pipe` returns **zero hits across all 15**. The only remaining `art_pipe` references are `ArtPipe__PipelineRoot=/opt/art_pipe` — which is the preserved executor contract the epic explicitly requires ("tree built at `/opt/art_pipe` verbatim, zero .NET changes") — plus prose in the documented backout blocks. **`HF_HUB_OFFLINE=1` is on 15 of 15 stacks.** **The legacy image is referenced only in backout comments** (`spikersoft-artpipe-model-blender/docker-stack.yml:66`, `-safety:57`, `-sdxl:61`), not as a live `image:` anywhere. That matches the stated backout plan — legacy kept buildable, not deployed. **Phase 3's blocker cleared today.** Per my audit of #519: `spikerj/artpipe-model-prodstages` now has four versions, the newest built **2026-07-29T15:43Z**, and the modeling stack points at it with the fusionio bind removed from the service spec. The tier-3 image that was the named blocker in the 2026-07-22 sweep exists and is building repeatedly, not just once. **What's left is verification, not construction:** 1. **The acid test itself** — move the `artpipe-gpu` label to the 4090 with no checkout at `/mnt/fusionio/spikersoft/art_pipe` and confirm every stack goes healthy on pull alone. Config says it should; only the live run proves it. 2. **Phase 4 cleanup** — retiring the legacy image build and the runbook's "clone + bootstrap venvs" step. **Two cautions before running the acid test**, both from other audits today: - **#857**: the registry still issues anonymous pull tokens publicly (re-verified live today). This epic's whole model is "a fresh node runs any model with `docker pull` alone" — which is exactly the capability currently available to anyone on the internet, for ~40 GB model-env images. - **#775 / #821**: blob-integrity hardening is unimplemented, and #821 records **three** corrupt giant layers from env run 17775 — including 15.75 GB and 17.25 GB, which match #858's two mega-layer `RUN` loops exactly. A pull-only architecture depends entirely on blob integrity, so those two are worth settling before declaring this done. Neither blocks closing the epic on its own terms, but both are load-bearing for the property it delivers. Worth recording the linkage here so #515 doesn't close as "pull-only works" while the pull path itself has open integrity and access tickets.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#515