spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra][GPU] artpipe lane down on 4090 — workers Rejected on missing /mnt/fusionio/spikersoft/art_pipe bind after artpipe-gpu label move (collateral of #500) #501

New Issue
Closed
opened 2026-07-12 07:05:39 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-12 07:05:39 +00:00
Owner
Copy Link
Copy Source

Summary

Moving the artpipe-gpu swarm label to the 4090 node (done to serve the image-description worker — see #500/#403) collaterally took the artpipe lane down. The artpipe stacks follow the same artpipe-gpu label, so they rescheduled onto the 4090 — which is not provisioned for artpipe (missing the /mnt/fusionio/spikersoft/art_pipe bind-mount source). Both workers are now Rejected in a tight loop:

spikersoft-artpipe-model-safety_artpipe-model-safety  → 0/1
spikersoft-artpipe-modeling_artpipe-modeling          → 0/1
  Rejected (repeating, every ~5s) on node 4090:
  invalid mount config for type "bind": bind source path does not exist:
  /mnt/fusionio/spikersoft/art_pipe

artpipe-model-safety is the always-on SafetyCheck resident, so this is a live outage of the art pipeline while the label sits on the 4090.

Root cause

Two services share one placement lever (node.labels.artpipe-gpu == true) but have different node-provisioning requirements:

  • image-description — needs only the GPU (self-contained image).
  • artpipe stacks — additionally bind-mount /mnt/fusionio/spikersoft/art_pipe (the art_pipe checkout + bootstrapped model venvs).

docs/4090-burst-node-runbook.md "one-time provisioning" step provides that checkout on the 4090:

sudo git clone https://git.spikersoft.com/spikerj/spikersoft-artpipe.git /mnt/fusionio/spikersoft/art_pipe
# then bootstrap model venvs inside a worker container

That step was not done for this label move (the move was for image-description, not a full artpipe burst), so the artpipe binds fail.

Options

(a) Provision the 4090 for artpipe — run the burst-runbook one-time provisioning (git clone art_pipe to /mnt/fusionio/spikersoft/art_pipe + venv bootstrap), then redeploy the six artpipe stacks (runbook Session START steps 3-4). Restores the lane on the 24 GB card.

(b) Decouple image-description placement from the shared artpipe-gpu label so serving the book-pipeline captioner on the 4090 doesn't drag the artpipe stacks with it. Cleanest long-term: image-description gets its own image-desc-gpu (or reuses a generic gpu) label, independent of the artpipe lane.

Interim if artpipe isn't needed right now: leave as-is (SafetyCheck/modeling stay 0/1) or move the label back to SERVER — but that re-breaks image-description's lease (#500). This label collision is the real design gap.

Repro / evidence

  • docker service ps spikersoft-artpipe-model-safety_artpipe-model-safety → Rejected loop, error above, node 4090.
  • Same for spikersoft-artpipe-modeling_artpipe-modeling.
  • Both were 1/1 on SERVER before the artpipe-gpu label moved to 4090 (~2026-07-12 06:3x).

Related

  • #500 — the gpu-coordinator/label move for image-description that triggered this.
  • #403 — image-description deploy (why the 4090 lane was wanted).
  • #480 — same failure class (task Rejected on a missing bind source).
  • docs/4090-burst-node-runbook.md — the provisioning + single-lane procedure.
## Summary Moving the `artpipe-gpu` swarm label to the **4090** node (done to serve the `image-description` worker — see #500/#403) collaterally **took the artpipe lane down**. The artpipe stacks follow the same `artpipe-gpu` label, so they rescheduled onto the 4090 — which is **not provisioned for artpipe** (missing the `/mnt/fusionio/spikersoft/art_pipe` bind-mount source). Both workers are now Rejected in a tight loop: ``` spikersoft-artpipe-model-safety_artpipe-model-safety → 0/1 spikersoft-artpipe-modeling_artpipe-modeling → 0/1 Rejected (repeating, every ~5s) on node 4090: invalid mount config for type "bind": bind source path does not exist: /mnt/fusionio/spikersoft/art_pipe ``` `artpipe-model-safety` is the **always-on SafetyCheck resident**, so this is a live outage of the art pipeline while the label sits on the 4090. ## Root cause Two services share one placement lever (`node.labels.artpipe-gpu == true`) but have **different node-provisioning requirements**: - `image-description` — needs only the GPU (self-contained image). - artpipe stacks — additionally bind-mount `/mnt/fusionio/spikersoft/art_pipe` (the art_pipe checkout + bootstrapped model venvs). `docs/4090-burst-node-runbook.md` "one-time provisioning" step provides that checkout on the 4090: ``` sudo git clone https://git.spikersoft.com/spikerj/spikersoft-artpipe.git /mnt/fusionio/spikersoft/art_pipe # then bootstrap model venvs inside a worker container ``` That step was not done for this label move (the move was for image-description, not a full artpipe burst), so the artpipe binds fail. ## Options **(a) Provision the 4090 for artpipe** — run the burst-runbook one-time provisioning (`git clone` art_pipe to `/mnt/fusionio/spikersoft/art_pipe` + venv bootstrap), then redeploy the six artpipe stacks (runbook Session START steps 3-4). Restores the lane on the 24 GB card. **(b) Decouple image-description placement from the shared `artpipe-gpu` label** so serving the book-pipeline captioner on the 4090 doesn't drag the artpipe stacks with it. Cleanest long-term: image-description gets its own `image-desc-gpu` (or reuses a generic `gpu`) label, independent of the artpipe lane. Interim if artpipe isn't needed right now: leave as-is (SafetyCheck/modeling stay 0/1) or move the label back to SERVER — but that re-breaks image-description's lease (#500). This label collision is the real design gap. ## Repro / evidence - `docker service ps spikersoft-artpipe-model-safety_artpipe-model-safety` → Rejected loop, error above, node 4090. - Same for `spikersoft-artpipe-modeling_artpipe-modeling`. - Both were 1/1 on SERVER before the `artpipe-gpu` label moved to 4090 (~2026-07-12 06:3x). ## Related - #500 — the gpu-coordinator/label move for image-description that triggered this. - #403 — image-description deploy (why the 4090 lane was wanted). - #480 — same failure class (task Rejected on a missing bind source). - `docs/4090-burst-node-runbook.md` — the provisioning + single-lane procedure.
spikerj commented 2026-07-12 18:28:27 +00:00
Author
Owner
Copy Link
Copy Source

Analysis + decision needed (this is an ops choice, not a code fix — laying out the paths):

Why it broke: the six artpipe stacks follow the movable artpipe-gpu label AND bind /mnt/fusionio/spikersoft/art_pipe — which exists only on SERVER. Label moved to the 4090 (#500) → artpipe tasks Rejected on missing bind. Note the image-description worker is UNAFFECTED (its stack deliberately has no fusionio binds — images + model both come from MinIO) and is captioning fine on the 4090.

Resolution options:

  1. Provision the art_pipe tree on the 4090 (the burst runbook's implicit step): rsync /mnt/fusionio/spikersoft/art_pipe from SERVER to the same path on the 4090 (venvs are arch-compatible x86→x86; CUDA venvs may need a re-bootstrap pass per #432/#357). ArtStudio then runs on the 24 GB card — the point of the burst session.
  2. Split the lane: move artpipe-gpu back to SERVER (+ coordinator Nodes flip per the runbook) once caption-heavy work is done — image-description would then starve again (16 GB > 8 GB), so this only makes sense between caption sessions. The single-lane invariant (#367 U2 would remove it) forces this either/or today.
  3. Longer-term (the #497/#413 trajectory): artpipe model weights into ai-models + boot-time venv bootstrap on any node — removes the data-gravity pin the same way image-description's was removed. Real work, tracked under #413/#432.

Recommend 1 for the current session (one rsync + venv re-bootstrap), 3 as the durable direction. Whoever runs option 1: docker service ps the six artpipe stacks after — they should schedule on the 4090 once the bind source exists.

Analysis + decision needed (this is an ops choice, not a code fix — laying out the paths): **Why it broke**: the six artpipe stacks follow the movable `artpipe-gpu` label AND bind `/mnt/fusionio/spikersoft/art_pipe` — which exists only on SERVER. Label moved to the 4090 (#500) → artpipe tasks Rejected on missing bind. Note the image-description worker is UNAFFECTED (its stack deliberately has no fusionio binds — images + model both come from MinIO) and is captioning fine on the 4090. **Resolution options:** 1. **Provision the art_pipe tree on the 4090** (the burst runbook's implicit step): rsync `/mnt/fusionio/spikersoft/art_pipe` from SERVER to the same path on the 4090 (venvs are arch-compatible x86→x86; CUDA venvs may need a re-bootstrap pass per #432/#357). ArtStudio then runs on the 24 GB card — the point of the burst session. 2. **Split the lane**: move `artpipe-gpu` back to SERVER (+ coordinator Nodes flip per the runbook) once caption-heavy work is done — image-description would then starve again (16 GB > 8 GB), so this only makes sense between caption sessions. The single-lane invariant (#367 U2 would remove it) forces this either/or today. 3. **Longer-term (the #497/#413 trajectory)**: artpipe model *weights* into `ai-models` + boot-time venv bootstrap on any node — removes the data-gravity pin the same way image-description's was removed. Real work, tracked under #413/#432. Recommend 1 for the current session (one rsync + venv re-bootstrap), 3 as the durable direction. Whoever runs option 1: `docker service ps` the six artpipe stacks after — they should schedule on the 4090 once the bind source exists.
spikerj commented 2026-07-13 01:56:16 +00:00
Author
Owner
Copy Link
Copy Source

Safety-lane half of this is CLEARED (2026-07-13 01:51Z): the #517 baked-image cutover deployed and spikersoft-artpipe-model-safety is 1/1 Running on the 4090 with no /mnt/fusionio/spikersoft/art_pipe bind — the 'invalid mount config' reject-loop is gone for that service. spikersoft-artpipe-modeling still reject-loops on the missing bind and stays broken until its lane cuts over (Phase 2/3, #518/#519) or the bind is provisioned on the 4090. Suggest re-scoping this ticket to the modeling lane only.

Safety-lane half of this is CLEARED (2026-07-13 01:51Z): the #517 baked-image cutover deployed and spikersoft-artpipe-model-safety is 1/1 Running on the 4090 with no /mnt/fusionio/spikersoft/art_pipe bind — the 'invalid mount config' reject-loop is gone for that service. spikersoft-artpipe-modeling still reject-loops on the missing bind and stays broken until its lane cuts over (Phase 2/3, #518/#519) or the bind is provisioned on the 4090. Suggest re-scoping this ticket to the modeling lane only.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#501