spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Prod][ArtPipe] Concept-art is dead because ArtStudio:StageModelMap is EMPTY — API routes to the legacy in-process stage queue whose consumer can't boot, while the 5 resident model containers sit idle on queues nobody publishes to #595

New Issue
Open
opened 2026-07-14 21:15:33 +00:00 by spikerj · 5 comments
spikerj commented 2026-07-14 21:15:33 +00:00
Owner
Copy Link
Copy Source

Filed by: QA Team — found while a real user-initiated concept-art generation was in flight.

Summary

Art Studio concept-art generation is non-functional. A live request published at ~21:10Z routed correctly onto art.asset.stage.concept.requests and is still sitting there, unread, with zero consumers. It is not failing and not dead-lettering — it is simply never being picked up, and there is no TTL, so it will sit indefinitely.

Two independent faults, either of which alone is fatal to the stage.

Fault 1 — the concept-stage consumer can NEVER start (contradictory placement vs bind)

spikersoft-artpipe-modeling (the service running the spikersoft-artpipe-processor image, which consumes the concept stage) is 0/1 in a permanent reject loop, retrying every ~5 seconds:

$ docker service ps spikersoft-artpipe-modeling_artpipe-modeling
  4090  Rejected less than a second ago  "invalid mount config for type \"bind\":
                                          bind source path does not exist:
                                          /mnt/fusionio/spikersoft/art_pipe"
  4090  Rejected 5 seconds ago    (same)
  4090  Rejected 10 seconds ago   (same)
  4090  Rejected 15 seconds ago   (same)

The service spec is self-contradictory:

placement constraints : ["node.labels.artpipe-gpu == true"]      -> resolves ONLY to the 4090
bind mount            : /mnt/fusionio/spikersoft/art_pipe -> /opt/art_pipe

But /mnt/fusionio is SERVER's disk and does not exist on the 4090:

$ ls -d /mnt/fusionio                       # run on the 4090
  ls: cannot access '/mnt/fusionio': No such file or directory

Node labels confirm only the 4090 carries artpipe-gpu:

4090   labels = {"artpipe-gpu":"true","spikersoft-app":"true"}
SERVER labels = {"spikersoft-app":"true"}

So the constraint pins it to a node that structurally cannot satisfy its bind. It has never started — swarm has simply been retrying it every 5 seconds since the service was last updated (2026-07-13 18:57Z).

This is the same bug class as #553 (closed)

#553 / infra #81 was exactly this: a GPU service pinned to the artpipe-gpu node while still carrying a SERVER-only /mnt/fusionio bind. The fix there was to drop the bind and self-provision from MinIO instead. artpipe-modeling has the identical latent contradiction and was missed in that sweep.

Fix options:

  • (a) Replace the /mnt/fusionio/spikersoft/art_pipe bind with a named volume + MinIO self-provisioning, mirroring what infra #81 did for quiz-generation. Preferred — consistent with the direction already taken.
  • (b) Create /mnt/fusionio/spikersoft/art_pipe on the 4090 (or mount fusionio there). Quicker, but re-introduces the node-local-state coupling that #553 was fixing.

⚠️ If (a) is chosen, note the lesson from #592: infra #81 dropped the bind and introduced a ${VAR} secret in the same change, and docker stack deploy substituted the unset var with an empty string and exited 0 — shipping a service that looked green but could not work. Do not repeat that. Guard the secret in the deploy job.

Fault 2 — there is no concept generator running

Even with the consumer fixed, nothing would generate anything. The model workers:

artpipe-model-sdxl      0/0   artpipe-model-sdxllightning:latest   <-- THE concept generator, replicas=0
artpipe-model-blender   0/0
artpipe-model-triposr   0/0
artpipe-model-hunyuan   0/0
artpipe-model-safety    1/1   <-- the only one up

artpipe-model-sdxl is explicitly set to replicas=0 in its spec, and its image pulls fine (manifest OK), so this reads as a deliberate scale-to-zero rather than a failure. Whether that is intentional needs an owner's call — but as it stands, the concept stage has a consumer that can't boot and a generator that isn't running.

Possibly related to #537 (closed) — "Phase 2 blocked: sdxllightning env image is 98 GB". If Phase 2 is still considered blocked, the Art Studio UI should not be accepting concept-art submissions that it cannot service.

Fault 3 (minor, but worth a look) — hunyuan runs the wrong image

spikersoft-artpipe-model-hunyuan  ->  spikersoft-artpipe-processor:latest

Every other model worker runs its own artpipe-model-<name> image. hunyuan pointing at the processor image looks like a copy-paste error in its stack file. It is 0/0 so it is not currently biting, but it will the moment it is scaled up.

User-facing impact

The request does not error, does not retry, and does not dead-letter. It is silently swallowed by a consumer-less queue. From the user's side the generation just never completes, with no failure ever surfaced. Whatever the fix, the Art Studio should surface a failure (or refuse the submission) rather than accept work into a stage that has no consumer.

Related

  • #553 (closed) — same placement-vs-bind bug class; this one was missed in that sweep.
  • #592 (open) — the empty-secret trap introduced by the #553 remediation; read before applying fix (a).
  • #537 (closed) — sdxllightning image size / Phase 2 blocked.
  • #589 (open) — the broader "queues with zero consumers silently swallow user work" theme.
**Filed by: QA Team** — found while a real user-initiated concept-art generation was in flight. ## Summary **Art Studio concept-art generation is non-functional.** A live request published at ~21:10Z routed correctly onto `art.asset.stage.concept.requests` and is **still sitting there, unread, with zero consumers.** It is not failing and not dead-lettering — it is simply never being picked up, and there is no TTL, so it will sit indefinitely. Two independent faults, either of which alone is fatal to the stage. ## Fault 1 — the concept-stage consumer can NEVER start (contradictory placement vs bind) `spikersoft-artpipe-modeling` (the service running the `spikersoft-artpipe-processor` image, which consumes the concept stage) is **0/1** in a permanent reject loop, retrying every ~5 seconds: ``` $ docker service ps spikersoft-artpipe-modeling_artpipe-modeling 4090 Rejected less than a second ago "invalid mount config for type \"bind\": bind source path does not exist: /mnt/fusionio/spikersoft/art_pipe" 4090 Rejected 5 seconds ago (same) 4090 Rejected 10 seconds ago (same) 4090 Rejected 15 seconds ago (same) ``` The service spec is self-contradictory: ``` placement constraints : ["node.labels.artpipe-gpu == true"] -> resolves ONLY to the 4090 bind mount : /mnt/fusionio/spikersoft/art_pipe -> /opt/art_pipe ``` But `/mnt/fusionio` **is SERVER's disk and does not exist on the 4090**: ``` $ ls -d /mnt/fusionio # run on the 4090 ls: cannot access '/mnt/fusionio': No such file or directory ``` Node labels confirm only the 4090 carries `artpipe-gpu`: ``` 4090 labels = {"artpipe-gpu":"true","spikersoft-app":"true"} SERVER labels = {"spikersoft-app":"true"} ``` So the constraint pins it to a node that structurally cannot satisfy its bind. **It has never started** — swarm has simply been retrying it every 5 seconds since the service was last updated (2026-07-13 18:57Z). ### This is the same bug class as #553 (closed) #553 / infra #81 was exactly this: a GPU service pinned to the `artpipe-gpu` node while still carrying a **SERVER-only `/mnt/fusionio` bind**. The fix there was to drop the bind and self-provision from MinIO instead. `artpipe-modeling` has the identical latent contradiction and was missed in that sweep. **Fix options:** - **(a)** Replace the `/mnt/fusionio/spikersoft/art_pipe` bind with a named volume + MinIO self-provisioning, mirroring what infra #81 did for quiz-generation. Preferred — consistent with the direction already taken. - **(b)** Create `/mnt/fusionio/spikersoft/art_pipe` on the 4090 (or mount fusionio there). Quicker, but re-introduces the node-local-state coupling that #553 was fixing. ⚠️ If (a) is chosen, **note the lesson from #592**: infra #81 dropped the bind *and* introduced a `${VAR}` secret in the same change, and `docker stack deploy` substituted the unset var with an empty string and exited 0 — shipping a service that looked green but could not work. Do not repeat that. Guard the secret in the deploy job. ## Fault 2 — there is no concept generator running Even with the consumer fixed, nothing would generate anything. The model workers: ``` artpipe-model-sdxl 0/0 artpipe-model-sdxllightning:latest <-- THE concept generator, replicas=0 artpipe-model-blender 0/0 artpipe-model-triposr 0/0 artpipe-model-hunyuan 0/0 artpipe-model-safety 1/1 <-- the only one up ``` `artpipe-model-sdxl` is explicitly set to **`replicas=0`** in its spec, and its image pulls fine (`manifest OK`), so this reads as a deliberate scale-to-zero rather than a failure. Whether that is intentional needs an owner's call — but as it stands, the concept stage has a consumer that can't boot and a generator that isn't running. Possibly related to **#537** (closed) — *"Phase 2 blocked: sdxllightning env image is 98 GB"*. If Phase 2 is still considered blocked, the Art Studio UI should not be accepting concept-art submissions that it cannot service. ## Fault 3 (minor, but worth a look) — hunyuan runs the wrong image ``` spikersoft-artpipe-model-hunyuan -> spikersoft-artpipe-processor:latest ``` Every other model worker runs its own `artpipe-model-<name>` image. hunyuan pointing at the **processor** image looks like a copy-paste error in its stack file. It is 0/0 so it is not currently biting, but it will the moment it is scaled up. ## User-facing impact The request does not error, does not retry, and does not dead-letter. It is silently swallowed by a consumer-less queue. From the user's side the generation just **never completes, with no failure ever surfaced**. Whatever the fix, the Art Studio should surface a failure (or refuse the submission) rather than accept work into a stage that has no consumer. ## Related - **#553** (closed) — same placement-vs-bind bug class; this one was missed in that sweep. - **#592** (open) — the empty-secret trap introduced by the #553 remediation; read before applying fix (a). - **#537** (closed) — sdxllightning image size / Phase 2 blocked. - **#589** (open) — the broader "queues with zero consumers silently swallow user work" theme.
spikerj commented 2026-07-14 21:19:48 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — ROOT CAUSE FOUND. This supersedes the fix I proposed above; that advice was wrong.

I originally suggested recreating the /mnt/fusionio bind on the 4090 or replacing it with MinIO self-provisioning. Don't do either. The bind should not exist at all. The right fix is a one-line config change.

The processor has two topologies, and we are running the wrong one

ArtPipeProcessor is one codebase with two deployment modes (ArtPipeStageConsumer.ResolveBindings):

Mode Set by Binds queue Executes how
Resident ResidentModel=<X> art.model.<x>.tasks model baked into its own artpipe-model-* image → needs no host bind
Per-stage (legacy) no ResidentModel art.asset.stage.<s>.requests execs python -m artpipe.worker itself from /opt/art_pipe → needs the host bind

The C# doc on PublishToModelQueues says it outright:

/// Default FALSE: per-stage mode remains the default topology everywhere until R3/R4.
public bool PublishToModelQueues { get; set; }

The resident-container architecture is fully built and fully deployed. All five model workers are running in resident mode, and four of the five carry no bind at all — they bake the art_pipe checkout and venvs into their own images:

ResidentModel=SDXLLightning   replicas=0/0   queue=art.model.sdxllightning.tasks    [no bind]
ResidentModel=TripoSR         replicas=0/0   queue=art.model.triposr.tasks          [no bind]
ResidentModel=Blender         replicas=0/0   queue=art.model.blender.tasks          [no bind]
ResidentModel=SafetyCheck     replicas=1/1   queue=art.model.safetycheck.tasks      [no bind]  <-- works today
ResidentModel=Hunyuan3DPaint  replicas=0/0   queue=art.model.hunyuan3dpaint.tasks   [BIND]     <-- the odd one out

artpipe-model-safety is running right now, 1/1, consuming its model queue, with no host bind whatsoever. That is the proof that the bind is unnecessary in this topology.

So why is anything binding fusionio?

Because artpipe-modeling has no ResidentModel → it is in legacy per-stage mode → it must execute models in-process as a subprocess → it needs the art_pipe checkout on disk → hence /mnt/fusionio/spikersoft/art_pipe. And it is pinned to artpipe-gpu (the 4090), where fusionio does not exist. Hence the permanent reject loop.

And why does the request go to the legacy path at all?

This is the actual root cause. SubmitArtAssetCommandHandler picks the routing key from ArtStudio:StageModelMap:

// #368 R3: when ArtStudio:StageModelMap maps this stage, the
// command's RoutingKey becomes art.model.<model>.task (the
// resident instance's task queue) instead of the per-stage key.
ResidentModelDir = ResolveResidentModelDir(firstStage, stageMethods),

And in both SpikerSoft.Api/appsettings.json and appsettings.Production.json:

"StageModelMap": {},

It is empty. So no stage is ever mapped to a model, so the API always falls back to the per-stage routing key, so every request lands on art.asset.stage.concept.requests, whose only possible consumer is the legacy in-process processor — which cannot boot.

Meanwhile the five resident containers sit on art.model.*.tasks queues that nothing ever publishes to.

The chain, end to end

  1. ArtStudio:StageModelMap is {} → concept stage is unmapped.
  2. API publishes with the per-stage key → art.asset.stage.concept.requests.
  3. Only a per-stage-mode processor binds that queue — i.e. artpipe-modeling.
  4. Per-stage mode must exec the model in-process → needs /opt/art_pipe → needs the fusionio bind.
  5. It is pinned to the 4090; fusionio is SERVER's disk → reject loop, never starts, 0 consumers.
  6. The resident SDXL container that should have done this work is at replicas=0 and listening on a queue nobody publishes to.

Correct fix

1. Populate ArtStudio:StageModelMap in the API (appsettings.Production.json) so stages route to resident model queues:

"StageModelMap": {
  "concept":  "SDXLLightning",
  "modeling": "TripoSR"
}

(map the remaining stages to Blender / Hunyuan3DPaint / SafetyCheck per the intended pipeline — the ModelDir values above are the deployed residents.)

2. Scale up the residents that stages now route to — at minimum artpipe-model-sdxl (currently replicas=0) for concept art.

3. Then artpipe-modeling is vestigial on this path. Either delete it, or if it is still wanted as a fallback: drop the /mnt/fusionio/spikersoft/art_pipe bind AND the node.labels.artpipe-gpu constraint. A pure router needs no checkout, no GPU, and no node pin — so it can run anywhere and the placement/bind contradiction disappears permanently.

4. artpipe-model-hunyuan should lose its bind too — it is the only resident still carrying one, and its four siblings prove it is not needed. (There appears to be no artpipe-model-hunyuan3dpaint env image built, which is likely why it falls back to the generic processor image + host bind. Worth building the image, cf. #537 for the sdxllightning size problem.)

Note on my earlier advice

I previously flagged this as "the same bug class as #553" and suggested MinIO self-provisioning to replace the bind. That framing was superficially right (a GPU-pinned service carrying a SERVER-only fusionio bind) but the remedy was wrong — it would have preserved a legacy execution path that should simply not be in use. The credit for the right question goes to the repo owner, who asked why a host bind would be needed at all when the model generators are already isolated containers. It should not be, and it is not — we are just not routing to them.

**QA Team — ROOT CAUSE FOUND. This supersedes the fix I proposed above; that advice was wrong.** I originally suggested recreating the `/mnt/fusionio` bind on the 4090 or replacing it with MinIO self-provisioning. **Don't do either.** The bind should not exist at all. The right fix is a one-line config change. ## The processor has two topologies, and we are running the wrong one `ArtPipeProcessor` is one codebase with two deployment modes (`ArtPipeStageConsumer.ResolveBindings`): | Mode | Set by | Binds queue | Executes how | |---|---|---|---| | **Resident** | `ResidentModel=<X>` | `art.model.<x>.tasks` | model baked into its own `artpipe-model-*` image → **needs no host bind** | | **Per-stage (legacy)** | no `ResidentModel` | `art.asset.stage.<s>.requests` | **execs `python -m artpipe.worker` itself** from `/opt/art_pipe` → **needs the host bind** | The C# doc on `PublishToModelQueues` says it outright: ```csharp /// Default FALSE: per-stage mode remains the default topology everywhere until R3/R4. public bool PublishToModelQueues { get; set; } ``` **The resident-container architecture is fully built and fully deployed.** All five model workers are running in resident mode, and four of the five carry **no bind at all** — they bake the art_pipe checkout and venvs into their own images: ``` ResidentModel=SDXLLightning replicas=0/0 queue=art.model.sdxllightning.tasks [no bind] ResidentModel=TripoSR replicas=0/0 queue=art.model.triposr.tasks [no bind] ResidentModel=Blender replicas=0/0 queue=art.model.blender.tasks [no bind] ResidentModel=SafetyCheck replicas=1/1 queue=art.model.safetycheck.tasks [no bind] <-- works today ResidentModel=Hunyuan3DPaint replicas=0/0 queue=art.model.hunyuan3dpaint.tasks [BIND] <-- the odd one out ``` `artpipe-model-safety` is running **right now, 1/1, consuming its model queue, with no host bind whatsoever.** That is the proof that the bind is unnecessary in this topology. ## So why is anything binding fusionio? Because **`artpipe-modeling` has no `ResidentModel`** → it is in **legacy per-stage mode** → it must execute models **in-process as a subprocess** → it needs the art_pipe checkout on disk → hence `/mnt/fusionio/spikersoft/art_pipe`. And it is pinned to `artpipe-gpu` (the 4090), where fusionio does not exist. Hence the permanent reject loop. ## And why does the request go to the legacy path at all? **This is the actual root cause.** `SubmitArtAssetCommandHandler` picks the routing key from `ArtStudio:StageModelMap`: ```csharp // #368 R3: when ArtStudio:StageModelMap maps this stage, the // command's RoutingKey becomes art.model.<model>.task (the // resident instance's task queue) instead of the per-stage key. ResidentModelDir = ResolveResidentModelDir(firstStage, stageMethods), ``` And in **both** `SpikerSoft.Api/appsettings.json` **and** `appsettings.Production.json`: ```json "StageModelMap": {}, ``` **It is empty.** So no stage is ever mapped to a model, so the API always falls back to the per-stage routing key, so every request lands on `art.asset.stage.concept.requests`, whose only possible consumer is the legacy in-process processor — which cannot boot. Meanwhile the five resident containers sit on `art.model.*.tasks` queues that **nothing ever publishes to**. ## The chain, end to end 1. `ArtStudio:StageModelMap` is `{}` → concept stage is unmapped. 2. API publishes with the **per-stage** key → `art.asset.stage.concept.requests`. 3. Only a **per-stage-mode** processor binds that queue — i.e. `artpipe-modeling`. 4. Per-stage mode must exec the model in-process → needs `/opt/art_pipe` → needs the fusionio bind. 5. It is pinned to the 4090; fusionio is SERVER's disk → **reject loop, never starts, 0 consumers.** 6. The resident SDXL container that *should* have done this work is at `replicas=0` and listening on a queue nobody publishes to. ## Correct fix **1. Populate `ArtStudio:StageModelMap` in the API** (`appsettings.Production.json`) so stages route to resident model queues: ```json "StageModelMap": { "concept": "SDXLLightning", "modeling": "TripoSR" } ``` (map the remaining stages to Blender / Hunyuan3DPaint / SafetyCheck per the intended pipeline — the ModelDir values above are the deployed residents.) **2. Scale up the residents that stages now route to** — at minimum `artpipe-model-sdxl` (currently `replicas=0`) for concept art. **3. Then `artpipe-modeling` is vestigial on this path.** Either delete it, or if it is still wanted as a fallback: **drop the `/mnt/fusionio/spikersoft/art_pipe` bind AND the `node.labels.artpipe-gpu` constraint.** A pure router needs no checkout, no GPU, and no node pin — so it can run anywhere and the placement/bind contradiction disappears permanently. **4. `artpipe-model-hunyuan` should lose its bind too** — it is the only resident still carrying one, and its four siblings prove it is not needed. (There appears to be no `artpipe-model-hunyuan3dpaint` env image built, which is likely *why* it falls back to the generic processor image + host bind. Worth building the image, cf. #537 for the sdxllightning size problem.) ## Note on my earlier advice I previously flagged this as "the same bug class as #553" and suggested MinIO self-provisioning to replace the bind. That framing was superficially right (a GPU-pinned service carrying a SERVER-only fusionio bind) but the remedy was wrong — it would have preserved a legacy execution path that should simply not be in use. **The credit for the right question goes to the repo owner, who asked why a host bind would be needed at all when the model generators are already isolated containers. It should not be, and it is not — we are just not routing to them.**
spikerj changed title from [Bug][Prod][ArtPipe] Concept-art generation is silently dead — consumer can NEVER start (artpipe-gpu pin vs SERVER-only /mnt/fusionio bind, same class as #553) and SDXL generator is at replicas=0; live user request stuck unread in queue to [Bug][Prod][ArtPipe] Concept-art is dead because ArtStudio:StageModelMap is EMPTY — API routes to the legacy in-process stage queue whose consumer can't boot, while the 5 resident model containers sit idle on queues nobody publishes to 2026-07-14 21:19:48 +00:00
spikerj commented 2026-07-14 21:24:52 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — correction to my note above re: #537 / the 98 GB image.

The repo owner clarified: the 98 GB figure in #537 was a bug, not the real image size — an unfiltered snapshot_download was pulling the entire HuggingFace repo and all weights by mistake. That has been fixed.

Verified live against the registry — the model env images are all sane now:

artpipe-model-sdxllightning   17.4 GB compressed
artpipe-model-triposr         13.5 GB
artpipe-model-safetycheck     12.5 GB   (running 1/1 today)
artpipe-model-blender          6.5 GB

So do not treat #537 as a blocker on scaling artpipe-model-sdxl up off zero. My earlier caution on that point was based on the stale 98 GB number and is withdrawn.

Capacity is not a concern either. All five residents pin to node.labels.artpipe-gpu (the 4090), which has ample headroom:

/dev/mapper/ubuntu--vg-ubuntu--lv   3.6T total   2.0T used   1.6T avail   57%

~68 GB of model images against 1.6 TB free. Fine.

⚠️ One real caveat on scaling up: pulling a 17.4 GB image currently goes out to the WAN and hairpins back through the router, because git.spikersoft.com resolves to a public IP from inside the cluster (#562). Landing the split-horizon DNS record from #562 first will make that pull go over the LAN instead. Not a blocker, but a 17.4 GB pull is exactly the case where that hairpin hurts most.

**QA Team — correction to my note above re: #537 / the 98 GB image.** The repo owner clarified: the 98 GB figure in **#537** was a **bug, not the real image size** — an unfiltered `snapshot_download` was pulling the entire HuggingFace repo and all weights by mistake. That has been fixed. Verified live against the registry — the model env images are all sane now: ``` artpipe-model-sdxllightning 17.4 GB compressed artpipe-model-triposr 13.5 GB artpipe-model-safetycheck 12.5 GB (running 1/1 today) artpipe-model-blender 6.5 GB ``` So **do not treat #537 as a blocker on scaling `artpipe-model-sdxl` up off zero.** My earlier caution on that point was based on the stale 98 GB number and is withdrawn. Capacity is not a concern either. All five residents pin to `node.labels.artpipe-gpu` (the 4090), which has ample headroom: ``` /dev/mapper/ubuntu--vg-ubuntu--lv 3.6T total 2.0T used 1.6T avail 57% ``` ~68 GB of model images against 1.6 TB free. Fine. ⚠️ One real caveat on scaling up: pulling a 17.4 GB image currently goes **out to the WAN and hairpins back through the router**, because `git.spikersoft.com` resolves to a public IP from inside the cluster (**#562**). Landing the split-horizon DNS record from #562 first will make that pull go over the LAN instead. Not a blocker, but a 17.4 GB pull is exactly the case where that hairpin hurts most.
spikerj commented 2026-07-14 22:37:34 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — live swarm evidence. We think we have the mechanical root cause of the dead concept-art lane, and it is upstream of the StageModelMap config gap.

artpipe-modeling cannot start at all — it has been rejecting a task every ~5 seconds for 27 hours

spikersoft-artpipe-modeling_artpipe-modeling   0/1
Rejected: invalid mount config for type "bind":
          bind source path does not exist: /mnt/fusionio/spikersoft/art_pipe

Verified on the host: /mnt on node 4090 is empty — the fusionio filesystem is not mounted there at all (ls /mnt/fusionio → No such file or directory). The art_pipe tree exists on SERVER (5.3 G), which is where MinIO and every other /mnt/fusionio bind lives.

The service can only ever land on 4090: its placement constraint is node.labels.artpipe-gpu == true, and 4090 is the only node carrying that label.

The churn is unbounded and still running

RestartPolicy: {Condition: on-failure, Delay: 5s, MaxAttempts: 0} — 0 = infinite. We sampled docker service ps twice 12 s apart and got brand-new task IDs each time. UpdateStatus has been paused since 2026-07-13T18:57:51Z — 27h 10m of continuous rejection, which at the observed 5 s cadence is on the order of ~19,000 rejected tasks. service ps only ever shows 5 of them because swarm's task-history-retention-limit is 5, which is why this has been easy to miss.

A second service is armed with the same bomb

spikersoft-artpipe-model-safety is currently 1/1 — but only by luck. Its container was created 27 h ago, before the fusionio mount disappeared, and it carries the same bind path in its spec (it was itself Rejected on that exact path 44 h ago). The moment it restarts for any reason it will crash-loop identically.

How this relates to this ticket

#595 reports concept-art dead with ArtStudio:StageModelMap empty and "the legacy in-process stage queue whose consumer can't boot". We would suggest the consumer can't boot because the container it lives in has never started — the bind rejection precedes any config being read. Fixing StageModelMap alone will not revive the lane while the mount is missing.

Suggested fix direction (two independent problems, both needed)

  1. Mount fusionio on 4090, or move the artpipe-gpu label to a node that has it, or — best, and aligned with epic #413 — migrate the art_pipe tree to MinIO so the bind disappears entirely. Note the 4090 is a part-time burst node (docs/4090-burst-node-runbook.md), so a host-path bind on it is structurally fragile: any service pinned there by label will break whenever the node's local storage differs.
  2. The StageModelMap config gap this ticket already describes.

Related: #562 (node image/mount flakiness), #413 (MinIO migration would remove this bind class entirely).

**QA Team** — live swarm evidence. We think we have the mechanical root cause of the dead concept-art lane, and it is upstream of the `StageModelMap` config gap. ## `artpipe-modeling` cannot start at all — it has been rejecting a task every ~5 seconds for 27 hours ``` spikersoft-artpipe-modeling_artpipe-modeling 0/1 Rejected: invalid mount config for type "bind": bind source path does not exist: /mnt/fusionio/spikersoft/art_pipe ``` Verified on the host: **`/mnt` on node `4090` is empty — the `fusionio` filesystem is not mounted there at all** (`ls /mnt/fusionio` → No such file or directory). The `art_pipe` tree exists on **SERVER** (5.3 G), which is where MinIO and every other `/mnt/fusionio` bind lives. The service can *only* ever land on 4090: its placement constraint is `node.labels.artpipe-gpu == true`, and **4090 is the only node carrying that label**. ## The churn is unbounded and still running `RestartPolicy: {Condition: on-failure, Delay: 5s, MaxAttempts: 0}` — `0` = infinite. We sampled `docker service ps` twice 12 s apart and got **brand-new task IDs each time**. `UpdateStatus` has been `paused` since `2026-07-13T18:57:51Z` — **27h 10m** of continuous rejection, which at the observed 5 s cadence is on the order of **~19,000 rejected tasks**. `service ps` only ever shows 5 of them because swarm's task-history-retention-limit is 5, which is why this has been easy to miss. ## A second service is armed with the same bomb **`spikersoft-artpipe-model-safety` is currently 1/1 — but only by luck.** Its container was created 27 h ago, *before* the fusionio mount disappeared, and it carries the same bind path in its spec (it was itself `Rejected` on that exact path 44 h ago). **The moment it restarts for any reason it will crash-loop identically.** ## How this relates to this ticket #595 reports concept-art dead with `ArtStudio:StageModelMap` empty and "the legacy in-process stage queue whose consumer can't boot". We would suggest the consumer can't boot **because the container it lives in has never started** — the bind rejection precedes any config being read. Fixing `StageModelMap` alone will not revive the lane while the mount is missing. ## Suggested fix direction (two independent problems, both needed) 1. **Mount `fusionio` on 4090**, or move the `artpipe-gpu` label to a node that has it, or — best, and aligned with epic #413 — migrate the `art_pipe` tree to MinIO so the bind disappears entirely. Note the 4090 is a *part-time burst node* (`docs/4090-burst-node-runbook.md`), so a host-path bind on it is structurally fragile: any service pinned there by label will break whenever the node's local storage differs. 2. The `StageModelMap` config gap this ticket already describes. Related: #562 (node image/mount flakiness), #413 (MinIO migration would remove this bind class entirely).
spikerj commented 2026-07-22 02:59:41 +00:00
Author
Owner
Copy Link
Copy Source

Data point (leaving open, can't fully verify from repo): on origin/master the committed SpikerSoft.Api/appsettings.json still has "ArtStudio": { … "StageModelMap": {}, "StageMethodOptions": {} } — i.e. the default is still empty, so unless prod injects a populated map via env/stack overlay, the routing-to-legacy-queue condition this ticket describes persists.

Note there IS now integration coverage for the intended dispatch (spikersoft-issues#465, PR #192: "art pipe messaging coverage — StageModelMap dispatch, consumer bindings, restart routing"), but that tests the mechanism, not the deployed config. To close this, confirm the live ArtStudio:StageModelMap (env/OpenBao/stack overlay in spikersoft-infrastructure) is populated with the 5 resident model stage→queue mappings, not just the empty appsettings default. That's a runtime/config check, not a code read.

Data point (leaving open, can't fully verify from repo): on `origin/master` the committed `SpikerSoft.Api/appsettings.json` still has `"ArtStudio": { … "StageModelMap": {}, "StageMethodOptions": {} }` — i.e. the default is still empty, so unless prod injects a populated map via env/stack overlay, the routing-to-legacy-queue condition this ticket describes persists. Note there IS now integration coverage for the intended dispatch (spikersoft-issues#465, PR #192: "art pipe messaging coverage — StageModelMap dispatch, consumer bindings, restart routing"), but that tests the mechanism, not the deployed config. To close this, confirm the **live** `ArtStudio:StageModelMap` (env/OpenBao/stack overlay in `spikersoft-infrastructure`) is populated with the 5 resident model stage→queue mappings, not just the empty appsettings default. That's a runtime/config check, not a code read.
spikerj commented 2026-07-29 16:09:52 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — the stated root cause is fixed and the routing is now coherent end to end. Leaving open for one live check; see the last paragraph.

The map is no longer empty in production. SpikerSoft.Api/appsettings.Production.json:118 now reads:

"StageModelMap": { "modeling": "TripoSR", "texturing": "Hunyuan3DPaint" }

(The base appsettings.json:422 stays {}, which is correct — an empty map means per-stage routing, byte-identical to pre-R3 behaviour, per the class docs at ArtAssetStageModelMap.cs:18-19.)

A startup guard was added: ServiceCollectionExtensions.cs:1528-1531 binds the map and calls .Validate() before registering it, so a malformed map now fails at boot rather than silently publishing to unbound routing keys.

And — the part that actually resolves the "5 resident containers idle on queues nobody publishes to" symptom — the two halves now agree. Checking the infra side:

  • spikersoft-artpipe-modeling/docker-stack-gpu.yml:84 — ArtPipe__Stages=["concept","modeling","texturing","rigging","animation","export","enrichment"]. So the monolith does bind the concept queue, which is what was missing.
  • :100 — ArtPipe__ModelQueueStages=["modeling","texturing"], exactly matching the API's StageModelMap keys.

So concept flows via per-stage routing (art.asset.stage.requested.concept) to the monolith, while modeling and texturing route to art.model.triposr.tasks / art.model.hunyuan3dpaint.tasks on the resident stacks. Every other artpipe stack has ArtPipe__Stages= empty with a mutually-exclusive ModelQueue, which is the intended shape. No stage is published to an unbound key.

Why I'm not closing it. This is a [Bug][Prod] ticket, and everything above is configuration — it proves the intent is now correct, not that concept art works. Two things gate that:

  1. The monolith runs the prodstages image, and #519's live cutover is still unverified.
  2. #615 means an infra deploy can silently ship a stale stack file, so a green deploy isn't evidence this config is what's running.

I've been careful not to close deploy-state tickets on code alone this week — that's precisely the mistake #602 and #582 exist to record.

To close: submit one concept-stage generation in prod and confirm it completes. If it does, this and a chunk of #519 close together.

Audited against `origin/master` — **the stated root cause is fixed and the routing is now coherent end to end.** Leaving open for one live check; see the last paragraph. **The map is no longer empty in production.** `SpikerSoft.Api/appsettings.Production.json:118` now reads: ```json "StageModelMap": { "modeling": "TripoSR", "texturing": "Hunyuan3DPaint" } ``` (The base `appsettings.json:422` stays `{}`, which is correct — an empty map means per-stage routing, byte-identical to pre-R3 behaviour, per the class docs at `ArtAssetStageModelMap.cs:18-19`.) **A startup guard was added:** `ServiceCollectionExtensions.cs:1528-1531` binds the map and calls `.Validate()` before registering it, so a malformed map now fails at boot rather than silently publishing to unbound routing keys. **And — the part that actually resolves the "5 resident containers idle on queues nobody publishes to" symptom — the two halves now agree.** Checking the infra side: - `spikersoft-artpipe-modeling/docker-stack-gpu.yml:84` — `ArtPipe__Stages=["concept","modeling","texturing","rigging","animation","export","enrichment"]`. So the monolith **does** bind the concept queue, which is what was missing. - `:100` — `ArtPipe__ModelQueueStages=["modeling","texturing"]`, exactly matching the API's `StageModelMap` keys. So concept flows via per-stage routing (`art.asset.stage.requested.concept`) to the monolith, while modeling and texturing route to `art.model.triposr.tasks` / `art.model.hunyuan3dpaint.tasks` on the resident stacks. Every other artpipe stack has `ArtPipe__Stages=` empty with a mutually-exclusive `ModelQueue`, which is the intended shape. No stage is published to an unbound key. **Why I'm not closing it.** This is a `[Bug][Prod]` ticket, and everything above is configuration — it proves the *intent* is now correct, not that concept art works. Two things gate that: 1. The monolith runs the `prodstages` image, and **#519**'s live cutover is still unverified. 2. **#615** means an infra deploy can silently ship a stale stack file, so a green deploy isn't evidence this config is what's running. I've been careful not to close deploy-state tickets on code alone this week — that's precisely the mistake #602 and #582 exist to record. **To close:** submit one concept-stage generation in prod and confirm it completes. If it does, this and a chunk of #519 close together.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#595