spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[ArtPipe] Baked images Phase 3: prodstages image + live modeling-lane cutover (epic #515) #519

New Issue
Open
opened 2026-07-13 00:40:19 +00:00 by spikerj · 5 comments
spikerj commented 2026-07-13 00:40:19 +00:00
Owner
Copy Link
Copy Source

Parent epic: #515. Blocked by Phase 2. Highest blast radius — quiet window.

Scope: the live per-stage lane spikersoft-artpipe-modeling (all seven stages, per-task leases — the primary prod topology).

  1. Build artpipe-model-env-prodstages with MODELS="SDXLLightning TripoSR SDTurboTex Blender" — exactly the prod StageSettings map (concept→SDXLLightning, modeling→TripoSR per #455 workaround, texturing→SDTurboTex, rig/anim/export/enrichment→Blender; safety is remote). Add cross-reference comments in both appsettings.Production.json StageSettings and the workflow MODELS line — a stage remap without an image rebuild = missing model dir at runtime.
  2. Backend workflow builds spikerj/artpipe-model-prodstages.
  3. Fresh-pull test all four venvs + manifests present.
  4. Infra PR: spikersoft-artpipe-modeling/docker-stack-gpu.yml image swap + drop bind + HF_HUB_OFFLINE=1. Redeploy in a quiet window.
  5. Full E2E generation (concept → export → enrichment) with offline hub — this is what surfaces straggler downloads setup never pre-fetched; fix by extending manifest install.weights/env.checkpoints entries and rebuilding.
  6. Soak, then: with TripoSG baked into a future image, modeling can return to TripoSG during 4090 sessions (#455 proper fix; VRAM won't fit the 3070 Ti so SERVER stays TripoSR).

Backout: revert stack file to legacy image + bind (still building + intact).

Parent epic: #515. Blocked by Phase 2. **Highest blast radius — quiet window.** **Scope:** the live per-stage lane `spikersoft-artpipe-modeling` (all seven stages, per-task leases — the primary prod topology). 1. Build `artpipe-model-env-prodstages` with `MODELS="SDXLLightning TripoSR SDTurboTex Blender"` — exactly the prod StageSettings map (concept→SDXLLightning, modeling→TripoSR per #455 workaround, texturing→SDTurboTex, rig/anim/export/enrichment→Blender; safety is remote). **Add cross-reference comments** in both `appsettings.Production.json` StageSettings and the workflow MODELS line — a stage remap without an image rebuild = missing model dir at runtime. 2. Backend workflow builds `spikerj/artpipe-model-prodstages`. 3. Fresh-pull test all four venvs + manifests present. 4. Infra PR: `spikersoft-artpipe-modeling/docker-stack-gpu.yml` image swap + drop bind + `HF_HUB_OFFLINE=1`. Redeploy in a quiet window. 5. Full E2E generation (concept → export → enrichment) with offline hub — this is what surfaces straggler downloads `setup` never pre-fetched; fix by extending manifest `install.weights`/`env.checkpoints` entries and rebuilding. 6. Soak, then: with TripoSG baked into a future image, modeling can return to TripoSG during 4090 sessions (#455 proper fix; VRAM won't fit the 3070 Ti so SERVER stays TripoSR). Backout: revert stack file to legacy image + bind (still building + intact).
spikerj commented 2026-07-14 03:27:30 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — active blocker found 2026-07-14 ~03:35Z: spikersoft-artpipe-modeling_artpipe-modeling is in a hard reject-loop and the modeling lane is down.

  • Every ~5s since the 18:57Z Jul 13 spec update (8+ hours, thousands of attempts): invalid mount config for type "bind": bind source path does not exist: /mnt/fusionio/spikersoft/art_pipe on node 4090, and UpdateStatus: paused (update paused due to failure).
  • Root cause is a placement/spec mismatch: /mnt/fusionio doesn't exist at all on the 4090 node (verified — it's SERVER-local storage). The service now schedules on 4090 (GPU lane) but its spec still carries the legacy art_pipe bind, so it can never start there. Phase-3/Phase-4 (#520) intends to retire that bind — the reject-loop says the placement moved before the bind did.
  • Collateral: the backend CI deploy job for run 6613 failed at 02:24:56Z (spikersoft-artpipe-processor.yml, the #548-fix merge) — docker service update can't converge against the paused/rejecting service, so every artpipe-processor deploy will keep failing until this is resolved.

Fix options: drop the legacy bind from the spec now (if Phase 3's baked image no longer needs it — likely the intent), or constrain the service back to SERVER until the bind retires. Either unblocks deploys immediately.

**QA Team** — active blocker found 2026-07-14 ~03:35Z: `spikersoft-artpipe-modeling_artpipe-modeling` is in a hard reject-loop and the modeling lane is down. - Every ~5s since the **18:57Z Jul 13** spec update (8+ hours, thousands of attempts): `invalid mount config for type "bind": bind source path does not exist: /mnt/fusionio/spikersoft/art_pipe` on node **4090**, and `UpdateStatus: paused (update paused due to failure)`. - Root cause is a placement/spec mismatch: **`/mnt/fusionio` doesn't exist at all on the 4090 node** (verified — it's SERVER-local storage). The service now schedules on 4090 (GPU lane) but its spec still carries the legacy art_pipe bind, so it can never start there. Phase-3/Phase-4 (#520) intends to retire that bind — the reject-loop says the placement moved before the bind did. - Collateral: the backend CI **deploy job for run 6613 failed at 02:24:56Z** (spikersoft-artpipe-processor.yml, the #548-fix merge) — `docker service update` can't converge against the paused/rejecting service, so every artpipe-processor deploy will keep failing until this is resolved. **Fix options:** drop the legacy bind from the spec now (if Phase 3's baked image no longer needs it — likely the intent), or constrain the service back to SERVER until the bind retires. Either unblocks deploys immediately.
spikerj commented 2026-07-14 04:35:58 +00:00
Author
Owner
Copy Link
Copy Source

Blocked on the registry data path, not on code. Status 2026-07-14 ~04:35Z:

✅ Env image done: artpipe-model-env-prodstages:latest / :44daa91 pushed 04:05Z. All four stage venvs verified inside the image at build time (SDXLLightning, TripoSR, SDTurboTex on py3.10.20; Blender = server-interpreter, no venv) — that is gate 2 of the cutover PR satisfied by construction.
❌ Tier-3 image artpipe-model-prodstages NOT built yet — two consecutive ArtPipeProcessor runs died on registry transfers:

  • run 6624: docker push retried one blob for ~14 min, then gave up (it was competing with the concurrent env-image push).
  • run 6691: unexpected EOF 13 min into docker pull artpipe-model-env-triposr.

Both are the same failure: multi-GB transfers dropping mid-flight on a saturated registry route (#538 — the write path currently leaves the LAN and comes back). This is no longer just slow, it is failing CI.

Unblock order (not retrying into the same wall):

  1. Merge the two build-set PRs (backend #273 / artpipe #20) — the default set becomes safetycheck + prodstages, i.e. what actually runs. Run 6691 died pulling triposr's env image for a stack at replicas: 0; with those merged it would never have touched it.
  2. Merge + deploy the route fix (private infra PR) and ideally the host-resolver half of #538, so multi-GB blobs stop crossing the gateway.
  3. Then ONE targeted workflow_dispatch with images: prodstages → tier-3 image exists → the gated modeling-lane cutover PR can proceed (its remaining gates: fresh-pull venv test, quiet-window E2E).

Nothing in prod is affected by these failures — the deploy jobs skipped; running services untouched. safetycheck/blender/sdxllightning did get fresh tier-3 images at eca5877-44daa91.

— macbook-claude-session

**Blocked on the registry data path, not on code.** Status 2026-07-14 ~04:35Z: ✅ **Env image done**: `artpipe-model-env-prodstages:latest` / `:44daa91` pushed 04:05Z. All four stage venvs verified inside the image at build time (SDXLLightning, TripoSR, SDTurboTex on py3.10.20; Blender = server-interpreter, no venv) — that is gate 2 of the cutover PR satisfied by construction. ❌ **Tier-3 image `artpipe-model-prodstages` NOT built yet** — two consecutive ArtPipeProcessor runs died on registry transfers: - run 6624: `docker push` retried one blob for ~14 min, then gave up (it was competing with the concurrent env-image push). - run 6691: `unexpected EOF` 13 min into `docker pull artpipe-model-env-triposr`. Both are the same failure: multi-GB transfers dropping mid-flight on a saturated registry route (#538 — the write path currently leaves the LAN and comes back). This is no longer just slow, it is **failing CI**. **Unblock order (not retrying into the same wall):** 1. Merge the two build-set PRs (backend #273 / artpipe #20) — the default set becomes safetycheck + prodstages, i.e. what actually runs. Run 6691 died pulling **triposr's env image for a stack at replicas: 0**; with those merged it would never have touched it. 2. Merge + deploy the route fix (private infra PR) and ideally the host-resolver half of #538, so multi-GB blobs stop crossing the gateway. 3. Then ONE targeted `workflow_dispatch` with `images: prodstages` → tier-3 image exists → the gated modeling-lane cutover PR can proceed (its remaining gates: fresh-pull venv test, quiet-window E2E). Nothing in prod is affected by these failures — the deploy jobs skipped; running services untouched. safetycheck/blender/sdxllightning did get fresh tier-3 images at `eca5877-44daa91`. — macbook-claude-session
spikerj commented 2026-07-14 04:39:48 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — user-facing impact confirmed 2026-07-14 ~04:40Z, escalating the earlier reject-loop finding:

A live Art Studio concept text→image generation was submitted at 04:37:26Z: the API published RequestArtAssetStageCommand to art.asset.lifecycle/art.asset.stage.requested.concept (asset 6a52e856…, and note it's already retry 5 for this asset) — and nothing consumed it. The concept-stage consumer is the artpipe-modeling/artpipe-processor service that's been reject-looping on the missing /mnt/fusionio/spikersoft/art_pipe bind since 18:57Z. The only artpipe container actually running is the safety-checker (answering /healthz every 30s).

From the user's perspective this fails silently: Art Studio activity feed shows active=0 running=0, no error, the job just never appears — while the backend quietly re-publishes retries. So this isn't only a CI/deploy blocker anymore: concept generation has been down for ~9.5 hours and the GPU contention test we wanted (art vs captions vs quiz) can't even begin.

Same two immediate fix options as before (drop the legacy bind from the spec, or constrain to SERVER) — either restores the consumer.

**QA Team** — user-facing impact confirmed 2026-07-14 ~04:40Z, escalating the earlier reject-loop finding: A live Art Studio **concept text→image generation** was submitted at 04:37:26Z: the API published `RequestArtAssetStageCommand` to `art.asset.lifecycle/art.asset.stage.requested.concept` (asset `6a52e856…`, and note it's already **retry 5** for this asset) — and **nothing consumed it**. The concept-stage consumer is the `artpipe-modeling`/artpipe-processor service that's been reject-looping on the missing `/mnt/fusionio/spikersoft/art_pipe` bind since 18:57Z. The only artpipe container actually running is the safety-checker (answering /healthz every 30s). From the user's perspective this fails **silently**: Art Studio activity feed shows `active=0 running=0`, no error, the job just never appears — while the backend quietly re-publishes retries. So this isn't only a CI/deploy blocker anymore: **concept generation has been down for ~9.5 hours** and the GPU contention test we wanted (art vs captions vs quiz) can't even begin. Same two immediate fix options as before (drop the legacy bind from the spec, or constrain to SERVER) — either restores the consumer.
spikerj commented 2026-07-22 03:57:51 +00:00
Author
Owner
Copy Link
Copy Source

Board-sweep status (2026-07-22): code side of Phase 3 is fully merged — backend #269 (prodstages joins the tier-3 build set + StageSettings lockstep), #276/#277 (registry-hairpin resilience + workflow self-trigger), artpipe build-set trim (env images only for models that run), and the GATED infra cutover PR #68 (modeling lane → baked artpipe-model-prodstages, fusionio bind dropped). What remains is operational, not code: getting a tier-3 artpipe-model-prodstages image reliably into the registry (the superseded/cancelled-builds problem is now tracked as #717, and #595 says the concept lane is dead for a related config reason) and then activating the gated cutover with its verification gates (fresh-pull venv test, quiet-window E2E). Leaving open — this closes on the live modeling-lane cutover, which can't be verified from code.

Board-sweep status (2026-07-22): **code side of Phase 3 is fully merged** — backend #269 (prodstages joins the tier-3 build set + StageSettings lockstep), #276/#277 (registry-hairpin resilience + workflow self-trigger), artpipe build-set trim (env images only for models that run), and the GATED infra cutover PR #68 (modeling lane → baked `artpipe-model-prodstages`, fusionio bind dropped). **What remains is operational, not code:** getting a tier-3 `artpipe-model-prodstages` image reliably into the registry (the superseded/cancelled-builds problem is now tracked as #717, and #595 says the concept lane is dead for a related config reason) and then activating the gated cutover with its verification gates (fresh-pull venv test, quiet-window E2E). Leaving open — this closes on the live modeling-lane cutover, which can't be verified from code.
spikerj commented 2026-07-29 16:00:48 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — the blocking gate has cleared; only the live cutover verification remains. Staying open on that.

The tier-3 image now exists, which the 2026-07-22 board sweep named as the blocker. A package listing for spikerj/artpipe-model-prodstages returns 4 versions, newest latest and bba34a2-837a14a created 2026-07-29T15:43Z (plus 766cb8e-837a14a and e8bf57e-837a14a earlier the same day). So it's building successfully now, not just once.

The infra cutover is merged (PR #68): spikersoft-artpipe-modeling/docker-stack-gpu.yml:65 — image: git.spikersoft.com/spikerj/artpipe-model-prodstages:latest; :79 — HF_HUB_OFFLINE=1. The /mnt/fusionio/spikersoft/art_pipe bind is gone from the service spec — the only remaining fusionio mentions are prose at :36 and :45 documenting the backout path, which is the right thing to keep.

Remaining — ops, not code (items 3/5/6):

  1. Fresh-pull four-venv/manifest test.
  2. Quiet-window end-to-end concept → export → enrichment run against the offline hub.
  3. Soak.

Git cannot show whether the swarm service is actually running the prodstages image. Settling evidence: docker service inspect spikersoft-artpipe-modeling_artpipe-modeling showing the prodstages image, plus one successful end-to-end generation.

⚠️ Check #615 before trusting a deploy here. Root has no git credentials, so sudo git -C /mnt/infrastructure pull fails and the || echo "WARNING…" swallows it — meaning an infra deploy can report success while shipping the previous stack file. Since this ticket's whole change lives in a stack file, a green deploy is not evidence the cutover happened. Worth confirming the running image directly rather than inferring from the deploy result.

One item I did not independently verify: acceptance item 1 asks for cross-reference comments in ArtPipeProcessor/appsettings.Production.json StageSettings and both workflows' IMAGES/MODELS lines. The stack header at :33-41 documents the lockstep and points at them, but I didn't confirm the counterpart comments exist.

Audited against `origin/master` — **the blocking gate has cleared; only the live cutover verification remains.** Staying open on that. **The tier-3 image now exists**, which the 2026-07-22 board sweep named as the blocker. A package listing for `spikerj/artpipe-model-prodstages` returns 4 versions, newest `latest` and `bba34a2-837a14a` created **2026-07-29T15:43Z** (plus `766cb8e-837a14a` and `e8bf57e-837a14a` earlier the same day). So it's building successfully now, not just once. **The infra cutover is merged** (PR #68): `spikersoft-artpipe-modeling/docker-stack-gpu.yml:65` — `image: git.spikersoft.com/spikerj/artpipe-model-prodstages:latest`; `:79` — `HF_HUB_OFFLINE=1`. The `/mnt/fusionio/spikersoft/art_pipe` bind is **gone from the service spec** — the only remaining `fusionio` mentions are prose at `:36` and `:45` documenting the backout path, which is the right thing to keep. **Remaining — ops, not code (items 3/5/6):** 1. Fresh-pull four-venv/manifest test. 2. Quiet-window end-to-end concept → export → enrichment run against the offline hub. 3. Soak. Git cannot show whether the swarm service is actually running the prodstages image. Settling evidence: `docker service inspect spikersoft-artpipe-modeling_artpipe-modeling` showing the prodstages image, plus one successful end-to-end generation. ⚠️ **Check #615 before trusting a deploy here.** Root has no git credentials, so `sudo git -C /mnt/infrastructure pull` fails and the `|| echo "WARNING…"` swallows it — meaning an infra deploy can report success while shipping the *previous* stack file. Since this ticket's whole change lives in a stack file, a green deploy is not evidence the cutover happened. Worth confirming the running image directly rather than inferring from the deploy result. One item I did not independently verify: acceptance item 1 asks for cross-reference comments in `ArtPipeProcessor/appsettings.Production.json` StageSettings and both workflows' `IMAGES`/`MODELS` lines. The stack header at `:33-41` documents the lockstep and points at them, but I didn't confirm the counterpart comments exist.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#519