spikerj/spikersoft-issues
1
0 Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Prod] Art Studio 'modeling' stage fails — TripoSG model manifest missing on prod host #455

New Issue
Closed
opened 2026-07-07 22:10:24 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-07 22:10:24 +00:00
Owner
Copy link
Copy source

Symptom (production, learn.spikersoft.com)

The modeling stage fails for assets:

SpikerSoft.EventHandlers.ArtPipeProcessor.Services.ArtPipeStageException:
  Model manifest not found: /opt/art_pipe/models/TripoSG/artpipe.json
  at SubprocessArtPipeStageExecutor.ExecuteAsync(...):line 42
  at ArtPipeStageOrchestrator.ExecuteStageAsync(...):line 318
Stage 'modeling' failed for asset 6a4b0e76e3470f45ff654597   (07 Jul 2026 14:18:03)

Assets die at modeling, so no mesh / texture / rig / animation / export artifacts are ever produced → the Art Studio asset-detail 3D viewer and downloads have nothing to show. This is the likely root cause behind the "can't see/preview produced assets" report.

Root cause — prod provisioning gap, not a code bug

SubprocessArtPipeStageExecutor hard-requires {PipelineRoot}/models/{ModelDir}/artpipe.json (SubprocessArtPipeStageExecutor.cs:39-42) and fails fast if absent. Config maps ArtPipe:StageSettings:modeling.ModelDir = "TripoSG" (appsettings.json, in place since #348) and prod PipelineRoot = /opt/art_pipe. TripoSG (manifest + venv + weights) is not installed at /opt/art_pipe/models/TripoSG/ on the prod GPU worker.

Confirmed present in the dev art_pipe checkout (models/TripoSG/artpipe.json exists locally, alongside InstantMesh / TripoSR / Hunyuan3D / SF3D / ShapE / etc.), so the executor is behaving correctly — the model just isn't vendored on prod.

Remediation (pick one)

  1. Preferred: install/vendor TripoSG on the prod GPU worker at /opt/art_pipe/models/TripoSG/ (manifest + venv + weights), same as the other installed models. Ties into #432 (repo-vendoring model venvs) and #425 (full local pipeline).
  2. Interim workaround: repoint modeling.ModelDir to an image_to_3d-capable model that is installed on prod (InstantMesh / TripoSR / Hunyuan3D all expose image_to_3d) in appsettings.json StageSettings, then redeploy the ArtPipeProcessor.

Triage aid

On the GPU worker: ls /opt/art_pipe/models/*/artpipe.json to confirm what's actually vendored vs the StageSettings mapping (concept→SDXLLightning, modeling→TripoSG, texturing→Hunyuan3DPaint, rigging/animation/export→Blender). Any other stage whose configured ModelDir isn't installed will fail identically.

Related: #445 (restart-from-stage modeling 400), #432, #425. The separate Art-Studio-side UX gap (detail 3D viewer doesn't auto-load a model) is being fixed in an angular PR referencing #446.

## Symptom (production, learn.spikersoft.com) The `modeling` stage fails for assets: ``` SpikerSoft.EventHandlers.ArtPipeProcessor.Services.ArtPipeStageException: Model manifest not found: /opt/art_pipe/models/TripoSG/artpipe.json at SubprocessArtPipeStageExecutor.ExecuteAsync(...):line 42 at ArtPipeStageOrchestrator.ExecuteStageAsync(...):line 318 Stage 'modeling' failed for asset 6a4b0e76e3470f45ff654597 (07 Jul 2026 14:18:03) ``` Assets die at `modeling`, so **no mesh / texture / rig / animation / export artifacts are ever produced** → the Art Studio asset-detail 3D viewer and downloads have nothing to show. This is the likely root cause behind the "can't see/preview produced assets" report. ## Root cause — prod provisioning gap, not a code bug `SubprocessArtPipeStageExecutor` hard-requires `{PipelineRoot}/models/{ModelDir}/artpipe.json` (`SubprocessArtPipeStageExecutor.cs:39-42`) and fails fast if absent. Config maps `ArtPipe:StageSettings:modeling.ModelDir = "TripoSG"` (`appsettings.json`, in place since #348) and prod `PipelineRoot = /opt/art_pipe`. **TripoSG (manifest + venv + weights) is not installed at `/opt/art_pipe/models/TripoSG/` on the prod GPU worker.** Confirmed present in the dev art_pipe checkout (`models/TripoSG/artpipe.json` exists locally, alongside InstantMesh / TripoSR / Hunyuan3D / SF3D / ShapE / etc.), so the executor is behaving correctly — the model just isn't vendored on prod. ## Remediation (pick one) 1. **Preferred:** install/vendor TripoSG on the prod GPU worker at `/opt/art_pipe/models/TripoSG/` (manifest + venv + weights), same as the other installed models. Ties into #432 (repo-vendoring model venvs) and #425 (full local pipeline). 2. **Interim workaround:** repoint `modeling.ModelDir` to an `image_to_3d`-capable model that *is* installed on prod (InstantMesh / TripoSR / Hunyuan3D all expose `image_to_3d`) in `appsettings.json` `StageSettings`, then redeploy the ArtPipeProcessor. ## Triage aid On the GPU worker: `ls /opt/art_pipe/models/*/artpipe.json` to confirm what's actually vendored vs the `StageSettings` mapping (concept→SDXLLightning, modeling→**TripoSG**, texturing→Hunyuan3DPaint, rigging/animation/export→Blender). Any other stage whose configured `ModelDir` isn't installed will fail identically. Related: #445 (restart-from-stage modeling 400), #432, #425. The separate Art-Studio-side UX gap (detail 3D viewer doesn't auto-load a model) is being fixed in an angular PR referencing #446.
spikerj commented 2026-07-08 22:48:21 +00:00
Author
Owner
Copy link
Copy source

Status roll-up — the interim remediation is in flight, and this ticket now also carries the prod-vendoring residual from #456.

Remediation 2 (interim, in flight): the #368 cutover PRs repoint modeling to TripoSR (installed on prod) via Production overlays — spikersoft-backend PR #186 (StageSettings:modeling.ModelDir=TripoSR in the worker's appsettings.Production.json + ArtStudio:StageModelMap modeling→TripoSR in the API's). Once that merges and deploys, the modeling stage stops dying at the missing TripoSG manifest.

Remediation 1 (preferred, stays open here): vendor TripoSG on SERVER and repoint modeling back. The exact ops steps are now written down in spikersoft-infrastructure PR #13's docs/artpipe-server-cutover-runbook.md — §1 (git pull the /mnt/fusionio/spikersoft/art_pipe checkout; the prod checkout predates models/TripoSG, which is this bug), §2 (python3 bootstrap.py --model TripoSG inside a worker container), §6 (repoint StageModelMap modeling→TripoSG + add a spikersoft-artpipe-model-triposg resident stack).

New residual moved here from #456 (now closed): ShapE must be vendored on the SERVER mount the same way (bootstrap.py --model ShapE) before the shipped text→3D path produces meshes in prod. Same class of gap, same runbook procedure — one extra bootstrap line while an operator is already in there.

Host-side checklist to close this ticket:

  • git pull the SERVER art_pipe checkout
  • bootstrap.py --model TripoSG (venv + weights)
  • bootstrap.py --model ShapE (venv + weights — #456 residual)
  • repoint modeling→TripoSG (StageModelMap + worker overlay) and add the triposg resident stack
  • verify: modeling produces GridFS artifacts on prod; a Text→3D submission produces a mesh
Status roll-up — the interim remediation is in flight, and this ticket now also carries the prod-vendoring residual from #456. **Remediation 2 (interim, in flight):** the #368 cutover PRs repoint `modeling` to **TripoSR** (installed on prod) via Production overlays — spikersoft-backend PR #186 (`StageSettings:modeling.ModelDir=TripoSR` in the worker's `appsettings.Production.json` + `ArtStudio:StageModelMap` modeling→TripoSR in the API's). Once that merges and deploys, the modeling stage stops dying at the missing TripoSG manifest. **Remediation 1 (preferred, stays open here):** vendor **TripoSG** on SERVER and repoint modeling back. The exact ops steps are now written down in spikersoft-infrastructure PR #13's `docs/artpipe-server-cutover-runbook.md` — §1 (`git pull` the `/mnt/fusionio/spikersoft/art_pipe` checkout; the prod checkout predates `models/TripoSG`, which is this bug), §2 (`python3 bootstrap.py --model TripoSG` inside a worker container), §6 (repoint `StageModelMap` modeling→TripoSG + add a `spikersoft-artpipe-model-triposg` resident stack). **New residual moved here from #456 (now closed):** **ShapE** must be vendored on the SERVER mount the same way (`bootstrap.py --model ShapE`) before the shipped text→3D path produces meshes in prod. Same class of gap, same runbook procedure — one extra bootstrap line while an operator is already in there. Host-side checklist to close this ticket: - [ ] `git pull` the SERVER art_pipe checkout - [ ] `bootstrap.py --model TripoSG` (venv + weights) - [ ] `bootstrap.py --model ShapE` (venv + weights — #456 residual) - [ ] repoint modeling→TripoSG (`StageModelMap` + worker overlay) and add the triposg resident stack - [ ] verify: modeling produces GridFS artifacts on prod; a Text→3D submission produces a mesh
spikerj commented 2026-08-07 13:55:52 +00:00
Author
Owner
Copy link
Copy source

Verified obsolete 2026-08-07 — closing as no-longer-applicable.

What removed the subject: the entire provisioning model this ticket describes — a shared art_pipe checkout at /mnt/fusionio/spikersoft/art_pipe bind-mounted to /opt/art_pipe, with models vendored per host by git pull + bootstrap.py --model <X> — was replaced by the baked per-model images of epic #515. Each model now ships inside its own image with its manifest, venv and weights baked in, so "manifest missing on the prod host" is no longer a state the system can reach.

  • Code: spikersoft-infrastructure@86d03ff — spikersoft-artpipe-modeling/docker-stack-gpu.yml runs git.spikersoft.com/spikerj/artpipe-model-prodstages:latest with HF_HUB_OFFLINE=1; the fusionio bind survives only as prose at :36 and :45 documenting the backout path (infra PR #68).
  • Live: docker service inspect spikersoft-artpipe-modeling_artpipe-modeling shows Mounts containing only /etc/timezone and /etc/localtime — there is no host art_pipe checkout in the spec at all. SubprocessArtPipeStageExecutor's {PipelineRoot}/models/{ModelDir}/artpipe.json requirement is now satisfied from inside the image.
  • Both host-checklist items are moot, and both models are live as their own lanes: spikersoft-artpipe-model-triposg runs 1/1 from artpipe-model-triposg:latest (ArtPipeStageConsumer starting in MODEL-QUEUE mode (subprocess per job) for model TripoSG) — so the bootstrap.py --model TripoSG step is unnecessary. The #456 ShapE residual is likewise superseded: spikersoft-artpipe-model-shape is a deployed stack on the baked artpipe-model-shape:latest, and the text→3D path additionally has spikersoft-artpipe-model-textto3d running 1/1. TripoSG is also selectable directly by students — ArtStudioStageMethodOptions.cs:153 carries Key = "triposg".

The one live descendant, deliberately not closed with this: modeling's default is still TripoSR (SpikerSoft.Api/appsettings.Production.json:118), not TripoSG. That is now a one-line StageModelMap decision constrained by VRAM per lane, not a provisioning gap, and it is already carried as item 6 of the Phase-3 ticket — migrated to spikerj/spikersoft-infrastructure#192.

Not migrated: the bug as described cannot recur, and its residual lives on #192.

— Opus 5 Agent

Verified obsolete 2026-08-07 — closing as no-longer-applicable. **What removed the subject:** the entire provisioning model this ticket describes — a shared art_pipe checkout at `/mnt/fusionio/spikersoft/art_pipe` bind-mounted to `/opt/art_pipe`, with models vendored per host by `git pull` + `bootstrap.py --model <X>` — was replaced by the baked per-model images of epic #515. Each model now ships inside its own image with its manifest, venv and weights baked in, so "manifest missing on the prod host" is no longer a state the system can reach. - **Code:** `spikersoft-infrastructure@86d03ff` — `spikersoft-artpipe-modeling/docker-stack-gpu.yml` runs `git.spikersoft.com/spikerj/artpipe-model-prodstages:latest` with `HF_HUB_OFFLINE=1`; the fusionio bind survives only as prose at `:36` and `:45` documenting the backout path (infra PR #68). - **Live:** `docker service inspect spikersoft-artpipe-modeling_artpipe-modeling` shows `Mounts` containing **only** `/etc/timezone` and `/etc/localtime` — there is no host art_pipe checkout in the spec at all. `SubprocessArtPipeStageExecutor`'s `{PipelineRoot}/models/{ModelDir}/artpipe.json` requirement is now satisfied from inside the image. - **Both host-checklist items are moot, and both models are live as their own lanes:** `spikersoft-artpipe-model-triposg` runs 1/1 from `artpipe-model-triposg:latest` (`ArtPipeStageConsumer starting in MODEL-QUEUE mode (subprocess per job) for model TripoSG`) — so the `bootstrap.py --model TripoSG` step is unnecessary. The #456 ShapE residual is likewise superseded: `spikersoft-artpipe-model-shape` is a deployed stack on the baked `artpipe-model-shape:latest`, and the text→3D path additionally has `spikersoft-artpipe-model-textto3d` running 1/1. TripoSG is also selectable directly by students — `ArtStudioStageMethodOptions.cs:153` carries `Key = "triposg"`. **The one live descendant, deliberately not closed with this:** modeling's *default* is still `TripoSR` (`SpikerSoft.Api/appsettings.Production.json:118`), not TripoSG. That is now a one-line `StageModelMap` decision constrained by VRAM per lane, not a provisioning gap, and it is already carried as item 6 of the Phase-3 ticket — migrated to **spikerj/spikersoft-infrastructure#192**. Not migrated: the bug as described cannot recur, and its residual lives on #192. — Opus 5 Agent
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 epic
Cross-repo epic — children live in per-repo trackers
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#455