spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Prod] Art Studio 'modeling' stage fails — TripoSG model manifest missing on prod host #455

New Issue
Open
opened 2026-07-07 22:10:24 +00:00 by spikerj · 1 comment
spikerj commented 2026-07-07 22:10:24 +00:00
Owner
Copy Link
Copy Source

Symptom (production, learn.spikersoft.com)

The modeling stage fails for assets:

SpikerSoft.EventHandlers.ArtPipeProcessor.Services.ArtPipeStageException:
  Model manifest not found: /opt/art_pipe/models/TripoSG/artpipe.json
  at SubprocessArtPipeStageExecutor.ExecuteAsync(...):line 42
  at ArtPipeStageOrchestrator.ExecuteStageAsync(...):line 318
Stage 'modeling' failed for asset 6a4b0e76e3470f45ff654597   (07 Jul 2026 14:18:03)

Assets die at modeling, so no mesh / texture / rig / animation / export artifacts are ever produced → the Art Studio asset-detail 3D viewer and downloads have nothing to show. This is the likely root cause behind the "can't see/preview produced assets" report.

Root cause — prod provisioning gap, not a code bug

SubprocessArtPipeStageExecutor hard-requires {PipelineRoot}/models/{ModelDir}/artpipe.json (SubprocessArtPipeStageExecutor.cs:39-42) and fails fast if absent. Config maps ArtPipe:StageSettings:modeling.ModelDir = "TripoSG" (appsettings.json, in place since #348) and prod PipelineRoot = /opt/art_pipe. TripoSG (manifest + venv + weights) is not installed at /opt/art_pipe/models/TripoSG/ on the prod GPU worker.

Confirmed present in the dev art_pipe checkout (models/TripoSG/artpipe.json exists locally, alongside InstantMesh / TripoSR / Hunyuan3D / SF3D / ShapE / etc.), so the executor is behaving correctly — the model just isn't vendored on prod.

Remediation (pick one)

  1. Preferred: install/vendor TripoSG on the prod GPU worker at /opt/art_pipe/models/TripoSG/ (manifest + venv + weights), same as the other installed models. Ties into #432 (repo-vendoring model venvs) and #425 (full local pipeline).
  2. Interim workaround: repoint modeling.ModelDir to an image_to_3d-capable model that is installed on prod (InstantMesh / TripoSR / Hunyuan3D all expose image_to_3d) in appsettings.json StageSettings, then redeploy the ArtPipeProcessor.

Triage aid

On the GPU worker: ls /opt/art_pipe/models/*/artpipe.json to confirm what's actually vendored vs the StageSettings mapping (concept→SDXLLightning, modeling→TripoSG, texturing→Hunyuan3DPaint, rigging/animation/export→Blender). Any other stage whose configured ModelDir isn't installed will fail identically.

Related: #445 (restart-from-stage modeling 400), #432, #425. The separate Art-Studio-side UX gap (detail 3D viewer doesn't auto-load a model) is being fixed in an angular PR referencing #446.

## Symptom (production, learn.spikersoft.com) The `modeling` stage fails for assets: ``` SpikerSoft.EventHandlers.ArtPipeProcessor.Services.ArtPipeStageException: Model manifest not found: /opt/art_pipe/models/TripoSG/artpipe.json at SubprocessArtPipeStageExecutor.ExecuteAsync(...):line 42 at ArtPipeStageOrchestrator.ExecuteStageAsync(...):line 318 Stage 'modeling' failed for asset 6a4b0e76e3470f45ff654597 (07 Jul 2026 14:18:03) ``` Assets die at `modeling`, so **no mesh / texture / rig / animation / export artifacts are ever produced** → the Art Studio asset-detail 3D viewer and downloads have nothing to show. This is the likely root cause behind the "can't see/preview produced assets" report. ## Root cause — prod provisioning gap, not a code bug `SubprocessArtPipeStageExecutor` hard-requires `{PipelineRoot}/models/{ModelDir}/artpipe.json` (`SubprocessArtPipeStageExecutor.cs:39-42`) and fails fast if absent. Config maps `ArtPipe:StageSettings:modeling.ModelDir = "TripoSG"` (`appsettings.json`, in place since #348) and prod `PipelineRoot = /opt/art_pipe`. **TripoSG (manifest + venv + weights) is not installed at `/opt/art_pipe/models/TripoSG/` on the prod GPU worker.** Confirmed present in the dev art_pipe checkout (`models/TripoSG/artpipe.json` exists locally, alongside InstantMesh / TripoSR / Hunyuan3D / SF3D / ShapE / etc.), so the executor is behaving correctly — the model just isn't vendored on prod. ## Remediation (pick one) 1. **Preferred:** install/vendor TripoSG on the prod GPU worker at `/opt/art_pipe/models/TripoSG/` (manifest + venv + weights), same as the other installed models. Ties into #432 (repo-vendoring model venvs) and #425 (full local pipeline). 2. **Interim workaround:** repoint `modeling.ModelDir` to an `image_to_3d`-capable model that *is* installed on prod (InstantMesh / TripoSR / Hunyuan3D all expose `image_to_3d`) in `appsettings.json` `StageSettings`, then redeploy the ArtPipeProcessor. ## Triage aid On the GPU worker: `ls /opt/art_pipe/models/*/artpipe.json` to confirm what's actually vendored vs the `StageSettings` mapping (concept→SDXLLightning, modeling→**TripoSG**, texturing→Hunyuan3DPaint, rigging/animation/export→Blender). Any other stage whose configured `ModelDir` isn't installed will fail identically. Related: #445 (restart-from-stage modeling 400), #432, #425. The separate Art-Studio-side UX gap (detail 3D viewer doesn't auto-load a model) is being fixed in an angular PR referencing #446.
spikerj commented 2026-07-08 22:48:21 +00:00
Author
Owner
Copy Link
Copy Source

Status roll-up — the interim remediation is in flight, and this ticket now also carries the prod-vendoring residual from #456.

Remediation 2 (interim, in flight): the #368 cutover PRs repoint modeling to TripoSR (installed on prod) via Production overlays — spikersoft-backend PR #186 (StageSettings:modeling.ModelDir=TripoSR in the worker's appsettings.Production.json + ArtStudio:StageModelMap modeling→TripoSR in the API's). Once that merges and deploys, the modeling stage stops dying at the missing TripoSG manifest.

Remediation 1 (preferred, stays open here): vendor TripoSG on SERVER and repoint modeling back. The exact ops steps are now written down in spikersoft-infrastructure PR #13's docs/artpipe-server-cutover-runbook.md — §1 (git pull the /mnt/fusionio/spikersoft/art_pipe checkout; the prod checkout predates models/TripoSG, which is this bug), §2 (python3 bootstrap.py --model TripoSG inside a worker container), §6 (repoint StageModelMap modeling→TripoSG + add a spikersoft-artpipe-model-triposg resident stack).

New residual moved here from #456 (now closed): ShapE must be vendored on the SERVER mount the same way (bootstrap.py --model ShapE) before the shipped text→3D path produces meshes in prod. Same class of gap, same runbook procedure — one extra bootstrap line while an operator is already in there.

Host-side checklist to close this ticket:

  • git pull the SERVER art_pipe checkout
  • bootstrap.py --model TripoSG (venv + weights)
  • bootstrap.py --model ShapE (venv + weights — #456 residual)
  • repoint modeling→TripoSG (StageModelMap + worker overlay) and add the triposg resident stack
  • verify: modeling produces GridFS artifacts on prod; a Text→3D submission produces a mesh
Status roll-up — the interim remediation is in flight, and this ticket now also carries the prod-vendoring residual from #456. **Remediation 2 (interim, in flight):** the #368 cutover PRs repoint `modeling` to **TripoSR** (installed on prod) via Production overlays — spikersoft-backend PR #186 (`StageSettings:modeling.ModelDir=TripoSR` in the worker's `appsettings.Production.json` + `ArtStudio:StageModelMap` modeling→TripoSR in the API's). Once that merges and deploys, the modeling stage stops dying at the missing TripoSG manifest. **Remediation 1 (preferred, stays open here):** vendor **TripoSG** on SERVER and repoint modeling back. The exact ops steps are now written down in spikersoft-infrastructure PR #13's `docs/artpipe-server-cutover-runbook.md` — §1 (`git pull` the `/mnt/fusionio/spikersoft/art_pipe` checkout; the prod checkout predates `models/TripoSG`, which is this bug), §2 (`python3 bootstrap.py --model TripoSG` inside a worker container), §6 (repoint `StageModelMap` modeling→TripoSG + add a `spikersoft-artpipe-model-triposg` resident stack). **New residual moved here from #456 (now closed):** **ShapE** must be vendored on the SERVER mount the same way (`bootstrap.py --model ShapE`) before the shipped text→3D path produces meshes in prod. Same class of gap, same runbook procedure — one extra bootstrap line while an operator is already in there. Host-side checklist to close this ticket: - [ ] `git pull` the SERVER art_pipe checkout - [ ] `bootstrap.py --model TripoSG` (venv + weights) - [ ] `bootstrap.py --model ShapE` (venv + weights — #456 residual) - [ ] repoint modeling→TripoSG (`StageModelMap` + worker overlay) and add the triposg resident stack - [ ] verify: modeling produces GridFS artifacts on prod; a Text→3D submission produces a mesh
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#455