spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

ModelEnvImages: triposg venv unbuildable (fvdb needs Python >=3.11, manifest pins 3.10) — aborts the serial loop and silently blocks 4 later images #839

New Issue
Closed
opened 2026-07-25 01:39:55 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-25 01:39:55 +00:00
Owner
Copy Link
Copy Source

ModelEnvImages cannot build triposg, and because the build loop is serial and aborts on first failure, it takes every image after it down with it.

Failure

ERROR: Ignored the following versions that require a different python version:
  0.0.2 Requires-Python >=3.11; ... 2.5.1 Requires-Python >=3.12
ERROR: Could not find a version that satisfies the requirement fvdb (from versions: none)
ERROR: No matching distribution found for fvdb
[triposg] Command failed (exit 1): /opt/art_pipe/models/TripoSG/venv/bin/pip install -r models/TripoSG/requirements.txt

models/TripoSG/artpipe.json pins env.python: "3.10", but every published fvdb release requires Python >= 3.11 (the newest require >= 3.12). There is no 3.10 wheel, so this can never have worked.

Blast radius

DEFAULT_IMAGES="safetycheck prodstages photostack triposg shape instantmesh sf3d hunyuan3dpaint" is a serial loop, and the venv step is ... || exit 1 inside the Dockerfile RUN, so triposg's failure aborts the whole job. Everything ordered after it never builds:

image in registry
safetycheck / prodstages / photostack yes (built before triposg)
triposg / shape / instantmesh / sf3d / hunyuan3dpaint no — never built

This is why the #357 per-model modeling stacks were never deployable: artpipe-model-env-{triposg,shape,instantmesh,sf3d} have never existed in the registry. Combined with the API's ArtStudio:StageModelMap cutover already being live, that is the direct cause of the current modeling outage — submissions publish to art.model.triposr.tasks, spikersoft-artpipe-model-triposr sits at 0/0, and the #360 watchdog fails every run after 10 minutes.

Worked around for #832 by dispatching images: hunyuan3dpaint alone, which builds fine.

Two fixes, both worth doing

  1. triposg itself — bump its manifest to python: "3.11" (check TripoSG's other deps tolerate it), or drop/replace fvdb. Then verify with a probe like scripts/try-hunyuan-build.sh / try-sf3d-build.sh before re-adding it to the default set. Until then it should arguably come OUT of DEFAULT_IMAGES so it stops blocking everything.
  2. The loop should not be all-or-nothing. One unbuildable model silently costing four unrelated images is the real defect here — a failure this cheap to survive should not gate the whole set. Collect per-image failures, keep building the rest, and fail the job at the end with a summary of which images failed. As it stands the failure mode is invisible: the job just goes red and nobody can tell that four other images are now stale/missing.
`ModelEnvImages` cannot build **triposg**, and because the build loop is serial and aborts on first failure, it takes every image after it down with it. ## Failure ``` ERROR: Ignored the following versions that require a different python version: 0.0.2 Requires-Python >=3.11; ... 2.5.1 Requires-Python >=3.12 ERROR: Could not find a version that satisfies the requirement fvdb (from versions: none) ERROR: No matching distribution found for fvdb [triposg] Command failed (exit 1): /opt/art_pipe/models/TripoSG/venv/bin/pip install -r models/TripoSG/requirements.txt ``` `models/TripoSG/artpipe.json` pins `env.python: "3.10"`, but **every published `fvdb` release requires Python >= 3.11** (the newest require >= 3.12). There is no 3.10 wheel, so this can never have worked. ## Blast radius `DEFAULT_IMAGES="safetycheck prodstages photostack triposg shape instantmesh sf3d hunyuan3dpaint"` is a serial loop, and the venv step is `... || exit 1` inside the Dockerfile RUN, so triposg's failure aborts the whole job. Everything ordered after it never builds: | image | in registry | |---|---| | safetycheck / prodstages / photostack | yes (built before triposg) | | **triposg / shape / instantmesh / sf3d / hunyuan3dpaint** | **no — never built** | This is why the #357 per-model modeling stacks were never deployable: `artpipe-model-env-{triposg,shape,instantmesh,sf3d}` have never existed in the registry. Combined with the API's `ArtStudio:StageModelMap` cutover already being live, that is the direct cause of the current modeling outage — submissions publish to `art.model.triposr.tasks`, `spikersoft-artpipe-model-triposr` sits at 0/0, and the #360 watchdog fails every run after 10 minutes. Worked around for #832 by dispatching `images: hunyuan3dpaint` alone, which builds fine. ## Two fixes, both worth doing 1. **triposg itself** — bump its manifest to `python: "3.11"` (check TripoSG's other deps tolerate it), or drop/replace `fvdb`. Then verify with a probe like `scripts/try-hunyuan-build.sh` / `try-sf3d-build.sh` before re-adding it to the default set. Until then it should arguably come OUT of `DEFAULT_IMAGES` so it stops blocking everything. 2. **The loop should not be all-or-nothing.** One unbuildable model silently costing four unrelated images is the real defect here — a failure this cheap to survive should not gate the whole set. Collect per-image failures, keep building the rest, and fail the job at the end with a summary of which images failed. As it stands the failure mode is invisible: the job just goes red and nobody can tell that four *other* images are now stale/missing.
spikerj commented 2026-07-25 02:44:45 +00:00
Author
Owner
Copy Link
Copy Source

Root cause found — and it is not the Python version

fvdb and warp-lang are not real dependencies of TripoSG. They were guessed.

check result
Upstream VAST-AI-Research/TripoSG requirements.txt diffusers transformers einops huggingface_hub opencv-python trimesh omegaconf scikit-image numpy==1.22.3 peft jaxtyping typeguard diso pymeshlab — no fvdb, no warp-lang
Our hand-written models/TripoSG/requirements.txt has both, added in the initial init commit (77bdd17)
import fvdb / import warp anywhere in src/ none — the only hits are a code comment and an error-message string in a macOS guard in src/artpipe/models/triposg.py:47-49

So this was never a Python-3.10-vs-3.11 problem. Nothing imports these packages; they simply cannot resolve on 3.10 and the image has therefore never built once since it was added. Bumping the manifest to 3.11 would have "fixed" it by accident while keeping two unused multi-hundred-MB deps.

Verified fix

Delete the two lines. Verified on the 4090 with the same probe pattern as try-hunyuan-build.sh / try-sf3d-build.sh (artpipe.venv_setup --model TripoSG --no-weights in artpipe-base):

[triposg] Venv setup complete
TRIPOSG_BUILD_OK

The venv now resolves cleanly and installs upstream's real dependency set, including diso 0.1.4 (the CUDA mesh-extraction extension) via the existing --no-build-isolation retry path.

Caveat, stated plainly: this proves the image builds. It does not prove TripoSG runs — that still needs weights and a real image_to_3d job. TripoSG has never run in prod (prod modeling is TripoSR, see #455). A follow-up should exercise it end-to-end before anyone relies on it as a selectable modeling method.

One note for whoever picks this up: a bare python -c "import triposg" against the built venv fails, and that is expected, not a second bug. TripoSG ships no setup.py, so it is never pip-installed; it is reached through the manifest's sys_path: ['.deps/TripoSG'], which the worker applies at run time (src/artpipe/worker.py:182). Any smoke check has to replicate that or it reports a false failure.

Still recommended: make the loop non-fatal

Unchanged from the original report, and this incident is the argument for it. One unbuildable model cost four unrelated images — including the one #832 needed — and the only symptom was a red job. hunyuan3dpaint had to be built by an explicit workflow_dispatch to route around it.

## Root cause found — and it is not the Python version `fvdb` and `warp-lang` are **not real dependencies of TripoSG**. They were guessed. | check | result | |---|---| | Upstream `VAST-AI-Research/TripoSG` `requirements.txt` | `diffusers transformers einops huggingface_hub opencv-python trimesh omegaconf scikit-image numpy==1.22.3 peft jaxtyping typeguard diso pymeshlab` — **no `fvdb`, no `warp-lang`** | | Our hand-written `models/TripoSG/requirements.txt` | has both, added in the initial `init` commit (`77bdd17`) | | `import fvdb` / `import warp` anywhere in `src/` | **none** — the only hits are a code comment and an error-message string in a macOS guard in `src/artpipe/models/triposg.py:47-49` | So this was never a Python-3.10-vs-3.11 problem. Nothing imports these packages; they simply cannot resolve on 3.10 and the image has therefore never built once since it was added. Bumping the manifest to 3.11 would have "fixed" it by accident while keeping two unused multi-hundred-MB deps. ## Verified fix Delete the two lines. Verified on the 4090 with the same probe pattern as `try-hunyuan-build.sh` / `try-sf3d-build.sh` (`artpipe.venv_setup --model TripoSG --no-weights` in `artpipe-base`): ``` [triposg] Venv setup complete TRIPOSG_BUILD_OK ``` The venv now resolves cleanly and installs upstream's real dependency set, including `diso 0.1.4` (the CUDA mesh-extraction extension) via the existing `--no-build-isolation` retry path. **Caveat, stated plainly:** this proves the image *builds*. It does not prove TripoSG *runs* — that still needs weights and a real `image_to_3d` job. TripoSG has never run in prod (prod modeling is TripoSR, see #455). A follow-up should exercise it end-to-end before anyone relies on it as a selectable modeling method. One note for whoever picks this up: a bare `python -c "import triposg"` against the built venv fails, and that is expected, not a second bug. TripoSG ships no `setup.py`, so it is never pip-installed; it is reached through the manifest's `sys_path: ['.deps/TripoSG']`, which the worker applies at run time (`src/artpipe/worker.py:182`). Any smoke check has to replicate that or it reports a false failure. ## Still recommended: make the loop non-fatal Unchanged from the original report, and this incident is the argument for it. One unbuildable model cost four unrelated images — including the one #832 needed — and the only symptom was a red job. `hunyuan3dpaint` had to be built by an explicit `workflow_dispatch` to route around it.
spikerj commented 2026-07-29 15:29:37 +00:00
Author
Owner
Copy Link
Copy Source

Resolved in spikersoft-artpipe PRs #36 and #38 (merged to main). Verified against origin/main — both fixes from the root-cause comment are in:

1. The phantom deps are gone, and the manifest correctly stayed at 3.10. models/TripoSG/requirements.txt no longer lists fvdb or warp-lang; it mirrors upstream's real dependency set. git grep -n "fvdb\|warp-lang" origin/main now hits only a comment and a macOS-guard error string in src/artpipe/models/triposg.py:47-49. Worth recording explicitly: bumping "python" to 3.11 would have been the wrong fix — the dep never belonged there in the first place.

2. One bad model no longer aborts the serial loop. .gitea/workflows/model-env-images.yml: :230-232 collects BUILT_IMAGES/FAILED_IMAGES; :317-338 uses &&-chaining (with a comment explaining why if ( set -e; … ) would silently mask failures); :339-341 prints a summary; :458-463 fails the job at the end if any env image failed. The cascade at :351 now forwards only the images that actually built, so the 4 downstream images are no longer silently blocked.

Note: triposg deliberately stays in DEFAULT_IMAGES — it builds now, which supersedes the body's interim "take it out" suggestion. The comment's own caveat stands and is out of scope here: this proves the venv builds, not that TripoSG runs; that's a follow-up.

Closing.

Resolved in spikersoft-artpipe PRs #36 and #38 (merged to `main`). Verified against `origin/main` — both fixes from the root-cause comment are in: **1. The phantom deps are gone, and the manifest correctly stayed at 3.10.** `models/TripoSG/requirements.txt` no longer lists `fvdb` or `warp-lang`; it mirrors upstream's real dependency set. `git grep -n "fvdb\|warp-lang" origin/main` now hits only a comment and a macOS-guard error string in `src/artpipe/models/triposg.py:47-49`. Worth recording explicitly: bumping `"python"` to 3.11 would have been the *wrong* fix — the dep never belonged there in the first place. **2. One bad model no longer aborts the serial loop.** `.gitea/workflows/model-env-images.yml`: `:230-232` collects `BUILT_IMAGES`/`FAILED_IMAGES`; `:317-338` uses `&&`-chaining (with a comment explaining why `if ( set -e; … )` would silently mask failures); `:339-341` prints a summary; `:458-463` fails the job at the end if any env image failed. The cascade at `:351` now forwards only the images that actually built, so the 4 downstream images are no longer silently blocked. Note: triposg deliberately stays in `DEFAULT_IMAGES` — it builds now, which supersedes the body's interim "take it out" suggestion. The comment's own caveat stands and is out of scope here: this proves the venv *builds*, not that TripoSG *runs*; that's a follow-up. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#839