art_pipe: repo-vendoring model venvs miss cloned-repo requirements + need Python 3.10 #432

New Issue

2026-07-06T02:32:49Z

spikerj commented

2026-07-06 02:32:49 +00:00

Found running the ProArt pipeline end-to-end on a GPU box (follow-up to #425). The concept stage works on GPU, but modeling (TripoSG) and texturing (Hunyuan3DPaint) fail because their venvs are incomplete. Two coupled root causes, both fixed:

1. Cloned-repo requirements never installed (all platforms)

Models that vendor a source repo via install.repo (TripoSG -> VAST-AI-Research/TripoSG, Hunyuan3DPaint -> Tencent-Hunyuan/Hunyuan3D-2) add the repo to sys_path but are not pip-installed. venv_setup.setup_model_venv installed only the model's requirements.txt and an editable install iff the repo had setup.py/pyproject.toml. These repos have neither, so their requirements.txt (diffusers, einops, omegaconf, ...) was skipped and the stage failed at run time:

ModuleNotFoundError: No module named 'diffusers'   (triposg pipeline import)

Fix: spikersoft-artpipe PR #4 - install <dest>/requirements.txt when present.

2. Worker image lacks Python 3.10

The manifests request env.python: 3.10 and their pinned deps have no cp312 wheels (TripoSG pins numpy==1.22.3; diso/pymeshlab/fvdb are cp310-only). The ubuntu24.04 worker image only ships python3.12, so venv_setup._find_python('3.10') fell back to 3.12 and the venvs couldn't resolve.

Fix: spikersoft-backend PR #108 - install python3.10 (deadsnakes) in the ArtPipeProcessor image.

Status

spikersoft-artpipe PR #4 (install cloned-repo requirements)
spikersoft-backend PR #108 (python3.10 in image; also git + script fixes)
Verify modeling + texturing produce artifacts on GPU after re-bootstrap

Notes

Cause #1 is not platform-specific; these two models had likely never run end-to-end anywhere.
Related: cu128 torch pin (#426), Windows local worker (#425).

Found running the ProArt pipeline end-to-end on a GPU box (follow-up to #425). The `concept` stage works on GPU, but `modeling` (TripoSG) and `texturing` (Hunyuan3DPaint) fail because their venvs are incomplete. Two coupled root causes, both fixed: ## 1. Cloned-repo requirements never installed (all platforms) Models that vendor a source repo via `install.repo` (TripoSG -> VAST-AI-Research/TripoSG, Hunyuan3DPaint -> Tencent-Hunyuan/Hunyuan3D-2) add the repo to `sys_path` but are not pip-installed. `venv_setup.setup_model_venv` installed only the *model's* `requirements.txt` and an editable install *iff* the repo had `setup.py`/`pyproject.toml`. These repos have neither, so their `requirements.txt` (`diffusers`, `einops`, `omegaconf`, ...) was skipped and the stage failed at run time: ``` ModuleNotFoundError: No module named 'diffusers' (triposg pipeline import) ``` Fix: **spikersoft-artpipe PR #4** - install `<dest>/requirements.txt` when present. ## 2. Worker image lacks Python 3.10 The manifests request `env.python: 3.10` and their pinned deps have no cp312 wheels (TripoSG pins `numpy==1.22.3`; `diso`/`pymeshlab`/`fvdb` are cp310-only). The ubuntu24.04 worker image only ships python3.12, so `venv_setup._find_python('3.10')` fell back to 3.12 and the venvs couldn't resolve. Fix: **spikersoft-backend PR #108** - install python3.10 (deadsnakes) in the ArtPipeProcessor image. ## Status - [x] spikersoft-artpipe PR #4 (install cloned-repo requirements) - [x] spikersoft-backend PR #108 (python3.10 in image; also git + script fixes) - [ ] Verify modeling + texturing produce artifacts on GPU after re-bootstrap ## Notes - Cause #1 is not platform-specific; these two models had likely never run end-to-end anywhere. - Related: cu128 torch pin (#426), Windows local worker (#425).

spikerj commented

2026-07-06 03:30:23 +00:00

Update — getting TripoSG (modeling) actually running end-to-end on a cu128 host surfaced two more venv_setup bugs beyond the missing cloned-repo requirements.txt. Fixed in spikersoft-artpipe PR #5 (follow-up to art_pipe #4):

Torch installed with --extra-index-url was non-authoritative. pip kept preferring PyPI's newest torch (bundled cu130) over the manifest's pinned cu128 wheels. This is the actual root cause of the cu130-vs-cu128 mismatch we'd been band-aiding with a post-hoc Repair-TorchStack re-pin on Windows. Fix: use --index-url for the torch install so the declared CUDA build wins.
CUDA-extension repo deps couldn't build. TripoSG's cloned requirements.txt includes diso (a torch/CUDA C++ extension). Under pip build isolation the build can't see torch (ModuleNotFoundError: torch), and even with --no-build-isolation the compile fails (fatal error: cuda_runtime.h) because the CUDA toolkit headers aren't on the include path. Fix: retry the cloned-repo install with --no-build-isolation + a CUDA build env (CUDA_HOME/CPATH/PATH), auto-detected and a no-op on macOS.

Why #1 matters beyond the driver mismatch: the post-hoc torch re-pin ran after diso was compiled and after the repo pinned numpy==1.22.3, so it ABI-broke diso (undefined symbol) and scipy/scikit-image (numpy.dtype size changed). Pinning torch correctly up front keeps the venv internally consistent and makes the Windows guard dormant.

Verified on the RTX 4090 dev box: torch 2.11.0+cu128, cuda True, numpy 1.22.3, full TripoSG pipeline import OK (torch + diffusers + diso CUDA ext + scipy + scikit-image).

Update — getting TripoSG (`modeling`) actually running end-to-end on a cu128 host surfaced two more `venv_setup` bugs beyond the missing cloned-repo `requirements.txt`. Fixed in **spikersoft-artpipe PR #5** (follow-up to art_pipe #4): 1. **Torch installed with `--extra-index-url` was non-authoritative.** pip kept preferring PyPI's newest torch (bundled `cu130`) over the manifest's pinned `cu128` wheels. This is the actual root cause of the cu130-vs-cu128 mismatch we'd been band-aiding with a post-hoc `Repair-TorchStack` re-pin on Windows. Fix: use `--index-url` for the torch install so the declared CUDA build wins. 2. **CUDA-extension repo deps couldn't build.** TripoSG's cloned `requirements.txt` includes `diso` (a torch/CUDA C++ extension). Under pip build isolation the build can't see torch (`ModuleNotFoundError: torch`), and even with `--no-build-isolation` the compile fails (`fatal error: cuda_runtime.h`) because the CUDA toolkit headers aren't on the include path. Fix: retry the cloned-repo install with `--no-build-isolation` + a CUDA build env (`CUDA_HOME`/`CPATH`/`PATH`), auto-detected and a no-op on macOS. Why #1 matters beyond the driver mismatch: the post-hoc torch re-pin ran *after* `diso` was compiled and *after* the repo pinned `numpy==1.22.3`, so it ABI-broke `diso` (`undefined symbol`) and `scipy`/`scikit-image` (`numpy.dtype size changed`). Pinning torch correctly up front keeps the venv internally consistent and makes the Windows guard dormant. Verified on the RTX 4090 dev box: `torch 2.11.0+cu128, cuda True, numpy 1.22.3`, full TripoSG pipeline import OK (torch + diffusers + diso CUDA ext + scipy + scikit-image).

spikerj referenced this issue

2026-07-07 22:10:24 +00:00

[Bug][Prod] Art Studio 'modeling' stage fails — TripoSG model manifest missing on prod host #455

spikerj referenced this issue

2026-07-12 00:26:57 +00:00

[Bug][Prod] SafetyCheck art-pipe resident worker missing 'torch' → ArtPipeProcessor host crash-loops (24x) #486

spikerj referenced this issue

2026-07-12 02:22:04 +00:00

[Bug][Prod] ArtPipeProcessor image: /home/app not writable + no HF_HOME → HuggingFace cache PermissionError blocks model weight downloads #495

spikerj referenced this issue

2026-07-12 04:24:46 +00:00

[Bug][Prod] /mnt/fusionio/spikersoft/ai missing embedding model — book pipeline dies at embeddings (nomic-embed-text-v2-moe.f16.gguf not found) #497