spikerj/spikersoft-issues
1
0 Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

art_pipe: repo-vendoring model venvs miss cloned-repo requirements + need Python 3.10 #432

New Issue
Closed
opened 2026-07-06 02:32:49 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-06 02:32:49 +00:00
Owner
Copy link
Copy source

Found running the ProArt pipeline end-to-end on a GPU box (follow-up to #425). The concept stage works on GPU, but modeling (TripoSG) and texturing (Hunyuan3DPaint) fail because their venvs are incomplete. Two coupled root causes, both fixed:

1. Cloned-repo requirements never installed (all platforms)

Models that vendor a source repo via install.repo (TripoSG -> VAST-AI-Research/TripoSG, Hunyuan3DPaint -> Tencent-Hunyuan/Hunyuan3D-2) add the repo to sys_path but are not pip-installed. venv_setup.setup_model_venv installed only the model's requirements.txt and an editable install iff the repo had setup.py/pyproject.toml. These repos have neither, so their requirements.txt (diffusers, einops, omegaconf, ...) was skipped and the stage failed at run time:

ModuleNotFoundError: No module named 'diffusers'   (triposg pipeline import)

Fix: spikersoft-artpipe PR #4 - install <dest>/requirements.txt when present.

2. Worker image lacks Python 3.10

The manifests request env.python: 3.10 and their pinned deps have no cp312 wheels (TripoSG pins numpy==1.22.3; diso/pymeshlab/fvdb are cp310-only). The ubuntu24.04 worker image only ships python3.12, so venv_setup._find_python('3.10') fell back to 3.12 and the venvs couldn't resolve.

Fix: spikersoft-backend PR #108 - install python3.10 (deadsnakes) in the ArtPipeProcessor image.

Status

  • spikersoft-artpipe PR #4 (install cloned-repo requirements)
  • spikersoft-backend PR #108 (python3.10 in image; also git + script fixes)
  • Verify modeling + texturing produce artifacts on GPU after re-bootstrap

Notes

  • Cause #1 is not platform-specific; these two models had likely never run end-to-end anywhere.
  • Related: cu128 torch pin (#426), Windows local worker (#425).
Found running the ProArt pipeline end-to-end on a GPU box (follow-up to #425). The `concept` stage works on GPU, but `modeling` (TripoSG) and `texturing` (Hunyuan3DPaint) fail because their venvs are incomplete. Two coupled root causes, both fixed: ## 1. Cloned-repo requirements never installed (all platforms) Models that vendor a source repo via `install.repo` (TripoSG -> VAST-AI-Research/TripoSG, Hunyuan3DPaint -> Tencent-Hunyuan/Hunyuan3D-2) add the repo to `sys_path` but are not pip-installed. `venv_setup.setup_model_venv` installed only the *model's* `requirements.txt` and an editable install *iff* the repo had `setup.py`/`pyproject.toml`. These repos have neither, so their `requirements.txt` (`diffusers`, `einops`, `omegaconf`, ...) was skipped and the stage failed at run time: ``` ModuleNotFoundError: No module named 'diffusers' (triposg pipeline import) ``` Fix: **spikersoft-artpipe PR #4** - install `<dest>/requirements.txt` when present. ## 2. Worker image lacks Python 3.10 The manifests request `env.python: 3.10` and their pinned deps have no cp312 wheels (TripoSG pins `numpy==1.22.3`; `diso`/`pymeshlab`/`fvdb` are cp310-only). The ubuntu24.04 worker image only ships python3.12, so `venv_setup._find_python('3.10')` fell back to 3.12 and the venvs couldn't resolve. Fix: **spikersoft-backend PR #108** - install python3.10 (deadsnakes) in the ArtPipeProcessor image. ## Status - [x] spikersoft-artpipe PR #4 (install cloned-repo requirements) - [x] spikersoft-backend PR #108 (python3.10 in image; also git + script fixes) - [ ] Verify modeling + texturing produce artifacts on GPU after re-bootstrap ## Notes - Cause #1 is not platform-specific; these two models had likely never run end-to-end anywhere. - Related: cu128 torch pin (#426), Windows local worker (#425).
spikerj commented 2026-07-06 03:30:23 +00:00
Author
Owner
Copy link
Copy source

Update — getting TripoSG (modeling) actually running end-to-end on a cu128 host surfaced two more venv_setup bugs beyond the missing cloned-repo requirements.txt. Fixed in spikersoft-artpipe PR #5 (follow-up to art_pipe #4):

  1. Torch installed with --extra-index-url was non-authoritative. pip kept preferring PyPI's newest torch (bundled cu130) over the manifest's pinned cu128 wheels. This is the actual root cause of the cu130-vs-cu128 mismatch we'd been band-aiding with a post-hoc Repair-TorchStack re-pin on Windows. Fix: use --index-url for the torch install so the declared CUDA build wins.

  2. CUDA-extension repo deps couldn't build. TripoSG's cloned requirements.txt includes diso (a torch/CUDA C++ extension). Under pip build isolation the build can't see torch (ModuleNotFoundError: torch), and even with --no-build-isolation the compile fails (fatal error: cuda_runtime.h) because the CUDA toolkit headers aren't on the include path. Fix: retry the cloned-repo install with --no-build-isolation + a CUDA build env (CUDA_HOME/CPATH/PATH), auto-detected and a no-op on macOS.

Why #1 matters beyond the driver mismatch: the post-hoc torch re-pin ran after diso was compiled and after the repo pinned numpy==1.22.3, so it ABI-broke diso (undefined symbol) and scipy/scikit-image (numpy.dtype size changed). Pinning torch correctly up front keeps the venv internally consistent and makes the Windows guard dormant.

Verified on the RTX 4090 dev box: torch 2.11.0+cu128, cuda True, numpy 1.22.3, full TripoSG pipeline import OK (torch + diffusers + diso CUDA ext + scipy + scikit-image).

Update — getting TripoSG (`modeling`) actually running end-to-end on a cu128 host surfaced two more `venv_setup` bugs beyond the missing cloned-repo `requirements.txt`. Fixed in **spikersoft-artpipe PR #5** (follow-up to art_pipe #4): 1. **Torch installed with `--extra-index-url` was non-authoritative.** pip kept preferring PyPI's newest torch (bundled `cu130`) over the manifest's pinned `cu128` wheels. This is the actual root cause of the cu130-vs-cu128 mismatch we'd been band-aiding with a post-hoc `Repair-TorchStack` re-pin on Windows. Fix: use `--index-url` for the torch install so the declared CUDA build wins. 2. **CUDA-extension repo deps couldn't build.** TripoSG's cloned `requirements.txt` includes `diso` (a torch/CUDA C++ extension). Under pip build isolation the build can't see torch (`ModuleNotFoundError: torch`), and even with `--no-build-isolation` the compile fails (`fatal error: cuda_runtime.h`) because the CUDA toolkit headers aren't on the include path. Fix: retry the cloned-repo install with `--no-build-isolation` + a CUDA build env (`CUDA_HOME`/`CPATH`/`PATH`), auto-detected and a no-op on macOS. Why #1 matters beyond the driver mismatch: the post-hoc torch re-pin ran *after* `diso` was compiled and *after* the repo pinned `numpy==1.22.3`, so it ABI-broke `diso` (`undefined symbol`) and `scipy`/`scikit-image` (`numpy.dtype size changed`). Pinning torch correctly up front keeps the venv internally consistent and makes the Windows guard dormant. Verified on the RTX 4090 dev box: `torch 2.11.0+cu128, cuda True, numpy 1.22.3`, full TripoSG pipeline import OK (torch + diffusers + diso CUDA ext + scipy + scikit-image).
spikerj commented 2026-08-07 13:55:29 +00:00
Author
Owner
Copy link
Copy source

Verified complete 2026-08-07 — closing.

Both fixes are on their default branches, and the verification checkbox is satisfied by what replaced the re-bootstrap flow.

  • Code (cause 1, cloned-repo requirements): spikersoft-artpipe@7082d26 src/artpipe/venv_setup.py:500-528 — when install.dest exists, <dest>/requirements.txt is now installed explicitly, with a documented retry on failure using --no-build-isolation plus a CUDA build env (_cuda_build_env(), :615-648) for repos shipping torch/CUDA C++ extensions like TripoSG's diso. The follow-up torch fix from PR #5 is there too: :464-470 uses --index-url (not --extra-index-url) so the manifest's declared CUDA build wins, which was the real root cause behind the cu130-vs-cu128 band-aiding.
  • Code (cause 2, Python 3.10): spikersoft-backend@98102023 SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile:55-68 — deadsnakes PPA plus python3.10, python3.10-venv, python3.10-dev, with a comment naming exactly this failure (venv_setup._find_python falling back to 3.12 and the cp310-only wheels failing to resolve).
  • Live: both models that this ticket said had "likely never run end-to-end anywhere" now run as their own production lanes from baked env images — spikersoft-artpipe-model-triposg 1/1 (ArtPipeStageConsumer starting in MODEL-QUEUE mode (subprocess per job) for model TripoSG) and spikersoft-artpipe-model-hunyuan 1/1 on artpipe-model-hunyuan3dpaint:latest. Hunyuan3DPaint's manifest gates its own image build on install.verify_imports: ["torch", "custom_rasterizer", "hy3dgen.texgen"], so a venv missing those deps fails the build rather than shipping. And the texturing/export chain has demonstrably produced artifacts since — e.g. blender_export_1de22c8e.glb for art asset 6a62b6fdd453b392fdff4f13 on 2026-07-24, recorded on #825.
  • Shipped by: spikersoft-artpipe PR #4 and PR #5; spikersoft-backend PR #108.

The remaining checkbox ("verify after re-bootstrap") is moot in the intended sense: the per-host bootstrap.py re-bootstrap flow was superseded by the baked per-model images of #515, which run the same venv_setup code at image-build time with install.verify_imports as the gate.

Not migrated: nothing left to do.

— Opus 5 Agent

Verified complete 2026-08-07 — closing. Both fixes are on their default branches, and the verification checkbox is satisfied by what replaced the re-bootstrap flow. - **Code (cause 1, cloned-repo requirements):** `spikersoft-artpipe@7082d26` `src/artpipe/venv_setup.py:500-528` — when `install.dest` exists, `<dest>/requirements.txt` is now installed explicitly, with a documented retry on failure using `--no-build-isolation` plus a CUDA build env (`_cuda_build_env()`, `:615-648`) for repos shipping torch/CUDA C++ extensions like TripoSG's `diso`. The follow-up torch fix from PR #5 is there too: `:464-470` uses `--index-url` (not `--extra-index-url`) so the manifest's declared CUDA build wins, which was the real root cause behind the cu130-vs-cu128 band-aiding. - **Code (cause 2, Python 3.10):** `spikersoft-backend@98102023` `SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile:55-68` — deadsnakes PPA plus `python3.10`, `python3.10-venv`, `python3.10-dev`, with a comment naming exactly this failure (`venv_setup._find_python` falling back to 3.12 and the cp310-only wheels failing to resolve). - **Live:** both models that this ticket said had "likely never run end-to-end anywhere" now run as their own production lanes from baked env images — `spikersoft-artpipe-model-triposg` 1/1 (`ArtPipeStageConsumer starting in MODEL-QUEUE mode (subprocess per job) for model TripoSG`) and `spikersoft-artpipe-model-hunyuan` 1/1 on `artpipe-model-hunyuan3dpaint:latest`. Hunyuan3DPaint's manifest gates its own image build on `install.verify_imports: ["torch", "custom_rasterizer", "hy3dgen.texgen"]`, so a venv missing those deps fails the build rather than shipping. And the texturing/export chain has demonstrably produced artifacts since — e.g. `blender_export_1de22c8e.glb` for art asset `6a62b6fdd453b392fdff4f13` on 2026-07-24, recorded on #825. - **Shipped by:** spikersoft-artpipe PR #4 and PR #5; spikersoft-backend PR #108. The remaining checkbox ("verify after re-bootstrap") is moot in the intended sense: the per-host `bootstrap.py` re-bootstrap flow was superseded by the baked per-model images of #515, which run the same `venv_setup` code at image-build time with `install.verify_imports` as the gate. Not migrated: nothing left to do. — Opus 5 Agent
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 epic
Cross-repo epic — children live in per-repo trackers
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#432