art_pipe local dev: migrate to a CUDA 13 driver so default torch builds work #426

New Issue

2026-07-05T23:25:44Z

spikerj commented

2026-07-05 23:25:44 +00:00

Follow-up from #425 / spikersoft-backend PR #106 (Windows local ProArt worker).

Context

When bootstrapping the art_pipe model venvs, art_pipe's venv setup pulled a CUDA 13.0 torch build (torch 2.12.1+cu130), but the dev box's NVIDIA driver only advertises CUDA 12.9 (driver version 12090). Result: torch.cuda.is_available() was False and generation would fall back to CPU.

Current mitigation (shipped in PR #106)

scripts/windows-artpipe-bootstrap.ps1 pins torch to the cu128 index (CUDA 12.8 — the newest build compatible with a 12.9 driver, and the index the model manifests already declare). After each model bootstrap it checks CUDA availability in the venv and force-reinstalls torch/torchvision from cu128 if CUDA is unavailable. This keeps GPU generation working today without a host change.

The CUDA 13 path (this ticket)

We don't want to pin to cu128 forever. Decide + do the migration so the default (cu13x) torch builds work:

Upgrade the NVIDIA driver on the GPU dev/prod hosts to one that supports CUDA 13.x, then confirm a default pip install torch build reports cuda True.
Re-check the CUDA base image tag in SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile (currently nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04) and align it with the chosen toolkit.
Confirm the art_pipe model manifests' torch_index_url (currently cu128) — bump to cu13x once drivers are updated.
Once hosts are on a CUDA 13 driver, drop / relax the cu128 pin in windows-artpipe-bootstrap.ps1.
Verify the same holds for the production spikersoft-artpipe-* swarm stacks (SERVER GPU node driver version).

Acceptance

Default torch build reports cuda True on the target driver, no cu128 pin needed.
art_pipe stages (concept/modeling/texturing) run on GPU locally and in prod on the updated toolchain.

Follow-up from #425 / spikersoft-backend PR #106 (Windows local ProArt worker). ## Context When bootstrapping the art_pipe model venvs, art_pipe's venv setup pulled a **CUDA 13.0** torch build (`torch 2.12.1+cu130`), but the dev box's NVIDIA driver only advertises **CUDA 12.9** (driver version 12090). Result: `torch.cuda.is_available()` was `False` and generation would fall back to CPU. ## Current mitigation (shipped in PR #106) `scripts/windows-artpipe-bootstrap.ps1` pins torch to the **cu128** index (CUDA 12.8 — the newest build compatible with a 12.9 driver, and the index the model manifests already declare). After each model bootstrap it checks CUDA availability in the venv and force-reinstalls torch/torchvision from cu128 if CUDA is unavailable. This keeps GPU generation working today without a host change. ## The CUDA 13 path (this ticket) We don't want to pin to cu128 forever. Decide + do the migration so the default (cu13x) torch builds work: - Upgrade the NVIDIA driver on the GPU dev/prod hosts to one that supports CUDA 13.x, then confirm a default `pip install torch` build reports `cuda True`. - Re-check the CUDA base image tag in `SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile` (currently `nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04`) and align it with the chosen toolkit. - Confirm the art_pipe model manifests' `torch_index_url` (currently cu128) — bump to cu13x once drivers are updated. - Once hosts are on a CUDA 13 driver, drop / relax the cu128 pin in `windows-artpipe-bootstrap.ps1`. - Verify the same holds for the production spikersoft-artpipe-* swarm stacks (SERVER GPU node driver version). ## Acceptance - Default torch build reports `cuda True` on the target driver, no cu128 pin needed. - art_pipe stages (concept/modeling/texturing) run on GPU locally and in prod on the updated toolchain.