Art Studio: make the Hunyuan3D texturing lane actually run in production (never textured a single asset) #832

New Issue

2026-07-25T00:44:20Z

spikerj commented

2026-07-25 00:44:20 +00:00

Hunyuan3DPaint has been configured as the texturing model since the beginning and has never textured a single asset. Making it actually run in production.

Why it was dead

Four independent blockers, all of which had to be fixed:

No image. hunyuan3dpaint was excluded from DEFAULT_IMAGES in both model-env-images.yml and spikersoft-artpipe-processor.yml ("deferred — largest weights"), so no env image ever existed. The stack pointed at the generic spikersoft-artpipe-processor monolith, whose baked set (SDXLLightning/TripoSR/SDTurboTex/Blender) has no Hunyuan3DPaint venv.
No extension. hy3dgen.texgen needs the custom_rasterizer CUDA extension (nvcc). Nothing in the manifest built it, so import hy3dgen.texgen would have failed at run time even with weights present — the SF3D texture_baker failure mode.
Weights un-bakeable as declared. install.weights: tencent/Hunyuan3D-2 with no patterns pulls 74.9 GB — nine subfolders, mostly shape-generation DiT weights this backend never loads.
Stack could never schedule. replicas: 0 since 2026-07-11 (a 10 GB resident lease does not fit SERVER's 8 GB 3070 Ti), plus a bind mount to /mnt/fusionio/spikersoft/art_pipe, which does not exist on the 4090.

Prod evidence: 92 rows in art-asset-stage-runs, texturing is 5 runs all modelUsed: sd_turbo_tex, zero documents mention hunyuan.

What changed

spikersoft-artpipe

Manifest bakes only hunyuan3d-delight-v2-0/* + hunyuan3d-paint-v2-0-turbo/* → ~15 GB instead of 74.9 GB.
Counter-intuitive exclusion: the unet's .safetensors is the dead twin, not the .bin. The repo's own custom modules.py hardcodes torch.load(diffusion_pytorch_model.bin) and has no safetensors branch. New install.weight_ignore_patterns in venv_setup expresses this (an allow-list cannot).
setup_commands builds custom_rasterizer with TORCH_CUDA_ARCH_LIST=8.6;8.9.
New install.verify_imports build gate: setup-command failures are non-fatal by design, so a failed extension build previously produced a GREEN image that only died in prod. Now it fails the build. It caught a real ordering bug during development.
Backend: pins the hunyuan3d-paint-v2-0-turbo subfolder, resolves the local snapshot dir when HF_HUB_OFFLINE=1 (the #699 pattern), and scopes a trust_remote_code injection to custom-pipeline code inside our own baked .deps tree — diffusers >= 0.39 refuses upstream's local custom_pipeline load without it.
scripts/try-hunyuan-build.sh build probe (SF3D precedent).

spikersoft-backend

Production texturing StageSettings → Hunyuan3DPaint / texture_mesh.
ModelQueueStages: ["modeling", "texturing"] in the shared overlay, not just the monolith's stack env: the worker that publishes stage N+1 is whichever worker finished stage N, so after the per-model modeling lane runs it is a per-model container that publishes texturing.
image_path walk-back — texture_mesh needs a mesh and a reference image, but modeling emits only a mesh. Scoped to texture_mesh so other stages don't start receiving an image (export would map a walked-back concept image onto a PBR texture param). Falls back to the uploaded source image for image-first plans.
API StageModelMap gains texturing → Hunyuan3DPaint.
New guard tests pinning the routing invariants that the #357 modeling outage violated.

spikersoft-infrastructure

Stack rewritten resident → ModelQueue mode (per-job lease, idle = 0 VRAM), own image, replicas: 1, no /mnt/fusionio bind.

Verified on the 4090

Real prod asset ("a yellow magic frog", asset 6a5d3dbdcf3094d34c21e68d): its TripoSR mesh + concept image pulled from GridFS, run through the actual backend class with HF_HUB_OFFLINE=1.

25.9 s, output a 6.4 MB GLB — 84,963 verts, UV-unwrapped, 2048x2048 base-color texture, mean RGB [167.8, 123.0, 15.7] (yellow, matching the prompt), std ~44 so not a flat fill.
VramMB corrected 10000 → 14000: measured peak was 9948 MB allocated but 13070 MB reserved, and the card loses the reserved figure to torch's caching allocator. At 10000 the coordinator would grant two concurrent leases on a 24 GB card and OOM both.

Hunyuan3DPaint has been configured as the texturing model since the beginning and has **never textured a single asset**. Making it actually run in production. ## Why it was dead Four independent blockers, all of which had to be fixed: 1. **No image.** `hunyuan3dpaint` was excluded from `DEFAULT_IMAGES` in both `model-env-images.yml` and `spikersoft-artpipe-processor.yml` ("deferred — largest weights"), so no env image ever existed. The stack pointed at the generic `spikersoft-artpipe-processor` monolith, whose baked set (SDXLLightning/TripoSR/SDTurboTex/Blender) has no Hunyuan3DPaint venv. 2. **No extension.** `hy3dgen.texgen` needs the `custom_rasterizer` CUDA extension (nvcc). Nothing in the manifest built it, so `import hy3dgen.texgen` would have failed at run time even with weights present — the SF3D `texture_baker` failure mode. 3. **Weights un-bakeable as declared.** `install.weights: tencent/Hunyuan3D-2` with no patterns pulls **74.9 GB** — nine subfolders, mostly shape-generation DiT weights this backend never loads. 4. **Stack could never schedule.** `replicas: 0` since 2026-07-11 (a 10 GB resident lease does not fit SERVER's 8 GB 3070 Ti), plus a bind mount to `/mnt/fusionio/spikersoft/art_pipe`, which does not exist on the 4090. Prod evidence: 92 rows in `art-asset-stage-runs`, texturing is 5 runs all `modelUsed: sd_turbo_tex`, zero documents mention hunyuan. ## What changed **spikersoft-artpipe** - Manifest bakes only `hunyuan3d-delight-v2-0/*` + `hunyuan3d-paint-v2-0-turbo/*` → **~15 GB** instead of 74.9 GB. - Counter-intuitive exclusion: the unet's `.safetensors` is the dead twin, not the `.bin`. The repo's own custom `modules.py` hardcodes `torch.load(diffusion_pytorch_model.bin)` and has **no safetensors branch**. New `install.weight_ignore_patterns` in `venv_setup` expresses this (an allow-list cannot). - `setup_commands` builds `custom_rasterizer` with `TORCH_CUDA_ARCH_LIST=8.6;8.9`. - New **`install.verify_imports` build gate**: setup-command failures are non-fatal by design, so a failed extension build previously produced a GREEN image that only died in prod. Now it fails the build. It caught a real ordering bug during development. - Backend: pins the `hunyuan3d-paint-v2-0-turbo` subfolder, resolves the local snapshot dir when `HF_HUB_OFFLINE=1` (the #699 pattern), and scopes a `trust_remote_code` injection to custom-pipeline code inside our own baked `.deps` tree — diffusers >= 0.39 refuses upstream's local `custom_pipeline` load without it. - `scripts/try-hunyuan-build.sh` build probe (SF3D precedent). **spikersoft-backend** - Production texturing `StageSettings` → `Hunyuan3DPaint` / `texture_mesh`. - **`ModelQueueStages: ["modeling", "texturing"]` in the shared overlay**, not just the monolith's stack env: the worker that publishes stage N+1 is whichever worker finished stage N, so after the per-model modeling lane runs it is a per-model container that publishes texturing. - **`image_path` walk-back** — texture_mesh needs a mesh *and* a reference image, but modeling emits only a mesh. Scoped to `texture_mesh` so other stages don't start receiving an image (export would map a walked-back concept image onto a PBR texture param). Falls back to the uploaded source image for image-first plans. - API `StageModelMap` gains `texturing → Hunyuan3DPaint`. - New guard tests pinning the routing invariants that the #357 modeling outage violated. **spikersoft-infrastructure** - Stack rewritten resident → **ModelQueue** mode (per-job lease, idle = 0 VRAM), own image, `replicas: 1`, no `/mnt/fusionio` bind. ## Verified on the 4090 Real prod asset ("a yellow magic frog", asset `6a5d3dbdcf3094d34c21e68d`): its TripoSR mesh + concept image pulled from GridFS, run through the actual backend class with `HF_HUB_OFFLINE=1`. - **25.9 s**, output a 6.4 MB GLB — 84,963 verts, UV-unwrapped, **2048x2048** base-color texture, mean RGB [167.8, 123.0, 15.7] (yellow, matching the prompt), std ~44 so not a flat fill. - **VramMB corrected 10000 → 14000**: measured peak was 9948 MB *allocated* but **13070 MB reserved**, and the card loses the reserved figure to torch's caching allocator. At 10000 the coordinator would grant two concurrent leases on a 24 GB card and OOM both.

spikerj referenced this issue

2026-07-25 01:39:55 +00:00

ModelEnvImages: triposg venv unbuildable (fvdb needs Python >=3.11, manifest pins 3.10) — aborts the serial loop and silently blocks 4 later images #839

spikerj referenced this issue

2026-07-25 02:44:45 +00:00

ModelEnvImages: triposg venv unbuildable (fvdb needs Python >=3.11, manifest pins 3.10) — aborts the serial loop and silently blocks 4 later images #839

spikerj commented

2026-07-25 03:25:01 +00:00

Live and verified in production

First end-to-end asset through the Hunyuan3D texturing lane — asset 6a642bde6f3431ee51694f62, "a magical golden apple on a black background":

concept     Completed  model=sdxl_lightning    8.0s
modeling    Completed  model=triposr          28.3s
texturing   Completed  model=hunyuan3d_paint  18.2s   <-- first ever
export      Completed  model=blender           2.0s
enrichment  Completed  model=blender           4.4s

Both lanes consuming on the 4090 in MODEL-QUEUE mode (per-job VRAM lease, idle = 0 VRAM):

ArtPipeStageConsumer listening on art.model.triposr.tasks
ArtPipeStageConsumer listening on art.model.hunyuan3dpaint.tasks

The texturing artifact (hunyuan3d_textured_a35903f8.glb, 4.1 MB) pulled back out of MinIO and inspected:

vertices 32,505   faces 42,552
has UVs: True
baseColorTexture: (2048, 2048) RGB
texture mean RGB: [139.4, 84.9, 15.0]   std: [51.2, 52.4, 33.4]
distinct colors: 174,625

Golden/amber matching the prompt, 174k distinct colours — a real generated PBR texture, not a flat fill or a Blender fallback bake.

Note for downstream consumers

Texturing now emits a mesh, not a texture set. Old sd_turbo_tex produced Kind: texture-set (PNG); Hunyuan3DPaint produces Kind: mesh-glb. The export stage's PBR-param mapping and the rigging stage's mesh walk-back both already handle this (the orchestrator tests cover both shapes), but anything else reading texturing output by Kind should be checked.

Shipped

spikersoft-artpipe #32 — manifest weight patterns (74.9 GB -> ~15 GB), custom_rasterizer build, install.verify_imports build gate, offline loader + scoped trust_remote_code, scripts/try-hunyuan-build.sh
spikersoft-infrastructure #148 — stack resident -> ModelQueue, own baked image, /mnt/fusionio bind dropped, VramMB measured at 14000
spikersoft-backend #475 — texturing -> Hunyuan3DPaint, ModelQueueStages in the shared overlay, image_path walk-back, routing guard tests

Follow-ups tracked separately

#839 (ModelEnvImages robustness) — PRs open: spikersoft-artpipe#36, spikersoft-backend#476. Both were needed to get here: triposg's unbuildable fvdb pin aborted the serial build loop before hunyuan3dpaint, and triposr was missing from both default image sets despite being what prod routes modeling to. This lane currently depends on two images built by explicit workflow_dispatch; those PRs make the default set reproduce them.

Closing.

## Live and verified in production First end-to-end asset through the Hunyuan3D texturing lane — asset `6a642bde6f3431ee51694f62`, *"a magical golden apple on a black background"*: ``` concept Completed model=sdxl_lightning 8.0s modeling Completed model=triposr 28.3s texturing Completed model=hunyuan3d_paint 18.2s <-- first ever export Completed model=blender 2.0s enrichment Completed model=blender 4.4s ``` Both lanes consuming on the 4090 in MODEL-QUEUE mode (per-job VRAM lease, idle = 0 VRAM): ``` ArtPipeStageConsumer listening on art.model.triposr.tasks ArtPipeStageConsumer listening on art.model.hunyuan3dpaint.tasks ``` The texturing artifact (`hunyuan3d_textured_a35903f8.glb`, 4.1 MB) pulled back out of MinIO and inspected: ``` vertices 32,505 faces 42,552 has UVs: True baseColorTexture: (2048, 2048) RGB texture mean RGB: [139.4, 84.9, 15.0] std: [51.2, 52.4, 33.4] distinct colors: 174,625 ``` Golden/amber matching the prompt, 174k distinct colours — a real generated PBR texture, not a flat fill or a Blender fallback bake. ### Note for downstream consumers Texturing now emits a **mesh**, not a texture set. Old `sd_turbo_tex` produced `Kind: texture-set` (PNG); Hunyuan3DPaint produces `Kind: mesh-glb`. The export stage's PBR-param mapping and the rigging stage's mesh walk-back both already handle this (the orchestrator tests cover both shapes), but anything else reading texturing output by `Kind` should be checked. ### Shipped - spikersoft-artpipe #32 — manifest weight patterns (74.9 GB -> ~15 GB), `custom_rasterizer` build, `install.verify_imports` build gate, offline loader + scoped `trust_remote_code`, `scripts/try-hunyuan-build.sh` - spikersoft-infrastructure #148 — stack resident -> ModelQueue, own baked image, `/mnt/fusionio` bind dropped, `VramMB` measured at 14000 - spikersoft-backend #475 — texturing -> Hunyuan3DPaint, `ModelQueueStages` in the shared overlay, `image_path` walk-back, routing guard tests ### Follow-ups tracked separately #839 (ModelEnvImages robustness) — PRs open: spikersoft-artpipe#36, spikersoft-backend#476. Both were needed to get here: `triposg`'s unbuildable `fvdb` pin aborted the serial build loop before `hunyuan3dpaint`, and `triposr` was missing from both default image sets despite being what prod routes modeling to. This lane currently depends on two images built by explicit `workflow_dispatch`; those PRs make the default set reproduce them. Closing.

spikerj closed this issue

2026-07-25 03:25:01 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#832