[Bug][ArtStudio] Concept stage fails for every asset offline — SDXL loader completeness check + un-baked Lightning LoRA (fix: artpipe PR #25) #699

New Issue

2026-07-18T06:10:13Z

spikerj commented

2026-07-18 06:10:13 +00:00

Symptom

Art Studio concept stage fails for every asset under HF_HUB_OFFLINE=1 (prod monolith). sdxl_lightning can't load its model → stage publishes art.asset.stage.failed.concept ~4s after acquiring a GPU lease (no generation). Concept is stage 1, so the whole asset fails.

Root cause (both proven in the deployed container)

from_pretrained(local_files_only=True) runs a strict full-repo completeness check and raises IncompleteSnapshotError on repo docs/samples (.gitattributes, 01.png, LICENSE.md, README) that the weight_patterns allow-list correctly excluded (#537). The actual fp16 component weights are all present.
The Lightning LoRA ByteDance/SDXL-Lightning lives in a second repo and was never baked (install.weights pre-downloads only one repo → the SDXL base).

Fix — spikersoft-artpipe PR #25

Loader (sdxl_lightning.py): offline, resolve the base to its local snapshot dir with the fp16 allow-patterns, load from that path (variant=fp16, local_files_only=True). Online unchanged.
Prebake (venv_setup.py): new optional install.extra_weights list, baked alongside primary weights.
Manifest (SDXLLightning/artpipe.json): declare the 3 step-LoRA files under extra_weights.
Tests for extra_weights prefetch; existing behavior preserved. Offline load verified in the live env.

Deploy

Merging PR #25 → model-env-images.yml rebuilds prodstages (bakes the LoRA) → cascades to backend ArtPipeProcessor → redeploys the 36 GiB image. Monolith + gpu-coordinator dynamic-VRAM design unchanged.

## Symptom Art Studio **concept stage fails for every asset** under `HF_HUB_OFFLINE=1` (prod monolith). `sdxl_lightning` can't load its model → stage publishes `art.asset.stage.failed.concept` ~4s after acquiring a GPU lease (no generation). Concept is stage 1, so the whole asset fails. ## Root cause (both proven in the deployed container) 1. `from_pretrained(local_files_only=True)` runs a strict full-repo completeness check and raises `IncompleteSnapshotError` on repo docs/samples (`.gitattributes`, `01.png`, `LICENSE.md`, README) that the `weight_patterns` allow-list correctly excluded (#537). The actual fp16 component weights are all present. 2. The Lightning LoRA `ByteDance/SDXL-Lightning` lives in a second repo and was never baked (`install.weights` pre-downloads only one repo → the SDXL base). ## Fix — spikersoft-artpipe PR #25 - Loader (`sdxl_lightning.py`): offline, resolve the base to its local snapshot dir with the fp16 allow-patterns, load from that path (`variant=fp16`, `local_files_only=True`). Online unchanged. - Prebake (`venv_setup.py`): new optional `install.extra_weights` list, baked alongside primary weights. - Manifest (`SDXLLightning/artpipe.json`): declare the 3 step-LoRA files under `extra_weights`. - Tests for extra_weights prefetch; existing behavior preserved. Offline load verified in the live env. ## Deploy Merging PR #25 → `model-env-images.yml` rebuilds `prodstages` (bakes the LoRA) → cascades to backend ArtPipeProcessor → redeploys the 36 GiB image. Monolith + gpu-coordinator dynamic-VRAM design unchanged.

spikerj commented

2026-07-18 06:21:48 +00:00

PR #25 merged to artpipe main (merge 9984a25). Model rebuild kicked off automatically — model-env-images run 9507 in_progress (rebuilds prodstages, baking the Lightning LoRA + loader fix). On success it cascade-dispatches the backend ArtPipeProcessor to rebuild+redeploy the runnable 36 GiB image. Keeping this open until a concept generation is verified end-to-end on the redeployed image.

spikerj referenced this issue

2026-07-18 20:34:50 +00:00

[ArtPipe][Bug] Modeling stage fails offline — TripoSR's DINO image encoder (facebook/dino-vitb16) never baked (same class as #699) #702

spikerj commented

2026-07-22 02:47:56 +00:00

Verified fixed and merged. spikersoft-artpipe PR #25 ("fix(sdxl): concept stage fails offline — bake the Lightning LoRA + load base from local snapshot", merge 9984a25, fix commit 15c56f1) is an ancestor of origin/main. It bakes the SDXL Lightning LoRA into the env image, loads the base from a local snapshot (no network), tightens venv_setup.py weight patterns, and adds tests/test_venv_setup_weights.py. Matches this ticket's root cause (SDXL loader completeness check + un-baked Lightning LoRA). Closing.

Verified fixed and merged. spikersoft-artpipe **PR #25** ("fix(sdxl): concept stage fails offline — bake the Lightning LoRA + load base from local snapshot", merge `9984a25`, fix commit `15c56f1`) is an ancestor of `origin/main`. It bakes the SDXL Lightning LoRA into the env image, loads the base from a local snapshot (no network), tightens `venv_setup.py` weight patterns, and adds `tests/test_venv_setup_weights.py`. Matches this ticket's root cause (SDXL loader completeness check + un-baked Lightning LoRA). Closing.

spikerj closed this issue

2026-07-22 02:48:01 +00:00

spikerj referenced this issue

2026-07-25 00:44:20 +00:00

Art Studio: make the Hunyuan3D texturing lane actually run in production (never textured a single asset) #832

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#699