[ArtPipe][CI] Phase 2 blocked: sdxllightning env image is 98 GB — unfiltered snapshot_download bakes the entire SDXL repo; push dies #537
Assignees
spikerj (Joseph Spiker)
Clear assignees
spikerj (Joseph Spiker)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: spikerj/spikersoft-issues#537
Reference in New Issue
Block a user
Title: [ArtPipe][CI] Phase 2 blocked: sdxllightning env image is 98 GB — unfiltered snapshot_download bakes the entire SDXL repo; push dies
Run 10782 (first Phase 2 build on the 4090 runner, post-#534) failed at ~05:10Z after 33 min. NOT infra this time — MinIO/registry healthy throughout. Evidence from the runner host:
artpipe-model-env-sdxllightning:latestbuilt locally at 98.1 GB (design range per the workflow header: 10–40 GB; safetycheck is 22.5 GB and pushed fine at that size).docker history: a single 76.9 GB layer from the weights pre-download RUN step.Root cause:
src/artpipe/venv_setup.py:263—snapshot_download(weights, local_dir_use_symlinks=False)with no allow/ignore patterns, andmodels/SDXLLightning/artpipe.jsondeclaresweights: stabilityai/stable-diffusion-xl-base-1.0. That HF repo contains fp32+fp16 variants of every component plus .bin/.safetensors duplicates and ONNX/OpenVINO exports — the full snapshot is ~10x what the backend loads (diffusersfrom_pretrained(..., variant=fp16)uses one variant).Fix options:
snapshot_download(weights, ignore_patterns=["*.bin", "*.onnx", "*openvino*", "*.ckpt", "*flax*", "*fp32*"])or an explicitallow_patternslist per manifest (add an optionalinstall.weight_patternsfield to artpipe.json).DiffusionPipeline.from_pretrained(variant='fp16')) so only runtime-needed files land in the layer.Also worth a go/no-go gate (#516 mentioned one): fail the build fast if an env image exceeds ~50 GB, instead of spending 30 min building + failing on push. Blocks #518 Phase 2 (sdxllightning + triposr unpublished). Related: #515/#516/#518/#534.
Addendum: the same 98 GB build is what filled laptop-server's disk tonight (~05:14Z): leftover layers/cache from the killed laptop runs (10775/10778) filled / → gitea_postgres crash-looped on 'No space left on device' → git/API/CI down for ~25 min until manual cleanup. Cross-ref #514 — the prune timer needs to cover build cache, and the size gate here would have prevented the whole cascade.
Third failure mode confirmed — 98 GB pushes don't just fail, they HANG (2026-07-13 06:15Z). Run 10784 (with the 576b4bd cache sweep but NOT the snapshot_download filter) rebuilt sdxllightning at 98.1 GB and has now been 'pushing' for ~30 min with zero progress: no PATCH/PUT blob requests reaching Gitea in 5+ min (only my manifest GETs → 404), dockerd idle at 6%, no docker image events on the 4090. Classic silent docker-push stall on a broken upload session — it will sit there until the Actions job timeout unless cancelled.
Recommendation: cancel 10784; don't re-dispatch until the weights filter from this ticket lands (a re-run bakes the same 98 GB and rolls the same dice). Score so far: 22.5 GB safetycheck pushed fine repeatedly; 98 GB sdxl is 0-for-2 (one push failure, one push hang). The fix + a ~50 GB CI size gate turns this class of loss into a 30-second failure.
QA watch 2026-07-13 ~18:40Z — filter fix NOT landed, failures continuing.
artpipe main venv_setup.py:263-264 is still the unfiltered call: snapshot_download(weights, local_dir_use_symlinks=False) — no ignore_patterns / allow_patterns / fp16-variant selection.
ModelEnvImages failed twice more today AFTER the cache-sweep fix (746b1b6) merged: run 10819 (16:17Z) and 10820 (18:25Z); run 10821 is in flight now on the 4090 (routing #534 works for env images). Failure step is unverifiable (Gitea API exposes no action logs) but consistent with this ticket — the cache sweep fixes the BuildKit disk leak, not the 98GB image itself, so expect sdxllightning to keep dying at push until the download filter lands.
Silver lining: builds now run on the 4090 (3.6T disk, 24% used), so the laptop-server disk-full → gitea_postgres cascade (#514) is decoupled from these failures. Registry push size + tag-race (#536) exposure remains.
QA watch 2026-07-13 ~18:55Z — DOWNGRADE from hard blocker to optimization: the push now SUCCEEDS from the 4090. Run 10821 (success 18:48Z) published artpipe-model-env-sdxllightning:latest at 18:36Z — 65.2 GB compressed in the registry (~98 GB uncompressed) — plus triposr (14.4 GB). Narrative fits earlier failures being cache-cold/push-size on weaker infra: 10819 published blender then died; 10821 with hot caches finished sdxl + triposr.
Still worth fixing (keep open): 65 GB of MinIO registry storage and a ~98 GB pull/extract on every deploying node, mostly unused fp32/onnx/openvino duplicates of weights we only need once (fp16). The venv_setup.py:263 ignore_patterns/variant filter + CI size gate (~50 GB) remain the right fix — now as cost/latency hygiene, not a Phase-2 blocker.
Additional finding while explaining the image size (2026-07-13 ~19:35Z) — the bloat has an exact inverse bug: the weights the model DOES need at runtime are NOT all baked.
Bloat mechanics, precisely: models/SDXLLightning/artpipe.json declares install.weights = stabilityai/stable-diffusion-xl-base-1.0; venv_setup.py _download_model_weights() snapshot_download()s it unfiltered → the full repo including fp32 safetensors + fp16 safetensors + legacy .bin duplicates + complete ONNX + OpenVINO export trees (~90 GB of files, 76.9 GB weights layer measured). Runtime (src/artpipe/models/sdxl_lightning.py:72-76) loads with variant='fp16' → only the fp16 safetensors + config/tokenizer (~7 GB) are ever read. ~90% of the image is formats the code cannot touch.
The inverse gap: sdxl_lightning.py:93 does load_lora_weights('ByteDance/SDXL-Lightning', weight_name=step lora) at FIRST INFERENCE — that repo is not in the manifest and not pre-downloaded, so the 65 GB baked image still needs Hugging Face network access on first use (and fails offline). Fix both together: ignore_patterns/allow_patterns (fp16 + configs only) for the base, AND add ByteDance/SDXL-Lightning (or just the specific lora_file names in _LORA_WEIGHTS) to the pre-download so the image is genuinely self-contained. Then the CI size gate (~15-20 GB would suffice for this model, not 50).
Recurred tonight (run 6554: sequential step died 23:58Z pushing sdxllightning — same signature). Fix PR: spikersoft-artpipe#19 — install.weight_patterns allow-list in venv_setup + SDXLLightning manifest pinned to the fp16 loader set (~77GB weights layer → ~7GB). Verify via workflow_dispatch images=sdxllightning after merge. — macbook-claude-session
Fixed and verified end-to-end. spikersoft-artpipe PR #19 (
install.weight_patterns→snapshot_download(allow_patterns=…)) merged and the auto-triggeredbuild_env_imagesrun 6612 completed success on the 4090:artpipe-model-env-sdxllightning:latest+:44daa91pushed — digestsha256:9820f89fd41d2aa048ac92e08a3c77ca3badbb5da3e1374a2197937cf72984c8, registry confirms both tags at 2026-07-14T01:48Z. The weights layer pushed in ~8 min instead of dying (fp16 allow-list keeps the snapshot to the ~7 GB variant set instead of the full 76.9 GB repo).artpipe-model-env-triposrrebuilt/pushed in the same run (digestsha256:12aec0e0…), base layers mounted from the sdxllightning push.Phase 2 is unblocked from the artpipe side. Closing.
— macbook-claude-session