[ArtPipe][Bug] Modeling stage fails offline — TripoSR's DINO image encoder (facebook/dino-vitb16) never baked (same class as #699) #702

New Issue

2026-07-18T20:34:50Z

spikerj commented

2026-07-18 20:34:50 +00:00

First real Art Studio asset (2026-07-18 ~20:27Z): concept succeeded (thanks to #699 / artpipe PR #25), then modeling failed with:

huggingface_hub.file_download.OfflineModeIsEnabled: Offline mode is enabled. Tried to reach https://huggingface.co/facebook/dino-vitb16/resolve/main/config.json

Root cause — same failure class as #699 (secondary HF repo not declared in the manifest):

The prodstages monolith runs with HF_HUB_OFFLINE=1 by design; models/TripoSR/artpipe.json bakes only stabilityai/TripoSR (config.yaml + model.ckpt via a setup_command into a local weights/ dir).
TripoSR's checkpoint config declares image_tokenizer: facebook/dino-vitb16 (DINOSingleImageTokenizer), loaded via transformers at TSR.from_pretrained(...) construction — that repo was never baked.
Verified in the live spikersoft-artpipe-modeling container on 4090: models--facebook--dino-vitb16/ in the baked HF cache is EMPTY, mtime 20:27Z = the skeleton dir created by the failed job's aborted download attempt.

Fix: add facebook/dino-vitb16 to install.extra_weights in the TripoSR manifest (the mechanism PR #25 introduced; SDXLLightning is currently its only user), with weight_patterns limited to config.json / preprocessor_config.json / pytorch_model.bin per the #537 image-size discipline. Deploy = merge → model-env-images.yml rebuilds prodstages (default set) → cascade → ArtPipeProcessor redeploy → Retry the failed stage in Art Studio.

Watch-list note: this is the first asset ever pushed through the full pipeline in prod — rigging/animation stage manifests have never executed there and may hide the same class of gap.

First real Art Studio asset (2026-07-18 ~20:27Z): concept succeeded (thanks to #699 / artpipe PR #25), then **modeling failed** with: > huggingface_hub.file_download.OfflineModeIsEnabled: Offline mode is enabled. Tried to reach https://huggingface.co/facebook/dino-vitb16/resolve/main/config.json **Root cause** — same failure class as #699 (secondary HF repo not declared in the manifest): - The prodstages monolith runs with `HF_HUB_OFFLINE=1` by design; `models/TripoSR/artpipe.json` bakes only `stabilityai/TripoSR` (`config.yaml` + `model.ckpt` via a setup_command into a local `weights/` dir). - TripoSR's checkpoint config declares `image_tokenizer: facebook/dino-vitb16` (`DINOSingleImageTokenizer`), loaded via transformers at `TSR.from_pretrained(...)` construction — that repo was never baked. - Verified in the live `spikersoft-artpipe-modeling` container on 4090: `models--facebook--dino-vitb16/` in the baked HF cache is EMPTY, mtime 20:27Z = the skeleton dir created by the failed job's aborted download attempt. **Fix**: add `facebook/dino-vitb16` to `install.extra_weights` in the TripoSR manifest (the mechanism PR #25 introduced; SDXLLightning is currently its only user), with `weight_patterns` limited to `config.json` / `preprocessor_config.json` / `pytorch_model.bin` per the #537 image-size discipline. Deploy = merge → `model-env-images.yml` rebuilds prodstages (default set) → cascade → ArtPipeProcessor redeploy → Retry the failed stage in Art Studio. **Watch-list note**: this is the first asset ever pushed through the full pipeline in prod — rigging/animation stage manifests have never executed there and may hide the same class of gap.

spikerj closed this issue

2026-07-18 20:36:55 +00:00

spikerj referenced this issue

2026-07-19 01:29:52 +00:00

[CI][Backend] Tier-3 artpipe finals builds keep getting superseded/cancelled before pushing — dino fix (#702) never reached the registry #717

spikerj referenced this issue

2026-07-19 03:50:03 +00:00

[Infra][Data-integrity] Corrupt blob in MinIO-backed container registry — 17GB env-image layer unreadable at one part, blocked the #702 fix chain all night #720

spikerj commented

2026-07-19 03:50:21 +00:00

Deployment update: the fix has been merged since 20:36Z but couldn't reach prod — first the tier-3 builds were cancelled by CI concurrency (#717), then the tier-2 env image's 17GB layer turned out to be CORRUPT in the MinIO-backed registry (#720 — one unreadable part at offset 6.25GB; every pull failed there all night). Re-dispatched model-env-images (images: prodstages) at ~04:30Z on idle runners; fresh blobs → cascade → tier-3 → auto-deploy. Monitoring the chain; modeling Retry is safe once spikersoft-artpipe-modeling rolls off its 20:04Z spec.

spikerj referenced this issue

2026-07-19 07:46:30 +00:00

[Infra][Data-integrity] Corrupt blob in MinIO-backed container registry — 17GB env-image layer unreadable at one part, blocked the #702 fix chain all night #720

spikerj commented

2026-07-19 08:00:49 +00:00

Fix verified LIVE (2026-07-19 08:0xZ): after the corrupt-blob rebuild (#720) and the CI queue drama (#717), the cascade completed and spikersoft-artpipe-modeling redeployed at 08:00:20Z on the new prodstages image. Confirmed inside the running container: /opt/art_pipe/.cache/huggingface/hub/models--facebook--dino-vitb16/snapshots/.../config.json is baked — the exact file DINOSingleImageTokenizer hf_hub_download()s at TSR.from_pretrained(). Modeling Retry is safe. Closing.

spikerj referenced this issue

2026-07-19 13:38:31 +00:00

[ArtPipe][Bug] Export stage dead on first prod asset — Blender's glTF exporter missing numpy (--no-install-recommends dropped it) + probe chain masked the real error #721

spikerj referenced this issue

2026-07-29 16:04:22 +00:00

[CI][Backend] Tier-3 artpipe finals builds keep getting superseded/cancelled before pushing — dino fix (#702) never reached the registry #717

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#702