[Bug][Prod][GPU] image-description crash-loops on gated HuggingFace model — no HF token for Qwen3-VL-8B (401 GatedRepoError) #502

New Issue

2026-07-12T07:19:24Z

spikerj commented

2026-07-12 07:19:24 +00:00

Summary

The image-description worker (Qwen3-VL captioner) now gets its GPU lease (the #500 coordinator fix is live), loads the model on first request — and crash-loops because the model repo is gated on HuggingFace and no HF auth token is configured. Books with images still never get captioned; the failure moved one step further down the chain (#403 → #500 → this).

Evidence (2026-07-12 ~07:18, node 4090)

huggingface_hub.errors.GatedRepoError: 401 Client Error
Cannot access gated repo for url
  https://huggingface.co/prithivMLmods/Qwen3-VL-8B-Abliterated-Caption-it/resolve/main/config.json
Access to model prithivMLmods/Qwen3-VL-8B-Abliterated-Caption-it is restricted.
You must have access to it and be authenticated to access it. Please log in.
  → VisionModel.from_pretrained(...) → AutoConfig.from_pretrained(...) → cached_files(...)
OSError: You are trying to access a gated repo.

Task cycles: Running 4 min ago / Shutdown 5 min ago on node 4090 — it dies on every caption attempt (the worker process exits on the unhandled from_pretrained exception).

Root cause

The image-description service env carries no HuggingFace credentials — only GPU_MODEL_IDENTIFIER=qwen3-vl-8b. No HF_TOKEN / HUGGING_FACE_HUB_TOKEN, no HF_HOME, no offline/local model path. So from_pretrained hits the HF Hub anonymously and the gated repo returns 401.

Fix options

(a) Authenticated pull — add an HF_TOKEN (Docker secret) whose HF account has been granted access to prithivMLmods/Qwen3-VL-8B-Abliterated-Caption-it, injected as HUGGING_FACE_HUB_TOKEN/HF_TOKEN. Also set HF_HOME to a writable/persistent cache (see #495 for the ArtPipe HF-cache pattern) so it isn't re-downloaded every restart.

(b) Local mirror (infra-consistent, preferred) — pre-stage the weights on the model volume (the /mnt/fusionio/spikersoft/ai pattern from #497) and run the worker with HF_HUB_OFFLINE=1 pointing at the local dir. Avoids a runtime dependency on HF availability/auth on the hot path.

(c) Switch to a non-gated caption model if licensing/access to this abliterated variant isn't guaranteed.

Impact

Image-bearing book uploads still hang at Stage=ImageDescriptions (the worker now crashes instead of waiting on a lease). Same user-facing symptom, next root cause.

#500 — gpu-coordinator lease (now resolved: lease grants, which surfaced this).
#403 — image-description deploy.
#497 — model-provisioning-on-/mnt/fusionio/spikersoft/ai pattern (embedding model).
#495 — ArtPipe HF cache / HF_HOME permission pattern (closed) — reference for the cache env.

## Summary The `image-description` worker (Qwen3-VL captioner) now gets its GPU lease (the #500 coordinator fix is live), loads the model on first request — and **crash-loops** because the model repo is **gated on HuggingFace** and no HF auth token is configured. Books with images still never get captioned; the failure moved one step further down the chain (#403 → #500 → this). ## Evidence (2026-07-12 ~07:18, node 4090) ``` huggingface_hub.errors.GatedRepoError: 401 Client Error Cannot access gated repo for url https://huggingface.co/prithivMLmods/Qwen3-VL-8B-Abliterated-Caption-it/resolve/main/config.json Access to model prithivMLmods/Qwen3-VL-8B-Abliterated-Caption-it is restricted. You must have access to it and be authenticated to access it. Please log in. → VisionModel.from_pretrained(...) → AutoConfig.from_pretrained(...) → cached_files(...) OSError: You are trying to access a gated repo. ``` Task cycles: `Running 4 min ago / Shutdown 5 min ago` on node 4090 — it dies on every caption attempt (the worker process exits on the unhandled `from_pretrained` exception). ## Root cause The image-description service env carries **no HuggingFace credentials** — only `GPU_MODEL_IDENTIFIER=qwen3-vl-8b`. No `HF_TOKEN` / `HUGGING_FACE_HUB_TOKEN`, no `HF_HOME`, no offline/local model path. So `from_pretrained` hits the HF Hub anonymously and the gated repo returns 401. ## Fix options **(a) Authenticated pull** — add an `HF_TOKEN` (Docker secret) whose HF account has been granted access to `prithivMLmods/Qwen3-VL-8B-Abliterated-Caption-it`, injected as `HUGGING_FACE_HUB_TOKEN`/`HF_TOKEN`. Also set `HF_HOME` to a writable/persistent cache (see #495 for the ArtPipe HF-cache pattern) so it isn't re-downloaded every restart. **(b) Local mirror (infra-consistent, preferred)** — pre-stage the weights on the model volume (the `/mnt/fusionio/spikersoft/ai` pattern from #497) and run the worker with `HF_HUB_OFFLINE=1` pointing at the local dir. Avoids a runtime dependency on HF availability/auth on the hot path. **(c)** Switch to a non-gated caption model if licensing/access to this abliterated variant isn't guaranteed. ## Impact Image-bearing book uploads still hang at `Stage=ImageDescriptions` (the worker now crashes instead of waiting on a lease). Same user-facing symptom, next root cause. ## Related - #500 — gpu-coordinator lease (now resolved: lease grants, which surfaced this). - #403 — image-description deploy. - #497 — model-provisioning-on-`/mnt/fusionio/spikersoft/ai` pattern (embedding model). - #495 — ArtPipe HF cache / `HF_HOME` permission pattern (closed) — reference for the cache env.

spikerj referenced this issue

2026-07-12 07:19:24 +00:00

[Bug][Infra][GPU] image-description lease starves — gpu-coordinator single-lane budget still on SERVER (8192MB) vs 16000MB needed; enable 4090 lane #500

spikerj referenced this issue

2026-07-12 07:45:34 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services #503

spikerj referenced this issue

2026-07-12 17:35:43 +00:00

[Bug][Prod][GPU] image-description permanently dead-letters captions on null gpu_client — AttributeError misclassified as permanent (data loss) #506

spikerj commented

2026-07-12 17:35:55 +00:00

Related: filed #506 — a separate image-description defect surfaced while investigating this service. When the GPU-lease client is momentarily None (startup race before _gpu_client is set, or coordinator transiently unreachable per #503), _gpu_client.acquire() raises AttributeError, which is misclassified as a PERMANENT failure (_PERMANENT_FAILURES tuple, image_description_service.py:1004) and dead-lettered without retry → silent caption data loss. Confirmed 2 images DLQ'd 07:33 on 4090. Distinct fix from the HF-token issue here.

Related: filed #506 — a *separate* image-description defect surfaced while investigating this service. When the GPU-lease client is momentarily None (startup race before _gpu_client is set, or coordinator transiently unreachable per #503), `_gpu_client.acquire()` raises AttributeError, which is misclassified as a PERMANENT failure (_PERMANENT_FAILURES tuple, image_description_service.py:1004) and dead-lettered without retry → silent caption data loss. Confirmed 2 images DLQ'd 07:33 on 4090. Distinct fix from the HF-token issue here.

spikerj commented

2026-07-12 18:22:54 +00:00

Superseded by the MinIO model architecture and verified working — closing. The gated-repo problem was eliminated rather than tokened-around: the worker now syncs Qwen/Qwen3-VL-8B-Instruct (official, ungated) from the ai-models bucket (backend #228 + infra #41) and loads from disk; prod never contacts HuggingFace. Evidence: 110 completed captions in Mongo generated by the bucket-loaded model.

Superseded by the MinIO model architecture and verified working — closing. The gated-repo problem was eliminated rather than tokened-around: the worker now syncs `Qwen/Qwen3-VL-8B-Instruct` (official, ungated) from the `ai-models` bucket (backend #228 + infra #41) and loads from disk; prod never contacts HuggingFace. Evidence: **110 completed captions** in Mongo generated by the bucket-loaded model.

spikerj closed this issue

2026-07-12 18:22:55 +00:00

spikerj referenced this issue

2026-07-21 04:11:01 +00:00

Dream globe: context countries nearly invisible; show grey state boundaries for supported countries #763

spikerj referenced this issue

2026-07-21 04:11:05 +00:00

Dream globe: choropleth invisible while hovering and past a zoom threshold (small polygons) #764

spikerj referenced this issue

2026-07-21 04:11:13 +00:00

Dream globe: chip click flies globe to destination; polygon click offers "Explore in Geography" #765

spikerj referenced this issue