[Bug][Prod] ArtPipeProcessor image: /home/app not writable + no HF_HOME → HuggingFace cache PermissionError blocks model weight downloads #495

New Issue

2026-07-12T02:22:04Z

spikerj commented

2026-07-12 02:22:04 +00:00

Summary

Art-pipe GPU-model stages fail at inference when they download model weights: HuggingFace tries to create its cache under the app user's home /home/app, which the container never creates or makes writable, so it dies with PermissionError: [Errno 13] Permission denied: '/home/app'. There is also no HF_HOME set to redirect the cache elsewhere. This blocks concept art (SDXL Lightning) and will hit every model that pulls weights from HF.

Symptom (prod, learn.spikersoft.com — concept stage)

File ".../models/SDXLLightning/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", ...
  os.makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/app'
  at SubprocessArtPipeStageExecutor.ExecuteAsync(...)

(Reached only after fixing the upstream venv problems — see "context" below; this is the next blocker in the chain.)

Root cause (image)

SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile:

line 8: useradd -r -g app app — creates app as a system user. useradd -r does not create a home directory.
line 101: mkdir -p /app/work && chown -R app:app /app — chowns /app only.
line 103: USER app

So the process runs as app with home /home/app, but /home/app was never created or chowned to app. HuggingFace defaults its cache to ~/.cache/huggingface → /home/app/.cache/..., and os.makedirs('/home/app/...') fails because /home/app doesn't exist and /home isn't writable by app. No HF_HOME/HF_HUB_CACHE/XDG_CACHE_HOME is set anywhere (checked the six artpipe stacks and ArtPipeProcessor config) to redirect it.

Fix (pick one; both is fine)

Image (durable, preferred): in the Dockerfile, either
- RUN mkdir -p /home/app && chown -R app:app /home/app (give the user a real home), and/or
- ENV HF_HOME=/opt/art_pipe/.cache/huggingface so weights land on the persistent, app-owned mount (download once, shared by all model workers, survives restarts).
Stack env (fast, no image rebuild): add to each of the six artpipe stacks' environment::
```
- HF_HOME=/opt/art_pipe/.cache/huggingface
```
/opt/art_pipe is the rw FusionIO mount (already app-owned), so HF can create .cache/huggingface there. Concept only needs spikersoft-artpipe-modeling today; the other five (safety, sdxl, blender, hunyuan, triposr) need it before those stages run.

Prefer routing HF cache to the mount either way — otherwise weights re-download into the container's ephemeral layer on every reschedule (multi-GB per model).

Context (why this surfaced now)

Fresh GPU-host provisioning after the artpipe-gpu node label was lost (network-hardware upgrade). Fixing that exposed a chain of unbootstrapped/misprovisioned state on SERVER, all in the #357/#359/#432/#486 family:

artpipe-gpu label missing → workers unschedulable (fixed: relabeled SERVER).
Per-model venvs not bootstrapped → ModuleNotFoundError: No module named 'torch' (#486 = SafetyCheck; also hit SDXLLightning).
Server .venv had no pip (truncated during an out-of-space event) → recreated.
/opt/art_pipe tree had mixed root/app ownership → chown -R app:app on the mount.
This ticket: /home/app not writable + no HF_HOME → HF cache PermissionError.

Related: #357, #359, #432, #486. Consider folding the recurring theme into a single "art_pipe prod GPU-host provisioning is not reproducible" hardening task.

Filed from a live prod debugging session (concept-art generation on SERVER).

## Summary Art-pipe GPU-model stages fail at inference when they download model weights: HuggingFace tries to create its cache under the `app` user's home `/home/app`, which the container never creates or makes writable, so it dies with `PermissionError: [Errno 13] Permission denied: '/home/app'`. There is also no `HF_HOME` set to redirect the cache elsewhere. This blocks concept art (SDXL Lightning) and will hit every model that pulls weights from HF. ## Symptom (prod, learn.spikersoft.com — concept stage) ``` File ".../models/SDXLLightning/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", ... os.makedirs(head, exist_ok=exist_ok) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '/home/app' at SubprocessArtPipeStageExecutor.ExecuteAsync(...) ``` (Reached only after fixing the upstream venv problems — see "context" below; this is the next blocker in the chain.) ## Root cause (image) `SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile`: - line 8: `useradd -r -g app app` — creates `app` as a **system** user. `useradd -r` does **not** create a home directory. - line 101: `mkdir -p /app/work && chown -R app:app /app` — chowns `/app` only. - line 103: `USER app` So the process runs as `app` with home `/home/app`, but `/home/app` was never created or chowned to `app`. HuggingFace defaults its cache to `~/.cache/huggingface` → `/home/app/.cache/...`, and `os.makedirs('/home/app/...')` fails because `/home/app` doesn't exist and `/home` isn't writable by `app`. No `HF_HOME`/`HF_HUB_CACHE`/`XDG_CACHE_HOME` is set anywhere (checked the six artpipe stacks and ArtPipeProcessor config) to redirect it. ## Fix (pick one; both is fine) 1. **Image (durable, preferred):** in the Dockerfile, either - `RUN mkdir -p /home/app && chown -R app:app /home/app` (give the user a real home), **and/or** - `ENV HF_HOME=/opt/art_pipe/.cache/huggingface` so weights land on the persistent, app-owned mount (download once, shared by all model workers, survives restarts). 2. **Stack env (fast, no image rebuild):** add to each of the six artpipe stacks' `environment:`: ```yaml - HF_HOME=/opt/art_pipe/.cache/huggingface ``` `/opt/art_pipe` is the rw FusionIO mount (already app-owned), so HF can create `.cache/huggingface` there. Concept only needs `spikersoft-artpipe-modeling` today; the other five (safety, sdxl, blender, hunyuan, triposr) need it before those stages run. Prefer routing HF cache to the mount either way — otherwise weights re-download into the container's ephemeral layer on every reschedule (multi-GB per model). ## Context (why this surfaced now) Fresh GPU-host provisioning after the `artpipe-gpu` node label was lost (network-hardware upgrade). Fixing that exposed a chain of unbootstrapped/misprovisioned state on SERVER, all in the #357/#359/#432/#486 family: 1. `artpipe-gpu` label missing → workers unschedulable (fixed: relabeled SERVER). 2. Per-model venvs not bootstrapped → `ModuleNotFoundError: No module named 'torch'` (#486 = SafetyCheck; also hit SDXLLightning). 3. Server `.venv` had no pip (truncated during an out-of-space event) → recreated. 4. `/opt/art_pipe` tree had mixed root/app ownership → `chown -R app:app` on the mount. 5. **This ticket:** `/home/app` not writable + no `HF_HOME` → HF cache `PermissionError`. Related: #357, #359, #432, #486. Consider folding the recurring theme into a single "art_pipe prod GPU-host provisioning is not reproducible" hardening task. --- _Filed from a live prod debugging session (concept-art generation on SERVER)._

spikerj commented

2026-07-12 02:33:13 +00:00

Fix up in spikersoft-backend PR #216 (fix/495-artpipe-hf-cache-home). Root cause is exactly as diagnosed here: app is a useradd -r system user with no writable home, so HF's cache defaults to /home/app/.cache/... and 403s.

Approach — took the durable image path plus a code-level redirect:

ApplyWorkerEnvironment (shared by the subprocess and resident executors) now sets HF_HOME/HF_HUB_CACHE/XDG_CACHE_HOME onto the app-owned rw pipeline mount (/opt/art_pipe/.cache/...), so weights also persist across reschedules instead of re-pulling into the ephemeral layer.
Dockerfile now gives app a real home (useradd -m -d /home/app) for pip/git/other ~-defaulting tools.
2 new xUnit tests; full ArtPipeProcessor.Tests green (227), SpikerSoft.UnitTests.slnf clean.

Will comment+close once #216 merges. Leaving open until then per tracker rule.

Fix up in spikersoft-backend PR #216 (`fix/495-artpipe-hf-cache-home`). Root cause is exactly as diagnosed here: `app` is a `useradd -r` system user with no writable home, so HF's cache defaults to `/home/app/.cache/...` and 403s. Approach — took the durable image path plus a code-level redirect: - `ApplyWorkerEnvironment` (shared by the subprocess **and** resident executors) now sets `HF_HOME`/`HF_HUB_CACHE`/`XDG_CACHE_HOME` onto the app-owned rw pipeline mount (`/opt/art_pipe/.cache/...`), so weights also persist across reschedules instead of re-pulling into the ephemeral layer. - Dockerfile now gives `app` a real home (`useradd -m -d /home/app`) for pip/git/other `~`-defaulting tools. - 2 new xUnit tests; full ArtPipeProcessor.Tests green (227), `SpikerSoft.UnitTests.slnf` clean. Will comment+close once #216 merges. Leaving open until then per tracker rule.

spikerj commented

2026-07-12 02:34:53 +00:00

Resolved in spikersoft-backend PR #216 (merged to master, commit 0089f72). ApplyWorkerEnvironment now redirects HF_HOME/HF_HUB_CACHE/XDG_CACHE_HOME onto the app-owned rw pipeline mount (fixes the /home/app PermissionError for both subprocess and resident executors, and persists weights across reschedules), and the Dockerfile gives app a real home (useradd -m -d /home/app). Covered by 2 new xUnit tests. Deploy rolls automatically on the master merge. Closing.

Resolved in spikersoft-backend PR #216 (merged to `master`, commit 0089f72). `ApplyWorkerEnvironment` now redirects `HF_HOME`/`HF_HUB_CACHE`/`XDG_CACHE_HOME` onto the app-owned rw pipeline mount (fixes the `/home/app` PermissionError for both subprocess and resident executors, and persists weights across reschedules), and the Dockerfile gives `app` a real home (`useradd -m -d /home/app`). Covered by 2 new xUnit tests. Deploy rolls automatically on the master merge. Closing.

spikerj closed this issue

2026-07-12 02:35:03 +00:00

spikerj referenced this issue

2026-07-12 04:24:46 +00:00

[Bug][Prod] /mnt/fusionio/spikersoft/ai missing embedding model — book pipeline dies at embeddings (nomic-embed-text-v2-moe.f16.gguf not found) #497

spikerj referenced this issue

2026-07-12 07:19:24 +00:00

[Bug][Prod][GPU] image-description crash-loops on gated HuggingFace model — no HF token for Qwen3-VL-8B (401 GatedRepoError) #502

spikerj referenced this issue

2026-07-13 00:40:18 +00:00

[ArtPipe] Baked images Phase 0: build scaffolding (Dockerfiles, CI, setup flags) + disk/limits go-no-go gates (epic #515) #516

spikerj referenced this issue