[Backend][Bug] Embeddings (and latent: quiz-generation) model self-provision fails — named cache volume root-owned, worker runs as app (#527 fallout) #703

New Issue

2026-07-18T22:46:29Z

spikerj commented

2026-07-18 22:46:29 +00:00

Symptom (2026-07-18 ~22:42Z): embeddings failed for book 6a5ad504 with FileNotFoundException: Embedding model not found locally or in the ai-models bucket: /app/ai/nomic-embed-text-v2-moe.f16.gguf — while the model IS visible in the ai-models bucket.

Root cause — not S3. The worker logs show the bucket lookup succeeded ("Provisioning ai-models/nomic-embed-text-v2-moe.f16.gguf (913 MB) into /tmp/spikersoft-ai-models/...") and the DOWNLOAD then died with System.UnauthorizedAccessException: Access to the path '/tmp/spikersoft-ai-models/....tmp-...' is denied. ObjectFileCache.ResolveAsync degrades every failure to null (by design), so the caller throws the misleading "not found" message; the real exception is only in the preceding WRN line.

Why the write fails: the #527 stack change mounts a named volume at /tmp/spikersoft-ai-models (embeddings-models), the Dockerfile runs the service as USER app (uid 999), and a named volume first-mounted over a path that does not exist in the image is initialized root-owned 0755. Verified on the 4090: spikersoft-quiz-generation_quiz-generation-models is drwxr-xr-x 0:0 — quiz-generation has the identical latent bug (same volume pattern, same USER app) and will fail its next model self-provision the same way.

Impact: every embeddings request fails after retries → embeddings.process DLQ (book 6a5ad504's request dead-lettered ~22:47Z, needs replay after the fix); quiz generation rides embeddings so quizzes stall too. The math-book upload (6a5bf73f) will hit this as soon as its image descriptions finish.

Fixes:

Ops (immediate): chown the two existing volumes to 999:999 (docker run --rm -v spikersoft-quiz-generation_quiz-generation-models:/m alpine chown 999:999 /m on 4090; one-shot SERVER-pinned service for spikersoft-embeddings_embeddings-models).
Durable: Dockerfiles pre-create /tmp/spikersoft-ai-models chowned to app BEFORE USER app, so future volume initializations inherit app ownership (PR incoming).
Optional hardening: ObjectFileCache could surface non-transient IO failures distinctly instead of the misleading "not found" (the WRN does carry the real exception).
After fix: replay embeddings.process.dlq (also holds 2 older messages from today's 07:02Z incident).

**Symptom (2026-07-18 ~22:42Z):** embeddings failed for book 6a5ad504 with `FileNotFoundException: Embedding model not found locally or in the ai-models bucket: /app/ai/nomic-embed-text-v2-moe.f16.gguf` — while the model IS visible in the ai-models bucket. **Root cause — not S3.** The worker logs show the bucket lookup succeeded ("Provisioning ai-models/nomic-embed-text-v2-moe.f16.gguf (913 MB) into /tmp/spikersoft-ai-models/...") and the DOWNLOAD then died with `System.UnauthorizedAccessException: Access to the path '/tmp/spikersoft-ai-models/....tmp-...' is denied`. `ObjectFileCache.ResolveAsync` degrades every failure to null (by design), so the caller throws the misleading "not found" message; the real exception is only in the preceding WRN line. Why the write fails: the #527 stack change mounts a **named volume** at `/tmp/spikersoft-ai-models` (`embeddings-models`), the Dockerfile runs the service as `USER app` (uid 999), and a named volume first-mounted over a path that does not exist in the image is initialized **root-owned 0755**. Verified on the 4090: `spikersoft-quiz-generation_quiz-generation-models` is `drwxr-xr-x 0:0` — **quiz-generation has the identical latent bug** (same volume pattern, same USER app) and will fail its next model self-provision the same way. **Impact:** every embeddings request fails after retries → embeddings.process DLQ (book 6a5ad504's request dead-lettered ~22:47Z, needs replay after the fix); quiz generation rides embeddings so quizzes stall too. The math-book upload (6a5bf73f) will hit this as soon as its image descriptions finish. **Fixes:** 1. Ops (immediate): chown the two existing volumes to 999:999 (`docker run --rm -v spikersoft-quiz-generation_quiz-generation-models:/m alpine chown 999:999 /m` on 4090; one-shot SERVER-pinned service for `spikersoft-embeddings_embeddings-models`). 2. Durable: Dockerfiles pre-create `/tmp/spikersoft-ai-models` chowned to app BEFORE `USER app`, so future volume initializations inherit app ownership (PR incoming). 3. Optional hardening: ObjectFileCache could surface non-transient IO failures distinctly instead of the misleading "not found" (the WRN does carry the real exception). 4. After fix: replay embeddings.process.dlq (also holds 2 older messages from today's 07:02Z incident).

spikerj closed this issue

2026-07-18 22:51:05 +00:00

spikerj commented

2026-07-18 22:57:49 +00:00

Fix verified live end-to-end (2026-07-18 ~22:57Z): PR #421 merged + deployed, both prod volumes chowned to 999:999 (quiz-generation on 4090 directly; embeddings via one-shot SERVER-pinned job). The new embeddings task provisioned ai-models/nomic-embed-text-v2-moe.f16.gguf (913 MB) into the volume without error and is generating embeddings for book 6a5ad504 (342 pages, ~100-400ms/page) — the exact request that was failing. The book's last scheduled retry was still in flight so no DLQ replay was needed for it; embeddings.process.dlq still holds the 2 older messages from the 07:02Z broker incident (pre-dating this bug) — replay or purge at leisure. Closing.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#703