spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Prod] /mnt/fusionio/spikersoft/ai missing embedding model — book pipeline dies at embeddings (nomic-embed-text-v2-moe.f16.gguf not found) #497

New Issue
Closed
opened 2026-07-12 04:24:46 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-12 04:24:46 +00:00
Owner
Copy Link
Copy Source

Symptom (prod, book upload 04:21 UTC)

Book upload processed through scan → metadata → book-create → file-move, then failed at embeddings:

Loading embedding model from /app/ai/nomic-embed-text-v2-moe.f16.gguf
[Error] Failed to load embedding model ... Could not find file
3/3 attempts, ErrorType: io → poison-message pattern detected → DLQ

gpu-coordinator granted the lease fine (1536 MB on SERVER) — the worker started, the model file is simply absent from /mnt/fusionio/spikersoft/ai (bound ro as /app/ai).

Same family as #486/#495

The GPU-host provisioning chain (out-of-space event, ai-tree re-own) evidently also lost the model weights. Same class: host state not reproducible.

Expected contents of /mnt/fusionio/spikersoft/ai (from configs)

File Consumer
nomic-embed-text-v2-moe.f16.gguf Embeddings worker (AI:EmbeddingModelPath) AND the API's RedisVectorSearchService — this one blocks the book pipeline
gemma-3n-E2B-it-Q8_0.gguf, gemma-3-12b-it-Q4_K_M.gguf, Phi-4-reasoning-plus-Q4_K_M.gguf QuizGeneration model list
Mistral-7B-Instruct-v0.3.Q4_K_M.gguf QuizGeneration legacy AI:ModelPath fallback
dolphin-2.2.1-mistral-7B-GGUF-f16.gguf BookAnalysisService (likely dormant)

Run ls -la /mnt/fusionio/spikersoft/ai/ to see what actually survives.

Fix (ops, SERVER)

sudo mkdir -p /mnt/fusionio/spikersoft/ai
# embeddings (the blocker):
sudo curl -L -o /mnt/fusionio/spikersoft/ai/nomic-embed-text-v2-moe.f16.gguf       https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe-GGUF/resolve/main/nomic-embed-text-v2-moe.f16.gguf
# then re-upload the failed book (the original request was DLQ'd as poison)

Quiz-gen models as needed per the appsettings model list.

Durable direction

This is exactly what the ai-models bucket in docs/minio-storage-migration.md is for — models in MinIO, workers pull/verify on boot (or an mc mirror sync), so a host rebuild can't silently lose them. Related: #413, #486, #495, #432.

Verify

Seq: Embedding model loaded instead of the io error; book upload completes end-to-end; GET /healthz Vector Search stays Healthy.

## Symptom (prod, book upload 04:21 UTC) Book upload processed through scan → metadata → book-create → file-move, then **failed at embeddings**: Loading embedding model from /app/ai/nomic-embed-text-v2-moe.f16.gguf [Error] Failed to load embedding model ... Could not find file 3/3 attempts, ErrorType: io → poison-message pattern detected → DLQ gpu-coordinator granted the lease fine (1536 MB on SERVER) — the worker started, the **model file is simply absent** from `/mnt/fusionio/spikersoft/ai` (bound ro as `/app/ai`). ## Same family as #486/#495 The GPU-host provisioning chain (out-of-space event, ai-tree re-own) evidently also lost the model weights. Same class: host state not reproducible. ## Expected contents of /mnt/fusionio/spikersoft/ai (from configs) | File | Consumer | |---|---| | `nomic-embed-text-v2-moe.f16.gguf` | **Embeddings worker** (AI:EmbeddingModelPath) AND the API's RedisVectorSearchService — this one blocks the book pipeline | | `gemma-3n-E2B-it-Q8_0.gguf`, `gemma-3-12b-it-Q4_K_M.gguf`, `Phi-4-reasoning-plus-Q4_K_M.gguf` | QuizGeneration model list | | `Mistral-7B-Instruct-v0.3.Q4_K_M.gguf` | QuizGeneration legacy AI:ModelPath fallback | | `dolphin-2.2.1-mistral-7B-GGUF-f16.gguf` | BookAnalysisService (likely dormant) | Run `ls -la /mnt/fusionio/spikersoft/ai/` to see what actually survives. ## Fix (ops, SERVER) sudo mkdir -p /mnt/fusionio/spikersoft/ai # embeddings (the blocker): sudo curl -L -o /mnt/fusionio/spikersoft/ai/nomic-embed-text-v2-moe.f16.gguf https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe-GGUF/resolve/main/nomic-embed-text-v2-moe.f16.gguf # then re-upload the failed book (the original request was DLQ'd as poison) Quiz-gen models as needed per the appsettings model list. ## Durable direction This is exactly what the `ai-models` bucket in docs/minio-storage-migration.md is for — models in MinIO, workers pull/verify on boot (or an `mc mirror` sync), so a host rebuild can't silently lose them. Related: #413, #486, #495, #432. ## Verify Seq: `Embedding model loaded` instead of the io error; book upload completes end-to-end; `GET /healthz` Vector Search stays Healthy.
spikerj commented 2026-07-12 04:29:44 +00:00
Author
Owner
Copy Link
Copy Source

Immediate fix applied: Joey downloaded nomic-embed-text-v2-moe.f16.gguf to /mnt/fusionio/spikersoft/ai on SERVER. Awaiting a fresh book upload to confirm embeddings loads it.

Direction confirmed by Joey: model storage moves to MinIO (ai-models bucket) as the durable fix. Sketch of the pattern (future work, laying the path now):

  1. ai-models bucket + scoped keys (rw for an ops/sync identity; ro for embeddings/quiz-gen/book-analysis workers).
  2. Workers get a boot-time pull-and-verify step behind the same Storage:UseS3 seam used by the scanner/extractor: on startup, check the local model path; if missing (or hash-mismatched vs the bucket's ETag/checksum), download from ai-models/<file> into a local cache dir, THEN load. Filesystem bind stays as warm cache — first boot on a fresh host self-heals instead of crash-looping.
  3. Seeding/rotation via mc mirror /mnt/fusionio/spikersoft/ai spiker/ai-models (one-time), new models uploaded to the bucket first.
  4. Payoff beyond resilience: the /app/ai ro bind stops being a SERVER pin for embeddings/quiz-gen — models fetch over the network, so those workers can follow the GPU (e.g. the 4090) subject only to the gpu-coordinator lease model, not data gravity. This incident (host lost the weights silently) is exactly the failure mode the bucket eliminates.

Keeping this ticket open for (a) the upload verification and (b) tracking the pull-and-verify implementation, unless we'd rather fold (b) into #413.

Immediate fix applied: Joey downloaded `nomic-embed-text-v2-moe.f16.gguf` to `/mnt/fusionio/spikersoft/ai` on SERVER. Awaiting a fresh book upload to confirm embeddings loads it. **Direction confirmed by Joey: model storage moves to MinIO (`ai-models` bucket) as the durable fix.** Sketch of the pattern (future work, laying the path now): 1. `ai-models` bucket + scoped keys (rw for an ops/sync identity; **ro** for embeddings/quiz-gen/book-analysis workers). 2. Workers get a boot-time **pull-and-verify** step behind the same `Storage:UseS3` seam used by the scanner/extractor: on startup, check the local model path; if missing (or hash-mismatched vs the bucket's ETag/checksum), download from `ai-models/<file>` into a local cache dir, THEN load. Filesystem bind stays as warm cache — first boot on a fresh host self-heals instead of crash-looping. 3. Seeding/rotation via `mc mirror /mnt/fusionio/spikersoft/ai spiker/ai-models` (one-time), new models uploaded to the bucket first. 4. Payoff beyond resilience: the `/app/ai` **ro bind stops being a SERVER pin** for embeddings/quiz-gen — models fetch over the network, so those workers can follow the GPU (e.g. the 4090) subject only to the gpu-coordinator lease model, not data gravity. This incident (host lost the weights silently) is exactly the failure mode the bucket eliminates. Keeping this ticket open for (a) the upload verification and (b) tracking the pull-and-verify implementation, unless we'd rather fold (b) into #413.
spikerj commented 2026-07-12 04:35:09 +00:00
Author
Owner
Copy Link
Copy Source

Correction to the model table + second casualty confirmed: after provisioning the nomic GGUF, the pipeline advanced past embeddings and died at quiz-generation — Could not find file '/app/ai/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf' (LLamaWeights.LoadFromFileAsync, QuizGenerationService.InitializeForPresetAsync).

Mistral-7B is not the legacy fallback — it's THE production quiz model. appsettings.Production.json (2026-07-11) deliberately pins ALL presets (Fast/Balanced/Quality) to it because the base gemma/Phi-4 presets need 9000–12000 MB VRAM, ungrantable on SERVER's 8 GB RTX 3070 Ti; Mistral Q4_K_M (~4.2 GB, VramMB 5000) fits. So the required-on-prod set is exactly:

  • nomic-embed-text-v2-moe.f16.gguf — ✅ re-provisioned by Joey
  • Mistral-7B-Instruct-v0.3.Q4_K_M.gguf — ⏳ downloading:
    sudo curl -L -o /mnt/fusionio/spikersoft/ai/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf https://huggingface.co/MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF/resolve/main/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf
  • gemma-3n / gemma-3-12b / Phi-4 — NOT needed on current hardware (only if quiz-gen ever moves to the 24 GB 4090)
  • dolphin (BookAnalysis) — dormant, skip

This double-discovery (each missing model only surfaces when the pipeline reaches its stage) is more evidence for the boot-time pull-and-verify from the ai-models bucket: a worker should fail fast at startup — or better, self-heal — not one book at a time.

Correction to the model table + second casualty confirmed: after provisioning the nomic GGUF, the pipeline advanced past embeddings and died at **quiz-generation** — `Could not find file '/app/ai/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf'` (LLamaWeights.LoadFromFileAsync, QuizGenerationService.InitializeForPresetAsync). **Mistral-7B is not the legacy fallback — it's THE production quiz model.** appsettings.Production.json (2026-07-11) deliberately pins ALL presets (Fast/Balanced/Quality) to it because the base gemma/Phi-4 presets need 9000–12000 MB VRAM, ungrantable on SERVER's 8 GB RTX 3070 Ti; Mistral Q4_K_M (~4.2 GB, VramMB 5000) fits. So the required-on-prod set is exactly: - `nomic-embed-text-v2-moe.f16.gguf` — ✅ re-provisioned by Joey - `Mistral-7B-Instruct-v0.3.Q4_K_M.gguf` — ⏳ downloading: `sudo curl -L -o /mnt/fusionio/spikersoft/ai/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf https://huggingface.co/MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF/resolve/main/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf` - gemma-3n / gemma-3-12b / Phi-4 — NOT needed on current hardware (only if quiz-gen ever moves to the 24 GB 4090) - dolphin (BookAnalysis) — dormant, skip This double-discovery (each missing model only surfaces when the pipeline reaches its stage) is more evidence for the boot-time pull-and-verify from the `ai-models` bucket: a worker should fail fast at startup — or better, self-heal — not one book at a time.
spikerj commented 2026-07-12 18:22:56 +00:00
Author
Owner
Copy Link
Copy Source

All models verified loading in production — closing. nomic: 'Embedding model loaded successfully with 999 GPU layers' (17:25:25). Mistral: quiz generation produced real questions ('Total questions: 2', 17:33:53) with no FileNotFoundException. Qwen3-VL: 110 completed captions via the ai-models bucket. Durable half also done: the ENTIRE model set is now bucket-resident (ai-models, 24 GiB — ggufs + qwen snapshot), image-description already boot-syncs from it, and the C# workers' boot-time pull is sketched here for when it's wanted. A fresh host can no longer silently lose the fleet's models.

All models verified loading in production — closing. nomic: 'Embedding model loaded successfully with 999 GPU layers' (17:25:25). Mistral: quiz generation produced real questions ('Total questions: 2', 17:33:53) with no FileNotFoundException. Qwen3-VL: 110 completed captions via the ai-models bucket. Durable half also done: the ENTIRE model set is now bucket-resident (`ai-models`, 24 GiB — ggufs + qwen snapshot), image-description already boot-syncs from it, and the C# workers' boot-time pull is sketched here for when it's wanted. A fresh host can no longer silently lose the fleet's models.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#497