spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][MinIO] Migrate /mnt/fusionio/spikersoft/ai → ai-models bucket for embeddings + quiz-generation (epic #413) #527

New Issue
Closed
opened 2026-07-13 00:40:41 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-13 00:40:41 +00:00
Owner
Copy Link
Copy Source

Parent epic: #413.

Scope: ai-models bucket — /mnt/fusionio/spikersoft/ai (backend rw; embeddings + quiz-generation ro).

Current state: embeddings and quiz-generation read model files straight off the ro bind; both are pinned to SERVER solely because of it (spikersoft-embeddings/docker-stack*.yml, spikersoft-quiz-generation/docker-stack*.yml).

Steps: bucket + scoped ro keys per service → backend PR: model loader fetches from ai-models into a container-local cache dir at startup (models are read-once-at-load; an ephemeral/named local cache is fine — the decoupling goal is placement, not zero local disk for caches) → backfill mc mirror /mnt/fusionio/spikersoft/ai spiker/ai-models → dual-run → remove ro binds, relax placement to the GPU label.

Important caveat (from docs/minio-storage-migration.md non-goals): migrating ai-models is necessary but NOT sufficient to run embeddings/quiz on the 4090 — the gpu-coordinator still needs per-node lease routing (#367 U2). Placement relaxation lands together with that work.

Parent epic: #413. **Scope:** `ai-models` bucket — `/mnt/fusionio/spikersoft/ai` (backend rw; embeddings + quiz-generation ro). **Current state:** embeddings and quiz-generation read model files straight off the ro bind; both are pinned to SERVER solely because of it (`spikersoft-embeddings/docker-stack*.yml`, `spikersoft-quiz-generation/docker-stack*.yml`). **Steps:** bucket + scoped ro keys per service → backend PR: model loader fetches from `ai-models` into a container-local cache dir at startup (models are read-once-at-load; an ephemeral/named local cache is fine — the decoupling goal is *placement*, not zero local disk for caches) → backfill `mc mirror /mnt/fusionio/spikersoft/ai spiker/ai-models` → dual-run → remove ro binds, relax placement to the GPU label. **Important caveat (from `docs/minio-storage-migration.md` non-goals):** migrating `ai-models` is necessary but NOT sufficient to run embeddings/quiz on the 4090 — the gpu-coordinator still needs per-node lease routing (#367 U2). Placement relaxation lands together with that work.
spikerj commented 2026-07-13 20:09:31 +00:00
Author
Owner
Copy Link
Copy Source

Backend PR #261: ObjectFileCache in SpikerSoft.Storage (local-wins → atomic bucket download → null), embeddings resolves its gguf directly, quiz-generation via a ModelPathResolver delegate so Business.Ai.Workers stays storage-free. Placement relaxation deliberately NOT included — waits on #367 U2 per the migration-doc non-goal. Deploy: ai-models bucket + ro keys, mc mirror backfill, optionally a node-local named volume for the model cache.

Backend PR #261: ObjectFileCache in SpikerSoft.Storage (local-wins → atomic bucket download → null), embeddings resolves its gguf directly, quiz-generation via a ModelPathResolver delegate so Business.Ai.Workers stays storage-free. Placement relaxation deliberately NOT included — waits on #367 U2 per the migration-doc non-goal. Deploy: ai-models bucket + ro keys, mc mirror backfill, optionally a node-local named volume for the model cache.
spikerj commented 2026-07-13 20:33:03 +00:00
Author
Owner
Copy Link
Copy Source

Code MERGED (backend #261 — ObjectFileCache + embeddings direct + quiz via ModelPathResolver delegate). Remaining = ops: ai-models bucket + per-service ro keys, mc mirror /mnt/fusionio/spikersoft/ai (multi-GB), Storage__UseS3 + secrets on both GPU stacks, optional node-local named volume for the model cache. Placement relaxation stays gated on #367 U2 per the ticket.

Code MERGED (backend #261 — ObjectFileCache + embeddings direct + quiz via ModelPathResolver delegate). Remaining = ops: ai-models bucket + per-service ro keys, mc mirror /mnt/fusionio/spikersoft/ai (multi-GB), Storage__UseS3 + secrets on both GPU stacks, optional node-local named volume for the model cache. Placement relaxation stays gated on #367 U2 per the ticket.
spikerj commented 2026-07-17 02:07:00 +00:00
Author
Owner
Copy Link
Copy Source

Complete and verified live (2026-07-17):

  • Code: backend #261 (ObjectFileCache + embeddings direct + quiz ModelPathResolver) — merged earlier.
  • Infra: #120 (embeddings sources the model from the ai-models bucket, /app/ai bind dropped, named cache volume) — merged and deployed; quiz-generation stack likewise runs Storage__UseS3=true.
  • Live state checked on the swarm: both services Running; embeddings has NO fusionio bind (only the named model-cache volume) and resolves its model path; quiz-generation has a working Bao-sourced storage key and generated a quiz at 2026-07-16T15:03Z (17 in book-generated-quizzes), proving the bucket holds the weights and the on-demand fetch works.

Placement relaxation (un-pinning from SERVER/GPU node) stays gated on #367 U2 per the ticket scope. Closing; #592 (quiz-generation empty secret) closed with the same evidence.

Complete and verified live (2026-07-17): - Code: backend #261 (ObjectFileCache + embeddings direct + quiz ModelPathResolver) — merged earlier. - Infra: #120 (embeddings sources the model from the `ai-models` bucket, `/app/ai` bind dropped, named cache volume) — merged and deployed; quiz-generation stack likewise runs `Storage__UseS3=true`. - Live state checked on the swarm: both services Running; embeddings has NO fusionio bind (only the named model-cache volume) and resolves its model path; quiz-generation has a working Bao-sourced storage key and **generated a quiz at 2026-07-16T15:03Z** (17 in `book-generated-quizzes`), proving the bucket holds the weights and the on-demand fetch works. Placement relaxation (un-pinning from SERVER/GPU node) stays gated on #367 U2 per the ticket scope. Closing; #592 (quiz-generation empty secret) closed with the same evidence.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#527