spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][Architecture] Provider-agnostic object storage (MinIO -> S3/Azure Blob) to replace GlusterFS/host-path model #413

New Issue
Open
opened 2026-07-05 20:24:40 +00:00 by spikerj · 6 comments
spikerj commented 2026-07-05 20:24:40 +00:00
Owner
Copy Link
Copy Source

Context: The genuine structural limitation behind the host-locked Swarm and several path-based file findings.

Problem: Shared media/files live on a GlusterFS volume spanning DreamStream1–DreamStream7 (maintained by DreamStream4), while the SERVER and laptop-server hosts each only see their own local filesystem. Storage is tied to where a container is scheduled, so Swarm placement must be host-locked, and file handlers assume shared host paths (/app/ebooks/books/…, quarantine and upload-staging directories).

Fix: Introduce a provider-agnostic object-storage abstraction behind a single storage interface — self-hosted MinIO now (S3-compatible), swappable for AWS S3 / Azure Blob / GCS in a datacenter or cloud — so the app addresses content by bucket + key, never by host path. This decouples storage from node placement, frees Swarm scheduling, and turns the filesystem-path handlers (and their traversal risks) into safe, portable object operations.

Acceptance criteria:

  • A storage interface with a MinIO implementation; no direct host-path file I/O in app services
  • The same code runs on-prem (MinIO) and against a cloud blob store via config
  • Swarm placement no longer pinned by storage locality

Effort: L · Prerequisite for INF-2 (this batch). Unblocks the path-based handlers referenced in CR-4 / CR-6.

**Context:** The genuine structural limitation behind the host-locked Swarm and several path-based file findings. **Problem:** Shared media/files live on a GlusterFS volume spanning `DreamStream1`–`DreamStream7` (maintained by DreamStream4), while the `SERVER` and `laptop-server` hosts each only see their own local filesystem. Storage is tied to *where a container is scheduled*, so Swarm placement must be host-locked, and file handlers assume shared host paths (`/app/ebooks/books/…`, quarantine and upload-staging directories). **Fix:** Introduce a provider-agnostic object-storage abstraction behind a single storage interface — self-hosted **MinIO** now (S3-compatible), swappable for **AWS S3 / Azure Blob / GCS** in a datacenter or cloud — so the app addresses content by bucket + key, never by host path. This decouples storage from node placement, frees Swarm scheduling, and turns the filesystem-path handlers (and their traversal risks) into safe, portable object operations. **Acceptance criteria:** - A storage interface with a MinIO implementation; no direct host-path file I/O in app services - The same code runs on-prem (MinIO) and against a cloud blob store via config - Swarm placement no longer pinned by storage locality **Effort:** L · Prerequisite for INF-2 (this batch). Unblocks the path-based handlers referenced in CR-4 / CR-6.
spikerj added the agentic label 2026-07-05 20:24:40 +00:00
spikerj commented 2026-07-05 20:25:09 +00:00
Author
Owner
Copy Link
Copy Source

Prerequisite for the production deployment model INF-2 #414.

Prerequisite for the production deployment model **INF-2 #414**.
spikerj commented 2026-07-12 01:32:51 +00:00
Author
Owner
Copy Link
Copy Source

Grounding notes from a live look at the swarm + docs/minio-storage-migration.md (prompted by the artpipe-gpu label incident that pinned concept-art to SERVER):

  1. MinIO is not actually running. The doc's status line says 'MinIO deployed', but there is no minio service in the swarm and minio.spikersoft.com is not a Traefik router. Deploying the committed minio/docker-stack.yml (needs MINIO_ROOT_USER/PASSWORD exported) is prerequisite step 0. Filed as the first concrete step -> see the new pilot ticket.

  2. art_pipe is a poor FIRST candidate (the doc already flags it as 'deliberately not bucket material'): /mnt/fusionio/spikersoft/art_pipe is a live checkout + Python venvs the six artpipe stacks exec into — you can't exec a Python interpreter out of S3. Decoupling art_pipe from SERVER is really two separate jobs: (a) bake venvs/CUDA into the container image (or a per-node cache), and (b) sync model weights from the ai-models bucket at startup. That's a bigger, art_pipe-specific workstream — the reward for the pattern, not the pilot.

  3. Recommended pilot = security-scanner (single ro uploads bind, smallest blast radius), then metadata-extractor. Filed a scoped pilot ticket with the deploy-MinIO prerequisite + the full per-service recipe.

Motivation is now concrete: the artpipe-gpu label could not follow off SERVER because every artpipe stack bind-mounts /mnt/fusionio/spikersoft/art_pipe, which only exists on SERVER — exactly the host-path pin this epic exists to remove.

Grounding notes from a live look at the swarm + docs/minio-storage-migration.md (prompted by the artpipe-gpu label incident that pinned concept-art to SERVER): 1. **MinIO is not actually running.** The doc's status line says 'MinIO deployed', but there is no minio service in the swarm and minio.spikersoft.com is not a Traefik router. Deploying the committed minio/docker-stack.yml (needs MINIO_ROOT_USER/PASSWORD exported) is prerequisite step 0. Filed as the first concrete step -> see the new pilot ticket. 2. **art_pipe is a poor FIRST candidate** (the doc already flags it as 'deliberately not bucket material'): /mnt/fusionio/spikersoft/art_pipe is a live checkout + Python venvs the six artpipe stacks exec into — you can't exec a Python interpreter out of S3. Decoupling art_pipe from SERVER is really two separate jobs: (a) bake venvs/CUDA into the container image (or a per-node cache), and (b) sync model *weights* from the ai-models bucket at startup. That's a bigger, art_pipe-specific workstream — the reward for the pattern, not the pilot. 3. **Recommended pilot = security-scanner** (single ro uploads bind, smallest blast radius), then metadata-extractor. Filed a scoped pilot ticket with the deploy-MinIO prerequisite + the full per-service recipe. Motivation is now concrete: the artpipe-gpu label could not follow off SERVER because every artpipe stack bind-mounts /mnt/fusionio/spikersoft/art_pipe, which only exists on SERVER — exactly the host-path pin this epic exists to remove.
spikerj commented 2026-07-13 00:41:02 +00:00
Author
Owner
Copy Link
Copy Source

Child tickets filed for the remaining per-service migrations (2026-07-12), sequenced per docs/minio-storage-migration.md — standalone dirs first, uploads tree as one batch:

  • #522 blogs · #523 lesson-videos · #524 dlls · #525 assets · #526 quarantine · #527 ai→ai-models
  • #528 API static-file media serving → S3 delivery
  • #529 uploads-tree coordinated batch cutover (after the standalones + #493)
  • #530 GeoLite2 → reference-data bucket (also the GeoIP half of #489)
  • #531 Angular nginx root bind (last) · #532 sftp via MinIO SFTP mode (optional)
  • #533 shared IObjectStore refactor (the storage-interface acceptance criterion of this epic)

Related but separate workstream: #515 (art_pipe per-model baked images — removes the /opt/art_pipe bind the migration doc scoped out) and #521 (Gitea upstream-image mirror).

Child tickets filed for the remaining per-service migrations (2026-07-12), sequenced per docs/minio-storage-migration.md — standalone dirs first, uploads tree as one batch: - #522 blogs · #523 lesson-videos · #524 dlls · #525 assets · #526 quarantine · #527 ai→ai-models - #528 API static-file media serving → S3 delivery - #529 uploads-tree coordinated batch cutover (after the standalones + #493) - #530 GeoLite2 → reference-data bucket (also the GeoIP half of #489) - #531 Angular nginx root bind (last) · #532 sftp via MinIO SFTP mode (optional) - #533 shared IObjectStore refactor (the storage-interface acceptance criterion of this epic) Related but separate workstream: #515 (art_pipe per-model baked images — removes the /opt/art_pipe bind the migration doc scoped out) and #521 (Gitea upstream-image mirror).
spikerj commented 2026-07-13 20:33:04 +00:00
Author
Owner
Copy Link
Copy Source

EPIC SCOREBOARD (2026-07-13, after 19 loop iterations): every child ticket is code-complete. MERGED: #533 seam (closed), #530 GeoIP, #540 scheduler, #526 quarantine, #524 dlls, #525 assets, #522 blogs, #523 lesson-videos, #527 ai-models, #528 API media serving, infra #56-#59. GATED PRs open: backend #264 + infra #60 (uploads-batch dual-run prep + cutover runbook), angular #177 (SECURITY #542 — merge first) + #178 (#531 nginx proxy, the last bind). Critical path is now pure ops: runbook phase 0 (buckets/keys/secrets/backfills) → phase 1 soak → phase 2 cutover PR → angular #178 + bind drop. After that the swarm's app tier has zero /mnt/fusionio dependencies.

EPIC SCOREBOARD (2026-07-13, after 19 loop iterations): every child ticket is code-complete. MERGED: #533 seam (closed), #530 GeoIP, #540 scheduler, #526 quarantine, #524 dlls, #525 assets, #522 blogs, #523 lesson-videos, #527 ai-models, #528 API media serving, infra #56-#59. GATED PRs open: backend #264 + infra #60 (uploads-batch dual-run prep + cutover runbook), angular #177 (SECURITY #542 — merge first) + #178 (#531 nginx proxy, the last bind). Critical path is now pure ops: runbook phase 0 (buckets/keys/secrets/backfills) → phase 1 soak → phase 2 cutover PR → angular #178 + bind drop. After that the swarm's app tier has zero /mnt/fusionio dependencies.
spikerj commented 2026-07-18 18:02:04 +00:00
Author
Owner
Copy Link
Copy Source

EPIC SCOREBOARD — 2026-07-18 accuracy pass (supersedes the 2026-07-13 one)

Every claim below re-verified against current masters (backend/infra/angular) and the live ticket states, not carried forward.

Acceptance criteria

  • ✅ Storage interface with MinIO impl, no direct host-path I/O in the seam — SpikerSoft.Storage (IObjectStore/S3ObjectStore + keyed stores + per-op timeouts), all services consume it (#533 closed)
  • ✅ Same code runs on-prem/cloud via config (Storage__* env, path-style)
  • ❌ Swarm placement no longer pinned by storage — NOT met yet: every stack still carries its /mnt/fusionio binds + node.hostname == SERVER. This is the remaining work.

Children — CLOSED (code merged + buckets provisioned/backfilled 07-14)

#493 pilot · #498 gitea-packages · #522 blogs · #523 lesson-videos · #524 dlls · #525 assets · #526 quarantine · #527 ai-models · #528 API media serving · #533 seam · plus fixed enablers #581, #598, #599, #542.

Still OPEN — the real remaining work

Ticket What actually remains
#529 THE ops cutover (consolidated checklist posted there): key policies (#613, blogs-svc write, uploads-svc read-ebooks), lesson-video-processor deploy check, soak, then the phase-2 bind-drop + placement-relax PR
#696 ebooks slice → sole store (timeout half landed; internal MinIO route, write-primary, backfill-verify, API serving cutover, retire mirror seam)
#531 Angular nginx root bind (LAST) — rescoped: proxy blogs+lesson-videos only; ebooks stays API/BookAccess-served (never anonymous)
#530 GeoIP — code done; gated on #540 (Scheduler deployed nowhere)
#532 sftp retire/replace — one ls on SERVER decides it
Related #613 (policy fix, gates #529), #700 (MinIO/Gitea SPOF), #538 (LAN hairpin ≈ #696 P0 internal route), #521 (registry mirror)

Docs

The misleading "zero services migrated" status is fixed in infra PR #125 (migration doc + infrastructure-reality rule mirrors).

Net: the epic's code phase is done and proven; what stands between here and closing #413 is ops: key policies → soak → bind drops (#529), the ebooks sole-store phases (#696), and the final nginx bind (#531).

## EPIC SCOREBOARD — 2026-07-18 accuracy pass (supersedes the 2026-07-13 one) Every claim below re-verified against current masters (backend/infra/angular) and the live ticket states, not carried forward. ### Acceptance criteria - ✅ **Storage interface with MinIO impl, no direct host-path I/O in the seam** — `SpikerSoft.Storage` (`IObjectStore`/`S3ObjectStore` + keyed stores + per-op timeouts), all services consume it (#533 closed) - ✅ Same code runs on-prem/cloud via config (`Storage__*` env, path-style) - ❌ **Swarm placement no longer pinned by storage** — NOT met yet: every stack still carries its `/mnt/fusionio` binds + `node.hostname == SERVER`. This is the remaining work. ### Children — CLOSED (code merged + buckets provisioned/backfilled 07-14) #493 pilot · #498 gitea-packages · #522 blogs · #523 lesson-videos · #524 dlls · #525 assets · #526 quarantine · #527 ai-models · #528 API media serving · #533 seam · plus fixed enablers #581, #598, #599, #542. ### Still OPEN — the real remaining work | Ticket | What actually remains | |---|---| | **#529** | THE ops cutover (consolidated checklist posted there): key policies (#613, blogs-svc write, uploads-svc read-ebooks), lesson-video-processor deploy check, soak, then the phase-2 bind-drop + placement-relax PR | | **#696** | ebooks slice → sole store (timeout half landed; internal MinIO route, write-primary, backfill-verify, API serving cutover, retire mirror seam) | | **#531** | Angular nginx root bind (LAST) — **rescoped**: proxy blogs+lesson-videos only; ebooks stays API/BookAccess-served (never anonymous) | | **#530** | GeoIP — code done; gated on #540 (Scheduler deployed nowhere) | | **#532** | sftp retire/replace — one `ls` on SERVER decides it | | Related | #613 (policy fix, gates #529), #700 (MinIO/Gitea SPOF), #538 (LAN hairpin ≈ #696 P0 internal route), #521 (registry mirror) | ### Docs The misleading "zero services migrated" status is fixed in **infra PR #125** (migration doc + infrastructure-reality rule mirrors). **Net:** the epic's code phase is done and proven; what stands between here and closing #413 is ops: key policies → soak → bind drops (#529), the ebooks sole-store phases (#696), and the final nginx bind (#531).
spikerj commented 2026-07-29 17:14:14 +00:00
Author
Owner
Copy Link
Copy Source

Epic status roll-up — I've now audited most children against origin/master this week. Consolidating so the remaining work is visible in one place.

Child State
#783 art-asset-artifacts → MinIO ⚠️ code + migration done and recorded (74/74, ~4.5 GB, SHA-256 read-back, 0 failures); 3 operator steps outstanding
#530 GeoLite2 → reference-data ⚠️ code complete, #540 blocker now cleared; no GeoIpUpdate task has ever been created
#529 uploads-tree 7-service cutover ❌ phase 2 not started on any service
#696 ebooks sole-store ❌ dual-write seam intact; backfill unverified
#531 nginx root bind (LAST) ❌ not started — see below
#532 sftp → MinIO SFTP mode (optional) ❌ not started — sftp/docker-swarm.yml still present, no SFTP flag on the minio stack
#613 movement-svc key ⚠️ one command from done

#531 confirmed not started: spikersoft-angular/docker-stack.yml:10 still binds /mnt/fusionio/spikersoft:/usr/share/nginx/html/spikersoft. Correctly sequenced last, as the ticket says.

The single highest-leverage item is #613. All its code and provisioning are merged — the stack uses movement-svc (spikersoft-file-movement/docker-stack.yml:32-33), openbao/provision-minio-svc-users.sh:20 grants the right policy, and both rotation scripts honour ONLY_SVCS. What's missing is one command: ONLY_SVCS="movement" rotate-minio-ci-keys.sh with an admin-capable Bao token. That unblocks #529's phase-0 items and #783's remaining follow-up.

#529 is where the epic's remaining bulk sits, and it's genuinely at zero on phase 2. Storage__UseS3=true is set on all seven services, so dual-run works — but every one still binds /mnt/fusionio and is still pinned node.hostname == SERVER. So the payoff this epic exists for (scheduling off SERVER) isn't available on any service yet. Per-service line numbers are on that ticket.

Two cross-cutting cautions for the remaining cutovers:

  1. #825's prevention items were never written down. Its four asks — writer-first/reader-last ordering, backfill after all writers flip, a post-cutover verification sweep — are absent from docs/uploads-batch-cutover-runbook.md and docs/minio-storage-migration.md. That ordering rule has already cost two production incidents (#825 itself, then #856 two days later, same shape). #529 and #696 both have live backfill steps, so the runbook would currently let an operator repeat it.

  2. #615/#774 mean deploys can't be trusted to have applied. Root has no git credentials so sudo git -C /mnt/infrastructure pull fails and is swallowed by || echo "WARNING", and nothing polls for convergence afterwards. For an epic whose remaining work is almost entirely stack-file edits, that's the difference between "deployed" and "reported deployed". #820 is a live instance: an infra-only env change never reaches the artpipe workers because their deploy job only triggers on backend pushes.

Also worth confirming before this epic closes: #783's third operator step — the collection sweep proving Mongo GridFS is actually empty. This epic's headline claim rests on it, and nothing has recorded it running.

**Epic status roll-up** — I've now audited most children against `origin/master` this week. Consolidating so the remaining work is visible in one place. | Child | State | |---|---| | #783 art-asset-artifacts → MinIO | ⚠️ code + migration **done and recorded** (74/74, ~4.5 GB, SHA-256 read-back, 0 failures); 3 operator steps outstanding | | #530 GeoLite2 → reference-data | ⚠️ code complete, **#540 blocker now cleared**; no GeoIpUpdate task has ever been created | | #529 uploads-tree 7-service cutover | ❌ **phase 2 not started on any service** | | #696 ebooks sole-store | ❌ dual-write seam intact; backfill unverified | | #531 nginx root bind (LAST) | ❌ not started — see below | | #532 sftp → MinIO SFTP mode (optional) | ❌ not started — `sftp/docker-swarm.yml` still present, no SFTP flag on the minio stack | | #613 movement-svc key | ⚠️ **one command from done** | **#531 confirmed not started:** `spikersoft-angular/docker-stack.yml:10` still binds `/mnt/fusionio/spikersoft:/usr/share/nginx/html/spikersoft`. Correctly sequenced last, as the ticket says. **The single highest-leverage item is #613.** All its code and provisioning are merged — the stack uses `movement-svc` (`spikersoft-file-movement/docker-stack.yml:32-33`), `openbao/provision-minio-svc-users.sh:20` grants the right policy, and both rotation scripts honour `ONLY_SVCS`. What's missing is one command: `ONLY_SVCS="movement" rotate-minio-ci-keys.sh` with an admin-capable Bao token. That unblocks #529's phase-0 items and #783's remaining follow-up. **#529 is where the epic's remaining bulk sits**, and it's genuinely at zero on phase 2. `Storage__UseS3=true` is set on all seven services, so dual-run works — but **every one still binds `/mnt/fusionio` and is still pinned `node.hostname == SERVER`**. So the payoff this epic exists for (scheduling off SERVER) isn't available on any service yet. Per-service line numbers are on that ticket. **Two cross-cutting cautions for the remaining cutovers:** 1. **#825's prevention items were never written down.** Its four asks — writer-first/reader-last ordering, backfill after all writers flip, a post-cutover verification sweep — are absent from `docs/uploads-batch-cutover-runbook.md` and `docs/minio-storage-migration.md`. That ordering rule has already cost two production incidents (#825 itself, then #856 two days later, same shape). #529 and #696 both have live backfill steps, so the runbook would currently let an operator repeat it. 2. **#615/#774 mean deploys can't be trusted to have applied.** Root has no git credentials so `sudo git -C /mnt/infrastructure pull` fails and is swallowed by `|| echo "WARNING"`, and nothing polls for convergence afterwards. For an epic whose remaining work is almost entirely stack-file edits, that's the difference between "deployed" and "reported deployed". #820 is a live instance: an infra-only env change never reaches the artpipe workers because their deploy job only triggers on backend pushes. **Also worth confirming before this epic closes:** #783's third operator step — the collection sweep proving Mongo GridFS is actually empty. This epic's headline claim rests on it, and nothing has recorded it running.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels agentic
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#413