[Infra][MinIO] Uploads-tree coordinated batch cutover — 7 services off /mnt/fusionio (epic #413) #529
Assignees
spikerj (Joseph Spiker)
Clear assignees
spikerj (Joseph Spiker)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: spikerj/spikersoft-issues#529
Reference in New Issue
Block a user
Parent epic: #413. The coordinated batch — per the ordering constraint in
docs/minio-storage-migration.md, the shareduploadstree hands files between services by path, so these cut over together after the standalone dirs prove the pattern:backend API (staging writers:
FileProcessingService,LessonVideoStagingService,UploadBookCommandHandler,CacheExternalUrlCommandHandler), upload-coordinator (StagingMirror — dual-write EXISTS), security-scanner (S3StagedObjectStore dual-read EXISTS, #493), metadata-extractor (dual-read EXISTS), file-movement (mirror EXISTS), blog-media-processor + lesson-video-processor (S3 code from their standalone tickets).Cutover plan: flip
Storage:UseS3=trueeverywhere in one deploy window → finalmc mirrorre-run → watch Seq/Jaeger → remove ALL/mnt/fusionio/spikersoft/uploads*binds from the seven stack files → relax placements tonode.labels.spikersoft-app == true→ verify tasks can land on 4090 (burst runbook) → retire the host path after a soak.Blocked by: blogs, lesson-videos, dlls, quarantine standalone tickets + #493 completion.
Cutover prep landed as two PRs: backend #264 (deploy-secret guards for blog-media/lesson-video — the only batch members missing them) + infra #60 GATED (Storage__ env for those two stacks + docs/uploads-batch-cutover-runbook.md with the full phase 0-2 choreography, verification and rollback). All seven services' S3 code paths are now merged or in review; the remaining #529 work is the runbook's ops phases + a phase-2 bind-removal PR after the soak. Note from the audit: file-movement reuses metadata-svc — widen that policy to its four new buckets or issue a dedicated movement-svc key.
Epic #413 accuracy pass (2026-07-18) — this is now the SINGLE open ops ticket for the batch cutover. Consolidated state + remaining checklist.
Per-service tickets #493/#522/#523/#524/#525/#526/#528 are closed into here: every code path is merged on backend master, every one of the 10 stacks dual-runs (
Storage__UseS3=true, verified in the infra stack files), and phase 0 ran 2026-07-14 (buckets created; ebooks 1.6 GiB ✓, blogs ✓; the empty buckets match empty disk trees). Docs corrected in infra PR #125.Remaining checklist (in order)
Phase 0 stragglers — key policies (a 403 is as silent as the missing buckets were):
metadata-svckey needs rw on quarantine/dlls/assets/blogs (or a dedicatedmovement-svc); Access Denied proven in Seq 2026-07-16blogs-svc: confirm write onblogs(from #522)uploads-svc: add read onebooks(forBookImageLocator, #598 — from #528)docker service ls) — first-ever deploy unblocked 2026-07-15Phase 1 — soak:
Phase 2 — the cutover PR:
mc mirrorre-run → remove all/mnt/fusioniobinds from the 7 stacks → relaxnode.hostname == SERVER→node.labels.spikersoft-app == true→ verify a task lands on 4090Scope note — ebooks slice now rides #696
#696 (2026-07-18) supersedes the dual-run model for ebooks specifically: sole-store, write-primary with real error handling, internal MinIO route, timeouts (partially landed — backend #391). The ebooks bind-drop choreography should follow #696's phases rather than this runbook's generic phase 2; this ticket keeps the other six trees. Also note nginx no longer serves
/spikersoft/ebooksat all (denied since #618 — book serving is authz'd through the API), which simplifies this batch's serving story to blogs + lesson-videos (#531).Checklist progress (2026-07-18): the entire "phase 0 stragglers — key policies" block now has a single executable fix: infra PR #126 + backend PR #415 (see #613 for the sequence). One script run then covers:
movement-svc(file-movement quarantine/dlls/assets/blogs writes)uploads-svc+ebooks-ro(the #598 BookImageLocator gate)blogs-svcwrite confirmation — the script's new closing policy audit prints accessKey → attached policies for all svc users, which settles the #522 question with evidence instead of assumptionAfter the script run + backend #415 merge, remaining before phase 2: lesson-video-processor deploy check (
docker service ls | grep lesson), the phase-1 soak verifications (blog upload → bucket; EICAR → quarantine), then the bind-drop PR. I'll prepare the phase-2 bind-drop PR (GATED) so it's ready to merge the moment the soak is clean.Phase-0 checklist update after the 2026-07-18 provision run + merges (infra #125/#126/#127, backend #415):
#613 movement-svc— created in MinIO withmovement-svc-storage; one recovery step left: the Bao write 403'd (low-privilege token) → runONLY_SVCS="movement" sudo -E bash rotate-minio-ci-keys.shwith an admin token, then a file-movement deploy + EICAR verify closes it (details on #613)blogs-svcwrite CONFIRMED — audit showsblogs-svc-storageattached (rw blogs+quarantine). The #522 question is settled with evidence.uploads-svcread-ebooks DONE — audit showsebooks-ro,uploads-rw.docker service ls | grep lesson; note infra #123 fixed its 63-char service-name issue on 07-17, so post-#581+#123 it should finally exist)Also now in play (merged #127): the
miniointernal overlay. ⚠️ Time-sensitive ops step: on a manager,docker network create -d overlay --attachable minio, then redeploy theminiostack (needsMINIO_ROOT_USER/PASSWORDexported) — before the next worker CI deploy, since all 9 worker stacks now reference the external network and will fail fast withnetwork minio not founduntil it exists. Loud, not silent, but blocks deploys.Then: phase-1 soak (blog upload → bucket; EICAR → quarantine) → phase-2 bind-drop PR (I'll have it prepared GATED).
Audited against
origin/master— phase 2 has not started on any of the 7 services. Per-service state, so the remaining work is concrete rather than "the cutover":Storage__UseS3=trueis set on all seven (backend:30, upload-coordinator:44, security-scanner:50, metadata-extractor:57, file-movement:26, blog-media-processor:25, lesson-video-processor:24), so dual-run is genuinely enabled. But every one still binds/mnt/fusionioand is still pinned tonode.hostname == SERVER::67-75(uploads, ebooks, blogs, lesson-videos, ai, photographs):91SERVER:90-96(7 binds):103SERVER:103uploads:ro:110SERVER:97uploads:ro,:103ebooks:rw:110SERVER:52-58(7 binds):64SERVER:51,52,55:61SERVER:52,53:59SERVERSo the payoff this ticket exists for — letting these services schedule off SERVER — is not available yet on any of them.
Remaining:
mc mirrorre-run before the switch.node.hostname == SERVERtonode.labels.spikersoft-app == true.Plus the phase-0/1 items from this ticket that are still outstanding and unrecorded anywhere in repo: the movement-svc Bao rotate (#613), the lesson-video-processor deploy check, and the EICAR + blog-upload soak.
The ebooks bind is deferred to #696 by this ticket's own scope note, so I haven't double-counted it — though note #696 is also still at phase 0 there, so ebooks blocks metadata-extractor's bind removal in practice.