spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][MinIO] Migrate blogs dir + BlogMediaProcessor to S3 (blogs bucket, epic #413) #522

New Issue
Closed
opened 2026-07-13 00:40:39 +00:00 by spikerj · 4 comments
spikerj commented 2026-07-13 00:40:39 +00:00
Owner
Copy Link
Copy Source

Parent epic: #413. Per-service MinIO migration following the recipe in spikersoft-infrastructure/docs/minio-storage-migration.md and the pattern proven by #493 / metadata-extractor.

Scope: the blogs bucket — /mnt/fusionio/spikersoft/blogs (+ the uploads/blogs staging prefix handled under the uploads batch ticket).

Current state: SpikerSoft.EventHandlers.BlogMediaProcessor is filesystem-only — no AWSSDK.S3, no Storage:UseS3 flag (unlike UploadCoordinator/FileMovement/MetadataExtractor/SecurityScanner which already dual-run). Key files: Services/BlogMediaOrchestrator.cs, BlogMediaMoveConsumer.cs, BlogMediaScanConsumer.cs. Writers of /app/blogs also include backend API, file-movement, upload-coordinator.

Steps (per the doc):

  1. blogs bucket + scoped access key (mc admin user add + policy; never root creds).
  2. Backend PR: add S3 mirror/read path behind Storage:UseS3 (mirror the EbookArtifactMirror pattern from MetadataExtractor).
  3. Backfill mc mirror /mnt/fusionio/spikersoft/blogs spiker/blogs.
  4. Dual-run a few days, watch Seq/Jaeger.
  5. Cut over: remove the bind from spikersoft-blog-media-processor/docker-stack.yml, relax placement to node.labels.spikersoft-app == true.

Depends on: nothing (standalone dir). Blocks: uploads-batch cutover ticket (the /app/blogs writers in that batch).

Parent epic: #413. Per-service MinIO migration following the recipe in `spikersoft-infrastructure/docs/minio-storage-migration.md` and the pattern proven by #493 / metadata-extractor. **Scope:** the `blogs` bucket — `/mnt/fusionio/spikersoft/blogs` (+ the `uploads/blogs` staging prefix handled under the uploads batch ticket). **Current state:** `SpikerSoft.EventHandlers.BlogMediaProcessor` is **filesystem-only** — no AWSSDK.S3, no `Storage:UseS3` flag (unlike UploadCoordinator/FileMovement/MetadataExtractor/SecurityScanner which already dual-run). Key files: `Services/BlogMediaOrchestrator.cs`, `BlogMediaMoveConsumer.cs`, `BlogMediaScanConsumer.cs`. Writers of `/app/blogs` also include backend API, file-movement, upload-coordinator. **Steps (per the doc):** 1. `blogs` bucket + scoped access key (`mc admin user add` + policy; never root creds). 2. Backend PR: add S3 mirror/read path behind `Storage:UseS3` (mirror the `EbookArtifactMirror` pattern from MetadataExtractor). 3. Backfill `mc mirror /mnt/fusionio/spikersoft/blogs spiker/blogs`. 4. Dual-run a few days, watch Seq/Jaeger. 5. Cut over: remove the bind from `spikersoft-blog-media-processor/docker-stack.yml`, relax placement to `node.labels.spikersoft-app == true`. Depends on: nothing (standalone dir). Blocks: uploads-batch cutover ticket (the `/app/blogs` writers in that batch).
spikerj commented 2026-07-13 19:58:23 +00:00
Author
Owner
Copy Link
Copy Source

Backend PR #260: BlogMediaMoveConsumer dual-writes moved media to the blogs bucket (same relative keys nginx serves — #531 cutover becomes a prefix swap); file-movement routes /app/blogs. Finding: API + upload-coordinator never write /app/blogs in code (API's bind is read-only static serving = #528). Bucket/backfill/bind-drop with #529.

Backend PR #260: BlogMediaMoveConsumer dual-writes moved media to the blogs bucket (same relative keys nginx serves — #531 cutover becomes a prefix swap); file-movement routes /app/blogs. Finding: API + upload-coordinator never write /app/blogs in code (API's bind is read-only static serving = #528). Bucket/backfill/bind-drop with #529.
spikerj commented 2026-07-13 20:32:40 +00:00
Author
Owner
Copy Link
Copy Source

Code MERGED (backend #260). Remaining = ops via #529 (blogs bucket + blogs-svc key + BLOGS_S3_SECRET_KEY secret — backend #264/infra #60 add the guard+env — backfill, phase-2 bind drop). Serving cutover rides #531 (angular #178).

Code MERGED (backend #260). Remaining = ops via #529 (blogs bucket + blogs-svc key + BLOGS_S3_SECRET_KEY secret — backend #264/infra #60 add the guard+env — backfill, phase-2 bind drop). Serving cutover rides #531 (angular #178).
spikerj commented 2026-07-14 22:38:36 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — found while running the epic #413 Phase 0 backfill today. This ticket's dual-write has been writing into the void.

The blogs bucket did not exist

blog-media-processor is deployed with Storage__UseS3=true and a working BLOGS_S3_SECRET_KEY (its deploy guard requires a non-empty key, so it did deploy), and its BlogMediaMirror dual-writes every processed blog image to spiker/blogs. But when we enumerated MinIO today, only 4 of 10 buckets existed — uploads, ebooks, ai-models, gitea-packages. blogs, lesson-videos, dlls, assets, quarantine, reference-data were all absent. Phase 0 step 1 (for b in ...; do mc mb -p spiker/$b) had never been run.

Why it was silent

The #413 mirrors are best-effort — they log and swallow on failure so they never block the real (disk) operation. So every blog-media dual-write since that deploy has been failing with NoSuchBucket and nobody saw it. We found no error in the current logs only because no blog media has been processed since the deploy — meaning the failure is latent: the next blog upload silently fails to mirror, and #522 would look "done" in code while writing nothing.

What we did

Ran Phase 0 on SERVER (additive only — mc mb -p + mc mirror --overwrite, no --remove, source mounted read-only):

  • Created all missing buckets.
  • blogs now backfilled: 2.3 MiB / 3 objects, matching /mnt/fusionio/spikersoft/blogs on disk (2.3 M).

So the historical blog media is now in the bucket and the target exists for future dual-writes.

Still open for #522 to verify

  1. Confirm blog-media-processor's blogs-svc key has write on blogs (the bucket existing is necessary but not sufficient — the scoped policy must allow it). A 403 would be as silent as the missing bucket was.
  2. After the next real blog upload, confirm the new object lands in spiker/blogs (was there previously no monitoring that would have caught the NoSuchBucket failures? If so, that gap is worth its own note — best-effort mirrors that swallow errors need a metric/alert, or every migration ticket is untestable).

Related: #528 (full bucket census in our findings comment there), #529 (Phase 0), #526 (quarantine — that bucket was also missing until today).

**QA Team** — found while running the epic #413 Phase 0 backfill today. This ticket's dual-write has been writing into the void. ## The `blogs` bucket did not exist `blog-media-processor` is deployed with `Storage__UseS3=true` and a working `BLOGS_S3_SECRET_KEY` (its deploy guard requires a non-empty key, so it *did* deploy), and its `BlogMediaMirror` dual-writes every processed blog image to `spiker/blogs`. But when we enumerated MinIO today, **only 4 of 10 buckets existed** — `uploads`, `ebooks`, `ai-models`, `gitea-packages`. **`blogs`, `lesson-videos`, `dlls`, `assets`, `quarantine`, `reference-data` were all absent.** Phase 0 step 1 (`for b in ...; do mc mb -p spiker/$b`) had never been run. ## Why it was silent The #413 mirrors are **best-effort** — they log and swallow on failure so they never block the real (disk) operation. So every blog-media dual-write since that deploy has been failing with `NoSuchBucket` and nobody saw it. We found no error in the current logs only because no blog media has been processed since the deploy — meaning the failure is **latent**: the *next* blog upload silently fails to mirror, and #522 would look "done" in code while writing nothing. ## What we did Ran Phase 0 on SERVER (additive only — `mc mb -p` + `mc mirror --overwrite`, no `--remove`, source mounted read-only): - Created all missing buckets. - `blogs` now backfilled: **2.3 MiB / 3 objects**, matching `/mnt/fusionio/spikersoft/blogs` on disk (2.3 M). So the historical blog media is now in the bucket and the target exists for future dual-writes. ## Still open for #522 to verify 1. Confirm `blog-media-processor`'s `blogs-svc` key has **write** on `blogs` (the bucket existing is necessary but not sufficient — the scoped policy must allow it). A `403` would be as silent as the missing bucket was. 2. After the next real blog upload, confirm the new object lands in `spiker/blogs` (was there previously no monitoring that would have caught the `NoSuchBucket` failures? If so, that gap is worth its own note — best-effort mirrors that swallow errors need a metric/alert, or every migration ticket is untestable). Related: #528 (full bucket census in our findings comment there), #529 (Phase 0), #526 (quarantine — that bucket was also missing until today).
spikerj commented 2026-07-18 17:59:02 +00:00
Author
Owner
Copy Link
Copy Source

Epic #413 accuracy pass (2026-07-18) — closing: this ticket's scope is complete; the two verify items are #529 phase-0/1 lines.

Verified against current masters:

  • Code merged: backend #260 (BlogMediaMirror + file-movement /app/blogs route in MovedFileMirror) — present on backend master
  • Stack live in dual-run: spikersoft-blog-media-processor/docker-stack.yml carries Storage__UseS3=true + blogs-svc
  • blogs bucket exists and is backfilled (2.3 MiB / 3 obj, matches disk) — QA created it 2026-07-14 after finding the dual-write had been targeting a missing bucket

The two items QA left open here are generic cutover-gate checks that apply to the whole batch, and they're already runbook lines in #529 phase 0/1:

  1. confirm blogs-svc has write on blogs (phase 0 step 2 — key policies; a 403 is as silent as the missing bucket was)
  2. confirm the next real blog upload lands in spiker/blogs (phase 1 soak verification)

Consolidating those into #529 so this group has exactly one open ops ticket. Also noting the QA process point stands: best-effort mirrors that swallow errors need a metric/alert — carried in #529's soak phase.

Residual tracked in: #529 (key-policy verification + soak + phase-2 bind drop).

**Epic #413 accuracy pass (2026-07-18) — closing: this ticket's scope is complete; the two verify items are #529 phase-0/1 lines.** Verified against current masters: - Code merged: backend #260 (`BlogMediaMirror` + file-movement `/app/blogs` route in `MovedFileMirror`) — present on backend master - Stack live in dual-run: `spikersoft-blog-media-processor/docker-stack.yml` carries `Storage__UseS3=true` + `blogs-svc` - `blogs` bucket exists and is backfilled (2.3 MiB / 3 obj, matches disk) — QA created it 2026-07-14 after finding the dual-write had been targeting a missing bucket The two items QA left open here are generic cutover-gate checks that apply to the whole batch, and they're already runbook lines in #529 phase 0/1: 1. confirm `blogs-svc` has **write** on `blogs` (phase 0 step 2 — key policies; a 403 is as silent as the missing bucket was) 2. confirm the next real blog upload lands in `spiker/blogs` (phase 1 soak verification) Consolidating those into #529 so this group has exactly one open ops ticket. Also noting the QA process point stands: best-effort mirrors that swallow errors need a metric/alert — carried in #529's soak phase. Residual tracked in: #529 (key-policy verification + soak + phase-2 bind drop).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#522