[Infra][MinIO] Pilot: deploy MinIO + migrate security-scanner off /mnt/fusionio to S3 (proves the pattern for #413) #493

New Issue

Closed

opened 2026-07-12 01:32:51 +00:00 by spikerj · 11 comments

spikerj commented 2026-07-12 01:32:51 +00:00

Owner

Copy Link

Copy Source

Goal

Prove the MinIO/S3 decoupling pattern end-to-end on the smallest possible service, so every later migration (up to the GPU workers) follows a known-good recipe. Pilot service: security-scanner (single, read-only uploads bind — smallest blast radius, per docs/minio-storage-migration.md).

Parent epic: #413. This ticket is the first concrete step under it.

Prerequisite (step 0): actually deploy MinIO

docs/minio-storage-migration.md says "MinIO deployed" but it is not running — no minio service in the swarm and minio.spikersoft.com is not a Traefik router. So first:

1. Export root creds and deploy the committed stack:

   export MINIO_ROOT_USER=spikersoft-admin
   export MINIO_ROOT_PASSWORD=<generated 16+ char secret>   # store as a secret, not in the stack
   docker stack deploy -c minio/docker-stack.yml minio
   

2. Confirm S3 API https://minio.spikersoft.com (path-style) and console https://minio-console.spikersoft.com are up via Traefik.
3. MinIO stays pinned to SERVER (/mnt/fusionio/minio) — that's fine; consumers reach it over the network.

Pilot: migrate security-scanner

Follow the per-service recipe from the migration doc:

1. Bucket + scoped creds: uploads bucket already needed; create a read-only access key limited to it (mc admin user add + mc admin policy create/attach). One key per service; never root creds.
2. App change (backend repo): swap the filesystem read for AWSSDK.S3 (ForcePathStyle = true, ServiceURL = https://minio.spikersoft.com); config via Storage:Endpoint/Storage:Bucket + key/secret from env. Keep path-based code behind a flag for dual-run.
3. Backfill: mc mirror /mnt/fusionio/spikersoft/uploads spiker/uploads on SERVER (re-run before cutover).
4. Dual-run: deploy with S3 enabled, bind still mounted; watch Seq/Jaeger for S3 errors a few days.
5. Cut over: remove the bind mount, relax placement to node.labels.spikersoft-app == true, redeploy; verify a task can land on 4090 (burst runbook).
6. Retire the ro binding once done (the shared uploads writers migrate later as a batch — see doc's ordering constraint).

Definition of done

• MinIO deployed + fronted.
• security-scanner reads objects from the uploads bucket, no /mnt/fusionio bind, and a task successfully schedules on 4090.
• Recipe confirmed repeatable for the next service (metadata-extractor).

Explicitly out of scope

The art_pipe GPU workers (see #413 comment) — those contain Python venvs workers exec into and are a separate "bake venvs into the image + sync weights from ai-models" workstream, not a path→S3 swap.

## Goal Prove the MinIO/S3 decoupling pattern end-to-end on the smallest possible service, so every later migration (up to the GPU workers) follows a known-good recipe. Pilot service: **security-scanner** (single, read-only `uploads` bind — smallest blast radius, per `docs/minio-storage-migration.md`). Parent epic: #413. This ticket is the first concrete step under it. ## Prerequisite (step 0): actually deploy MinIO `docs/minio-storage-migration.md` says "MinIO deployed" but it is **not running** — no `minio` service in the swarm and `minio.spikersoft.com` is not a Traefik router. So first: 1. Export root creds and deploy the committed stack: ``` export MINIO_ROOT_USER=spikersoft-admin export MINIO_ROOT_PASSWORD=<generated 16+ char secret> # store as a secret, not in the stack docker stack deploy -c minio/docker-stack.yml minio ``` 2. Confirm S3 API `https://minio.spikersoft.com` (path-style) and console `https://minio-console.spikersoft.com` are up via Traefik. 3. MinIO stays pinned to SERVER (`/mnt/fusionio/minio`) — that's fine; consumers reach it over the network. ## Pilot: migrate security-scanner Follow the per-service recipe from the migration doc: 1. **Bucket + scoped creds:** `uploads` bucket already needed; create a read-only access key limited to it (`mc admin user add` + `mc admin policy create/attach`). One key per service; never root creds. 2. **App change (backend repo):** swap the filesystem read for AWSSDK.S3 (`ForcePathStyle = true`, `ServiceURL = https://minio.spikersoft.com`); config via `Storage:Endpoint`/`Storage:Bucket` + key/secret from env. Keep path-based code behind a flag for dual-run. 3. **Backfill:** `mc mirror /mnt/fusionio/spikersoft/uploads spiker/uploads` on SERVER (re-run before cutover). 4. **Dual-run:** deploy with S3 enabled, bind still mounted; watch Seq/Jaeger for S3 errors a few days. 5. **Cut over:** remove the bind mount, relax placement to `node.labels.spikersoft-app == true`, redeploy; verify a task can land on 4090 (burst runbook). 6. **Retire** the ro binding once done (the shared `uploads` writers migrate later as a batch — see doc's ordering constraint). ## Definition of done - MinIO deployed + fronted. - security-scanner reads objects from the `uploads` bucket, no `/mnt/fusionio` bind, and a task successfully schedules on 4090. - Recipe confirmed repeatable for the next service (metadata-extractor). ## Explicitly out of scope The art_pipe GPU workers (see #413 comment) — those contain Python venvs workers `exec` into and are a separate "bake venvs into the image + sync weights from `ai-models`" workstream, not a path→S3 swap.

spikerj commented 2026-07-12 01:41:50 +00:00

Author

Owner

Copy Link

Copy Source

Step 2 (backend dual-run code) is up: spikersoft-backend PR feat/scanner-s3-dual-run-493 — IStagedObjectStore seam, filesystem default + S3 impl (path-style MinIO, filesystem-shaped StagingPaths mapped to uploads-bucket keys with boundary/traversal hardening), selected by Storage:UseS3 (default false → merging is a no-op until dual-run). 12 new xUnit tests. Steps 0/1/3-6 are swarm-side and stay on this ticket: deploy the committed minio stack, scoped ro key, mc mirror backfill, flag flip for dual-run, then bind removal + spikersoft-app placement and a 4090 scheduling check.

Step 2 (backend dual-run code) is up: spikersoft-backend PR feat/scanner-s3-dual-run-493 — IStagedObjectStore seam, filesystem default + S3 impl (path-style MinIO, filesystem-shaped StagingPaths mapped to uploads-bucket keys with boundary/traversal hardening), selected by Storage:UseS3 (default false → merging is a no-op until dual-run). 12 new xUnit tests. Steps 0/1/3-6 are swarm-side and stay on this ticket: deploy the committed minio stack, scoped ro key, mc mirror backfill, flag flip for dual-run, then bind removal + spikersoft-app placement and a 4090 scheduling check.

spikerj commented 2026-07-12 01:54:38 +00:00

Author

Owner

Copy Link

Copy Source

Step 0 DONE and externally verified (2026-07-11 ~21:55): https://minio.spikersoft.com/minio/health/live → 200, API root → 403 (auth enforced, as it should be), https://minio-console.spikersoft.com → 200, valid LE cert via Traefik. Running the -cpuv1 pin from infra PR #31. Remaining checklist: (1) merge backend PR #215 (scanner dual-run code — no-op until the flag flips); (2) in the console or via mc: create the uploads bucket + a scoped READ-ONLY access key for the scanner (mc admin user add + policy attach — never root creds); (3) backfill: mc mirror /mnt/fusionio/spikersoft/uploads spiker/uploads on SERVER; (4) dual-run: set Storage__UseS3=true + Storage__AccessKey/Storage__SecretKey env on the spikersoft-security-scanner stack (bind still mounted) and watch Seq for S3 errors; (5) cutover: drop the ro bind, relax placement to node.labels.spikersoft-app == true, redeploy, and confirm a scanner task schedules on the 4090 — that's the pilot's definition of done.

Step 0 DONE and externally verified (2026-07-11 ~21:55): https://minio.spikersoft.com/minio/health/live → 200, API root → 403 (auth enforced, as it should be), https://minio-console.spikersoft.com → 200, valid LE cert via Traefik. Running the -cpuv1 pin from infra PR #31. Remaining checklist: (1) merge backend PR #215 (scanner dual-run code — no-op until the flag flips); (2) in the console or via mc: create the uploads bucket + a scoped READ-ONLY access key for the scanner (mc admin user add + policy attach — never root creds); (3) backfill: mc mirror /mnt/fusionio/spikersoft/uploads spiker/uploads on SERVER; (4) dual-run: set Storage__UseS3=true + Storage__AccessKey/Storage__SecretKey env on the spikersoft-security-scanner stack (bind still mounted) and watch Seq for S3 errors; (5) cutover: drop the ro bind, relax placement to node.labels.spikersoft-app == true, redeploy, and confirm a scanner task schedules on the 4090 — that's the pilot's definition of done.

spikerj commented 2026-07-12 02:27:03 +00:00

Author

Owner

Copy Link

Copy Source

MinIO admin setup done — scanner is unblocked

Completed the object-storage side of the pilot against minio.spikersoft.com (root creds provided by Joey, out-of-band):

• Bucket: uploads created, confirmed private (mc anonymous get → private; anon HTTP already returns 403).
• Scoped service user: scanner-svc created with a dedicated scanner-readonly policy — s3:GetObject on uploads/* + s3:GetBucketLocation on uploads only. No write, delete, or cross-bucket access.
• Enforcement verified with the scoped key: stat/cat on an uploads object succeed; cp (PUT) is rejected with Insufficient permissions; bucket listing shows only uploads/ (no other buckets visible). Test probe object removed afterward — bucket left empty.

The scoped keypair (access key scanner-svc + secret) was handed to Joey privately for the scanner stack env — not posted here. It goes in as Storage__AccessKey / Storage__SecretKey alongside Storage__UseS3=true.

Remaining for cutover (SERVER-side, Joey):

1. Backfill existing files: mc mirror /mnt/fusionio/spikersoft/uploads spiker/uploads on SERVER.
2. Set Storage__UseS3=true + the two scoped-cred env vars on the security-scanner stack and redeploy (dual-run: reads shift to S3, filesystem bind still present as fallback).
3. Cutover / definition-of-done: drop the /mnt/fusionio read-only bind, relax placement from node.hostname == SERVER to node.labels.spikersoft-app == true, confirm the scanner task can schedule on 4090.

Follow-up reminder: root credentials should be rotated now that setup is complete. Rotating root means redeploying the minio stack with new MINIO_ROOT_USER/MINIO_ROOT_PASSWORD env — logging it here so it isn't forgotten. The scoped scanner-svc key is independent of root and survives rotation.

### MinIO admin setup done — scanner is unblocked Completed the object-storage side of the pilot against `minio.spikersoft.com` (root creds provided by Joey, out-of-band): - **Bucket:** `uploads` created, confirmed **private** (`mc anonymous get` → `private`; anon HTTP already returns 403). - **Scoped service user:** `scanner-svc` created with a dedicated `scanner-readonly` policy — `s3:GetObject` on `uploads/*` + `s3:GetBucketLocation` on `uploads` only. **No** write, delete, or cross-bucket access. - **Enforcement verified** with the scoped key: `stat`/`cat` on an `uploads` object succeed; `cp` (PUT) is rejected with *Insufficient permissions*; bucket listing shows only `uploads/` (no other buckets visible). Test probe object removed afterward — bucket left empty. The scoped keypair (access key `scanner-svc` + secret) was handed to Joey privately for the scanner stack env — **not** posted here. It goes in as `Storage__AccessKey` / `Storage__SecretKey` alongside `Storage__UseS3=true`. **Remaining for cutover (SERVER-side, Joey):** 1. Backfill existing files: `mc mirror /mnt/fusionio/spikersoft/uploads spiker/uploads` on SERVER. 2. Set `Storage__UseS3=true` + the two scoped-cred env vars on the security-scanner stack and redeploy (dual-run: reads shift to S3, filesystem bind still present as fallback). 3. Cutover / definition-of-done: drop the `/mnt/fusionio` read-only bind, relax placement from `node.hostname == SERVER` to `node.labels.spikersoft-app == true`, confirm the scanner task can schedule on `4090`. **Follow-up reminder:** root credentials should be rotated now that setup is complete. Rotating root means redeploying the `minio` stack with new `MINIO_ROOT_USER`/`MINIO_ROOT_PASSWORD` env — logging it here so it isn't forgotten. The scoped `scanner-svc` key is independent of root and survives rotation.

spikerj commented 2026-07-12 04:12:12 +00:00

Author

Owner

Copy Link

Copy Source

Pilot re-scope: the reader-only pilot can't land alone — first WRITER migrated

Finding (verified live tonight): Joey's book upload traced end-to-end in Seq — the scanner scanned the staging PDF from the filesystem, metadata-extractor wrote covers/pages/images to /app/ebooks/books/SPIKER6924601/, file-movement moved the PDF there, and MinIO saw zero S3 traffic. Two structural reasons:

1. Writer/reader split: #215 gave the scanner S3 reads, but every writer (API staging, coordinator, extractor, file-movement) still writes /mnt/fusionio. Flipping the scanner alone would make it look in the bucket for files that only exist on disk → every new upload quarantined. The one-time mc mirror backfill can't cover new writes.
2. Serving dependency: Angular's nginx serves covers/pages from the fusionio root bind — any S3-only writer would break the site.

Direction: dual-write, writer-first. spikersoft-backend PR #222 (merged) gives metadata-extractor an IEbookArtifactMirror — after each extraction, the finished books/<ISBN>/ tree is ALSO uploaded to the new ebooks bucket (created, private). Filesystem stays authoritative; the mirror never throws (failures degrade to filesystem-only). 13 new xUnit tests.

To activate (in order):

1. Joey: create the metadata-svc MinIO key — policy staged (rw ebooks/*, ro uploads/*); needs explicit authorization for me to mint credentials.
2. Add METADATA_S3_SECRET_KEY Actions secret (I'll do it once the key exists).
3. Merge backend #223 (deploy secret plumbing + secrets docs re-land) and infra #35 (stack env) → pull /mnt/infrastructure → dispatch the Metadata Extractor workflow.
4. Verify: upload a book → mc ls --recursive spiker/ebooks shows books/<ISBN>/covers|pages|images/….

Scanner status: dual-run reads stay OFF (correct — it would break on fresh uploads until the staging writers also dual-write). Sequencing: ebooks writers first (extractor done, file-movement + coordinator next), then staging, then the scanner flip + serving cutover. Hardening PRs backend #219 / infra #33 (the ${VAR:?} footgun) still open.

Also noting for posterity: two commits got stranded by the merge-fast race (branch merged+deleted before a second push — #217's docs, #222's workflow commit). Both re-landed via #223. If a PR gets a second push from me, hold merging for ~a minute.

### Pilot re-scope: the reader-only pilot can't land alone — first WRITER migrated **Finding (verified live tonight):** Joey's book upload traced end-to-end in Seq — the scanner scanned the staging PDF from the filesystem, metadata-extractor wrote covers/pages/images to `/app/ebooks/books/SPIKER6924601/`, file-movement moved the PDF there, and **MinIO saw zero S3 traffic**. Two structural reasons: 1. **Writer/reader split:** #215 gave the scanner S3 *reads*, but every *writer* (API staging, coordinator, extractor, file-movement) still writes `/mnt/fusionio`. Flipping the scanner alone would make it look in the bucket for files that only exist on disk → every new upload quarantined. The one-time `mc mirror` backfill can't cover new writes. 2. **Serving dependency:** Angular's nginx serves covers/pages from the fusionio root bind — any S3-*only* writer would break the site. **Direction: dual-write, writer-first.** spikersoft-backend PR #222 (merged) gives metadata-extractor an `IEbookArtifactMirror` — after each extraction, the finished `books/<ISBN>/` tree is ALSO uploaded to the new **`ebooks` bucket** (created, private). Filesystem stays authoritative; the mirror never throws (failures degrade to filesystem-only). 13 new xUnit tests. **To activate (in order):** 1. **Joey:** create the `metadata-svc` MinIO key — policy staged (rw `ebooks/*`, ro `uploads/*`); needs explicit authorization for me to mint credentials. 2. Add `METADATA_S3_SECRET_KEY` Actions secret (I'll do it once the key exists). 3. Merge backend #223 (deploy secret plumbing + secrets docs re-land) and infra #35 (stack env) → pull /mnt/infrastructure → dispatch the Metadata Extractor workflow. 4. Verify: upload a book → `mc ls --recursive spiker/ebooks` shows `books/<ISBN>/covers|pages|images/…`. **Scanner status:** dual-run reads stay OFF (correct — it would break on fresh uploads until the *staging* writers also dual-write). Sequencing: ebooks writers first (extractor done, file-movement + coordinator next), then staging, then the scanner flip + serving cutover. Hardening PRs backend #219 / infra #33 (the ${VAR:?} footgun) still open. Also noting for posterity: two commits got stranded by the merge-fast race (branch merged+deleted before a second push — #217's docs, #222's workflow commit). Both re-landed via #223. If a PR gets a second push from me, hold merging for ~a minute.

spikerj commented 2026-07-12 04:52:06 +00:00

Author

Owner

Copy Link

Copy Source

✅ Dual-write verified in production. Book upload 8809a6e6 (ISBN SPIKER4684262, 37 pages):

Seq 04:49:15 — Ebook mirror: uploaded 75/75 artifacts from /app/ebooks/books/SPIKER4684262 to bucket ebooks
mc du spiker/ebooks — 151MiB, 75 objects (covers png+avif, 37 page PDFs, page images)

Keys mirror the filesystem layout exactly (books/<ISBN>/covers|pages|images/…), so the serving cutover is a prefix swap when we get there. Filesystem untouched/authoritative; pipeline completed normally (book+quiz records, file moved, orchestrator fanned out 36 image-description requests).

The last blocker was subtle: MetadataExtractionConsumer constructs the extraction service directly (for its Rabbit channel), bypassing the DI factory the mirror was registered in — fixed in backend PR #224 by injecting IEbookArtifactMirror into that construction.

#493 remaining, in order: (1) migrate the other ebooks writers the same way (file-movement's final PDF move, coordinator/API staging writes); (2) staging (uploads bucket) dual-write; (3) THEN flip the scanner's S3 reads (safe once writers dual-write); (4) serving cutover + drop binds + relax SERVER pins. Plus the ai-models bucket direction from #497.

✅ **Dual-write verified in production.** Book upload 8809a6e6 (ISBN SPIKER4684262, 37 pages): Seq 04:49:15 — Ebook mirror: uploaded 75/75 artifacts from /app/ebooks/books/SPIKER4684262 to bucket ebooks mc du spiker/ebooks — 151MiB, 75 objects (covers png+avif, 37 page PDFs, page images) Keys mirror the filesystem layout exactly (`books/<ISBN>/covers|pages|images/…`), so the serving cutover is a prefix swap when we get there. Filesystem untouched/authoritative; pipeline completed normally (book+quiz records, file moved, orchestrator fanned out 36 image-description requests). The last blocker was subtle: MetadataExtractionConsumer constructs the extraction service directly (for its Rabbit channel), bypassing the DI factory the mirror was registered in — fixed in backend PR #224 by injecting IEbookArtifactMirror into that construction. **#493 remaining, in order:** (1) migrate the other ebooks writers the same way (file-movement's final PDF move, coordinator/API staging writes); (2) staging (uploads bucket) dual-write; (3) THEN flip the scanner's S3 reads (safe once writers dual-write); (4) serving cutover + drop binds + relax SERVER pins. Plus the ai-models bucket direction from #497.

spikerj referenced this issue 2026-07-12 05:04:36 +00:00

[infra drift] image-description & trellis3d deploy to /mnt/infrastructure folders that don't exist #403

spikerj referenced this issue 2026-07-12 05:09:38 +00:00

[Infra][MinIO] Move Gitea package/container-registry storage to the gitea-packages bucket #498

spikerj referenced this issue 2026-07-12 06:08:29 +00:00

[Bug][Backend] Book deletion leaves all artifacts behind — on disk AND in the MinIO ebooks bucket (no purge path exists) #499

spikerj referenced this issue 2026-07-12 06:21:26 +00:00

[Bug][Backend] Book deletion leaves all artifacts behind — on disk AND in the MinIO ebooks bucket (no purge path exists) #499

spikerj referenced this issue 2026-07-12 07:00:00 +00:00

[Bug][Infra][GPU] image-description lease starves — gpu-coordinator single-lane budget still on SERVER (8192MB) vs 16000MB needed; enable 4090 lane #500

spikerj referenced this issue 2026-07-12 08:06:53 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services #503

spikerj referenced this issue 2026-07-12 08:09:32 +00:00

[Bug][Infra] MinIO doesn't recover after a transient /mnt/fusionio drive blip — stuck '0 drives provided', 'mc ready local' healthcheck misses it, needs manual restart #504

spikerj referenced this issue 2026-07-12 12:33:39 +00:00

[Bug][Infra] MinIO single drive faulted under first sustained concurrent write load — writers aborted mid-stream, task left LB #505

spikerj referenced this issue 2026-07-12 17:42:46 +00:00

[Bug][Prod][GPU][CRITICAL] image-description crash-loops (0/1) — ModuleNotFoundError: 'message_failure' — phantom deploy of the #506 fix, Dockerfile missing COPY #508

spikerj referenced this issue 2026-07-12 18:13:05 +00:00

[Bug][Backend][Data] BookPages inflation — quiz saw 6 pages on a 3-page book; one book linked 570 pages (staleness-retry double-processing suspected) #509

spikerj commented 2026-07-12 18:25:10 +00:00

Author

Owner

Copy Link

Copy Source

Progress sweep: #500/#502/#508/#506/#497 all closed today with production evidence — captions flow from the 4090 with images AND model sourced from MinIO (110 completed captions in Mongo). New: PR #235 adds the file-movement dual-write so the moved book file itself joins its covers/pages in the ebooks bucket — each book's bucket footprint is now complete. Remaining #493 phases: staging (uploads) writers dual-write → scanner S3 flip → serving cutover → drop binds/pins.

Progress sweep: **#500/#502/#508/#506/#497 all closed today with production evidence** — captions flow from the 4090 with images AND model sourced from MinIO (110 completed captions in Mongo). New: **PR #235** adds the file-movement dual-write so the moved book file itself joins its covers/pages in the `ebooks` bucket — each book's bucket footprint is now complete. Remaining #493 phases: staging (`uploads`) writers dual-write → scanner S3 flip → serving cutover → drop binds/pins.

spikerj commented 2026-07-12 18:42:59 +00:00

Author

Owner

Copy Link

Copy Source

Scanner-flip preflight complete — plus a sequencing hazard to defuse:

✅ security.scan.requested has exactly ONE publisher (UploadOrchestrator) — and merged PR #236 mirrors the staging file immediately BEFORE that publish. Once the mirror deploys, every scanned file is bucket-guaranteed → the scanner flip is safe by construction.

⚠️ Hazard: the scanner stack has carried Storage__UseS3=true on master since infra #32, and scanner CI redeploys on any merge touching Common/**/Data/** — a redeploy before the mirror is live flips the scanner early → new uploads fail scans fail-closed. Defusal: merge infra #43 → pull /mnt/infrastructure → redeploy upload-coordinator → verify one upload lands in mc ls spiker/uploads/ebooks/ — after that the scanner's next redeploy is the safe, intended flip. Then serving cutover closes this ticket.

(Comment delayed by the laptop-server disk-full incident — gitea's package-blob cleanup cron hadn't run since before the registry wipe; triggered now.)

**Scanner-flip preflight complete — plus a sequencing hazard to defuse:** ✅ `security.scan.requested` has exactly ONE publisher (UploadOrchestrator) — and merged PR #236 mirrors the staging file immediately BEFORE that publish. Once the mirror deploys, every scanned file is bucket-guaranteed → the scanner flip is safe by construction. ⚠️ Hazard: the scanner stack has carried `Storage__UseS3=true` on master since infra #32, and scanner CI redeploys on any merge touching `Common/**`/`Data/**` — a redeploy before the mirror is live flips the scanner early → new uploads fail scans fail-closed. **Defusal: merge infra #43 → pull /mnt/infrastructure → redeploy upload-coordinator → verify one upload lands in `mc ls spiker/uploads/ebooks/`** — after that the scanner's next redeploy is the safe, intended flip. Then serving cutover closes this ticket. (Comment delayed by the laptop-server disk-full incident — gitea's package-blob cleanup cron hadn't run since before the registry wipe; triggered now.)

spikerj commented 2026-07-12 21:29:49 +00:00

Author

Owner

Copy Link

Copy Source

Staging mirror verified live in production (the gate this ticket was waiting on). Two real uploads today:

• 19:52:52 UTC — S3StagingMirror: Staging mirror: uploaded uploads/ebooks/spikerj_1783885962_...pdf (Seq) + object in the bucket (mc)
• 20:04:12 UTC — same for spikerj_1783886641_...pdf

Full pipeline trace on the first: staging save → mirror → SecurityScanConsumer scan (clean, 4s after mirror) → metadata extraction → file movement. The mirror precedes the scan request exactly as designed, so the scanner S3 flip is safe whenever the scanner (re)deploys with its long-standing Storage__UseS3=true env.

Remaining to close: (1) one-time confirmation the scanner actually reads from S3 (its boot log line or docker service inspect env on the running task), (2) then drop the staging bind-mount from the scanner stack. Serving-side cutover for book artifacts is already proven (captions/embeddings/quiz consumed MinIO-hosted artifacts end-to-end).

**Staging mirror verified live in production** (the gate this ticket was waiting on). Two real uploads today: - 19:52:52 UTC — `S3StagingMirror: Staging mirror: uploaded uploads/ebooks/spikerj_1783885962_...pdf` (Seq) + object in the bucket (mc) - 20:04:12 UTC — same for `spikerj_1783886641_...pdf` Full pipeline trace on the first: staging save → mirror → SecurityScanConsumer scan (clean, 4s after mirror) → metadata extraction → file movement. The mirror precedes the scan request exactly as designed, so **the scanner S3 flip is safe** whenever the scanner (re)deploys with its long-standing `Storage__UseS3=true` env. Remaining to close: (1) one-time confirmation the scanner actually reads from S3 (its boot log line or `docker service inspect` env on the running task), (2) then drop the staging bind-mount from the scanner stack. Serving-side cutover for book artifacts is already proven (captions/embeddings/quiz consumed MinIO-hosted artifacts end-to-end).

spikerj referenced this issue 2026-07-13 00:40:39 +00:00

[Infra][MinIO] Migrate blogs dir + BlogMediaProcessor to S3 (blogs bucket, epic #413) #522

spikerj referenced this issue 2026-07-13 00:40:41 +00:00

[Infra][MinIO] Uploads-tree coordinated batch cutover — 7 services off /mnt/fusionio (epic #413) #529

spikerj referenced this issue 2026-07-13 00:41:02 +00:00

[Infra][Architecture] Provider-agnostic object storage (MinIO -> S3/Azure Blob) to replace GlusterFS/host-path model #413

spikerj commented 2026-07-13 00:41:03 +00:00

Author

Owner

Copy Link

Copy Source

Status check while filing the #413 child tree (2026-07-12): metadata-extractor is already dual-running in prod (Storage__UseS3=true + metadata-svc key in its stack file), and the S3 code paths exist in four services (UploadCoordinator StagingMirror, MetadataExtractor EbookArtifactMirror, FileMovement MovedFileMirror/BookArtifactPurger, SecurityScanner S3StagedObjectStore). Remaining for this ticket: security-scanner cutover steps 4-6 (dual-run watch → remove ro bind → relax placement → verify a task lands on 4090). Follow-on tickets for everything else are listed on #413.

Status check while filing the #413 child tree (2026-07-12): metadata-extractor is already dual-running in prod (Storage__UseS3=true + metadata-svc key in its stack file), and the S3 code paths exist in four services (UploadCoordinator StagingMirror, MetadataExtractor EbookArtifactMirror, FileMovement MovedFileMirror/BookArtifactPurger, SecurityScanner S3StagedObjectStore). Remaining for this ticket: security-scanner cutover steps 4-6 (dual-run watch → remove ro bind → relax placement → verify a task lands on 4090). Follow-on tickets for everything else are listed on #413.

spikerj commented 2026-07-13 20:33:04 +00:00

Author

Owner

Copy Link

Copy Source

Pilot status: MinIO deployed+hardened (infra #54/#55), scanner dual-read code live since the flag flip, recipe proven and generalized into SpikerSoft.Storage (#533) — every later migration followed it. Remaining DoD item: drop the ro bind + schedule a scanner task on 4090, which is #529 phase 2 (runbook merged-pending in infra #60). Suggest closing this into #529 once phase 2 verifies.

Pilot status: MinIO deployed+hardened (infra #54/#55), scanner dual-read code live since the flag flip, recipe proven and generalized into SpikerSoft.Storage (#533) — every later migration followed it. Remaining DoD item: drop the ro bind + schedule a scanner task on 4090, which is #529 phase 2 (runbook merged-pending in infra #60). Suggest closing this into #529 once phase 2 verifies.

spikerj referenced this issue 2026-07-13 21:07:01 +00:00

[Epic][Security][Infra] Central secrets management — OpenBao vault + PKI, retire scattered passwords #543

spikerj referenced this issue 2026-07-13 21:07:25 +00:00

[Security][Backend] OpenBao Phase 2: SpikerSoft.Secrets host-builder seam — service creds from Bao (epic #543) #546

spikerj referenced this issue 2026-07-15 20:41:31 +00:00

[Security][Backend] OpenBao Phase 2: SpikerSoft.Secrets host-builder seam — service creds from Bao (epic #543) #546

spikerj referenced this issue 2026-07-18 01:16:29 +00:00

[Infra][MinIO] Cut over book/ebook artifacts to MinIO as sole store (ebooks slice of #413) + fix inline-mirror pipeline stall #696

spikerj commented 2026-07-18 17:58:48 +00:00

Author

Owner

Copy Link

Copy Source

Epic #413 accuracy pass (2026-07-18) — closing: the pilot's goal is achieved; the one residual step is #529's, not this ticket's.

What this ticket set out to prove is proven, verified against current masters:

• MinIO deployed, fronted, hardened (infra #31/#54/#55, drive-aware healthcheck, -cpuv1 pin)
• security-scanner dual-read live: Storage__UseS3=true + scanner-svc scoped key on the stack (spikersoft-security-scanner/docker-stack.yml), S3StagedObjectStore on backend master
• The recipe generalized into SpikerSoft.Storage (#533, closed) and was followed by every later migration — 10 stacks now carry Storage__UseS3=true
• Staging writer dual-write verified live in prod (2026-07-12 comments above), making the scanner flip safe by construction

The only unfinished DoD line — drop the :ro bind + relax node.hostname == SERVER + see a task land on 4090 — is literally #529 phase 2 for the whole 7-service batch (the scanner is row 3 of the runbook's table). Keeping two tickets open for one bind-drop is what made this group inaccurate; consolidating into #529, exactly as the 2026-07-13 comment recommended.

Residual tracked in: #529 (phase 2 bind drop + placement relax + 4090 verification).

**Epic #413 accuracy pass (2026-07-18) — closing: the pilot's goal is achieved; the one residual step is #529's, not this ticket's.** What this ticket set out to prove is proven, verified against current masters: - MinIO deployed, fronted, hardened (infra #31/#54/#55, drive-aware healthcheck, `-cpuv1` pin) - security-scanner dual-read live: `Storage__UseS3=true` + `scanner-svc` scoped key on the stack (`spikersoft-security-scanner/docker-stack.yml`), `S3StagedObjectStore` on backend master - The recipe generalized into `SpikerSoft.Storage` (#533, closed) and was followed by **every** later migration — 10 stacks now carry `Storage__UseS3=true` - Staging writer dual-write verified live in prod (2026-07-12 comments above), making the scanner flip safe by construction The only unfinished DoD line — *drop the `:ro` bind + relax `node.hostname == SERVER` + see a task land on 4090* — is literally **#529 phase 2** for the whole 7-service batch (the scanner is row 3 of the runbook's table). Keeping two tickets open for one bind-drop is what made this group inaccurate; consolidating into #529, exactly as the 2026-07-13 comment recommended. Residual tracked in: #529 (phase 2 bind drop + placement relax + 4090 verification).

spikerj closed this issue 2026-07-18 17:58:55 +00:00

spikerj referenced this issue 2026-07-18 18:02:04 +00:00

[Infra][Architecture] Provider-agnostic object storage (MinIO -> S3/Azure Blob) to replace GlusterFS/host-path model #413

spikerj referenced this issue 2026-07-21 01:54:35 +00:00

EPIC: Angular 22 + Nx 23 full-leverage modernization (signal forms, resource APIs, affected CI, remote cache) #722

spikerj referenced this issue 2026-07-27 15:18:36 +00:00

EPIC: Sponsor family card redesign — image-backed destination tiles, achievements strip, parent-gated age #868

spikerj referenced this issue 2026-07-29 15:28:25 +00:00

Destination background images: anonymous /destination-images MediaRoute from MinIO assets bucket #869

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

No results found.

Labels

Clear labels

agentic

Actionable surfaced by the agentic architecture & security review

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

sonarqube

Code quality issue surfaced by SonarQube (bug / code smell)

wontfix

This won't be fixed

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

spikerj (Joseph Spiker)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#493