spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][MinIO] Cut over book/ebook artifacts to MinIO as sole store (ebooks slice of #413) + fix inline-mirror pipeline stall #696

New Issue
Open
opened 2026-07-18 01:16:28 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-18 01:16:28 +00:00
Owner
Copy Link
Copy Source

Parent epic: #413 (provider-agnostic object storage). Pilot: #493 (dual-write mirror). Related: #613 (metadata-svc bucket-write Access Denied).

Goal

Make MinIO the sole store for book/ebook artifacts (covers, page PDFs, page images) — no local-filesystem authoritative copy, no dual-write mirror. Completes the ebooks slice of #413.

Decisions locked with @spikerj (2026-07-18):

  • Single-node MinIO is acceptable for now (no distributed/replicated MinIO in this scope).
  • The wedged book-upload workflow 86bfade3-bbbd-480a-a2c2-9c5adb71b89b is being left as a live specimen (not restarted) for diagnosis.

Motivating root cause — the inline mirror silently wedges the pipeline

A test book upload (86bfade3…, godot PDF, 342 pages) stalled permanently at Status: extracting_metadata (workflow book-upload-workflows), ErrorMessage: null, no downstream, no book created. Diagnosis:

  1. MetadataExtractionService.ExtractFromPdfAsync calls _artifactMirror.MirrorBookAsync(...) inline/awaited (line ~114), before publishing metadata.extracted. A hang there freezes the whole book's pipeline.
  2. The S3 client (SpikerSoft.Storage/ObjectStorageServiceCollectionExtensions.cs) is built with only ServiceURL + ForcePathStyle — no Timeout/ReadWriteTimeout/retry cap. A single wedged PutObject sits on a half-open socket forever. Observed: still wedged 27+ min later; MinIO itself healthy the whole time (1ms internal / 76ms public health response).
  3. The extractor is not on MinIO's network, so it reaches MinIO via the public minio.spikersoft.com — hairpinning through the edge + Traefik + TLS for every one of ~500 objects. Couples storage writes to Traefik and multiplies blast radius.

The mirror's "never throws / filesystem authoritative / degrade silently" contract makes this invisible: it swallows exceptions but cannot survive a hang.

Architecture facts that shape the cutover

  • MinIO data is bind-mounted to /mnt/fusionio/minio — the same physical disk as /app/ebooks local storage. So "MinIO-only" is an access-path unification (all I/O via S3 API), not a data relocation; bytes stay on fusionio.
  • Serving today: Angular nginx serves covers/pages directly off the /mnt/fusionio bind — the load-bearing piece to change.
  • MinIO is only on traefik-public, no published ports, pinned to SERVER.
  • S3 plumbing already exists: SpikerSoft.Storage (S3ObjectStore, IObjectStore, ObjectKeys.TryMapKey) + scanner read seam. Key layout already matches served URL shape → read cutover ≈ prefix swap.

Phased plan

Phase 0 — Foundations (low risk; also the genuine fix for the stall)

  • Give storage-touching services an internal route to MinIO (dedicated minio overlay or traefik-public); point Storage:ServiceUrl at http://minio:9000. Removes hairpin/TLS/Traefik coupling.
  • Add Timeout + ReadWriteTimeout + bounded retries to the S3 client. Mandatory before MinIO is critical.
  • Verify each service credential (esp. metadata-svc) can read and write the ebooks bucket; fix bucket policy/IAM (see #613).

Phase 1 — Write MinIO as primary. Producers write via IObjectStore with real error handling (failed write fails/retries the unit of work — not silent degrade). Filesystem kept as temporary secondary until reads move.

Phase 2 — Backfill. One-time sync of existing books/** from fusionio into ebooks bucket; verify counts/checksums so MinIO holds 100% history before reads cut over.

Phase 3 — Cut over reads/serving (highest risk). Re-point nginx/Angular from filesystem bind to MinIO (reverse-proxy /books/* → minio:9000 or presigned URLs), behind a flag, filesystem as instant rollback. Update backend read seams.

Phase 4 — Decommission. Remove filesystem write path, the /mnt/fusionio serving bind, and delete the IEbookArtifactMirror dual-write seam. (MinIO's own data bind stays.)

Acceptance

  • No authoritative filesystem copy of book artifacts; all reads/writes via MinIO S3 API.
  • S3 client has enforced timeouts; no unbounded storage operation on any critical path.
  • Book-upload pipeline cannot silently wedge on a storage blip.
  • IEbookArtifactMirror dual-write seam removed.
**Parent epic:** #413 (provider-agnostic object storage). **Pilot:** #493 (dual-write mirror). **Related:** #613 (metadata-svc bucket-write Access Denied). ## Goal Make **MinIO the sole store** for book/ebook artifacts (covers, page PDFs, page images) — no local-filesystem authoritative copy, no dual-write mirror. Completes the ebooks slice of #413. Decisions locked with @spikerj (2026-07-18): - **Single-node MinIO** is acceptable for now (no distributed/replicated MinIO in this scope). - The wedged book-upload workflow `86bfade3-bbbd-480a-a2c2-9c5adb71b89b` is being **left as a live specimen** (not restarted) for diagnosis. ## Motivating root cause — the inline mirror silently wedges the pipeline A test book upload (`86bfade3…`, godot PDF, 342 pages) stalled permanently at `Status: extracting_metadata` (workflow `book-upload-workflows`), `ErrorMessage: null`, no downstream, no book created. Diagnosis: 1. `MetadataExtractionService.ExtractFromPdfAsync` calls `_artifactMirror.MirrorBookAsync(...)` **inline/awaited (line ~114), before publishing `metadata.extracted`**. A hang there freezes the whole book's pipeline. 2. The S3 client (`SpikerSoft.Storage/ObjectStorageServiceCollectionExtensions.cs`) is built with **only `ServiceURL` + `ForcePathStyle` — no `Timeout`/`ReadWriteTimeout`/retry cap**. A single wedged `PutObject` sits on a half-open socket forever. Observed: still wedged 27+ min later; MinIO itself healthy the whole time (1ms internal / 76ms public health response). 3. The extractor is **not on MinIO's network**, so it reaches MinIO via the **public `minio.spikersoft.com`** — hairpinning through the edge + Traefik + TLS for every one of ~500 objects. Couples storage writes to Traefik and multiplies blast radius. The mirror's "never throws / filesystem authoritative / degrade silently" contract makes this invisible: it swallows exceptions but cannot survive a *hang*. ## Architecture facts that shape the cutover - **MinIO data is bind-mounted to `/mnt/fusionio/minio`** — the same physical disk as `/app/ebooks` local storage. So "MinIO-only" is an **access-path unification** (all I/O via S3 API), *not* a data relocation; bytes stay on fusionio. - Serving today: Angular nginx serves covers/pages **directly off the `/mnt/fusionio` bind** — the load-bearing piece to change. - MinIO is only on `traefik-public`, no published ports, pinned to SERVER. - S3 plumbing already exists: `SpikerSoft.Storage` (`S3ObjectStore`, `IObjectStore`, `ObjectKeys.TryMapKey`) + scanner read seam. Key layout already matches served URL shape → read cutover ≈ prefix swap. ## Phased plan **Phase 0 — Foundations (low risk; also the genuine fix for the stall)** - Give storage-touching services an internal route to MinIO (dedicated `minio` overlay or `traefik-public`); point `Storage:ServiceUrl` at `http://minio:9000`. Removes hairpin/TLS/Traefik coupling. - Add `Timeout` + `ReadWriteTimeout` + bounded retries to the S3 client. Mandatory before MinIO is critical. - Verify each service credential (esp. `metadata-svc`) can **read and write** the `ebooks` bucket; fix bucket policy/IAM (see #613). **Phase 1 — Write MinIO as primary.** Producers write via `IObjectStore` with real error handling (failed write fails/retries the unit of work — not silent degrade). Filesystem kept as temporary secondary until reads move. **Phase 2 — Backfill.** One-time sync of existing `books/**` from fusionio into `ebooks` bucket; verify counts/checksums so MinIO holds 100% history before reads cut over. **Phase 3 — Cut over reads/serving (highest risk).** Re-point nginx/Angular from filesystem bind to MinIO (reverse-proxy `/books/*` → `minio:9000` or presigned URLs), behind a flag, filesystem as instant rollback. Update backend read seams. **Phase 4 — Decommission.** Remove filesystem write path, the `/mnt/fusionio` *serving* bind, and delete the `IEbookArtifactMirror` dual-write seam. (MinIO's own data bind stays.) ## Acceptance - No authoritative filesystem copy of book artifacts; all reads/writes via MinIO S3 API. - S3 client has enforced timeouts; no unbounded storage operation on any critical path. - Book-upload pipeline cannot silently wedge on a storage blip. - `IEbookArtifactMirror` dual-write seam removed.
spikerj commented 2026-07-18 01:49:43 +00:00
Author
Owner
Copy Link
Copy Source

Phase 0 (partial) landed: backend PR #391 merged to master (merge 6ad002e2).

Added an opt-in per-operation timeout to S3ObjectStore (linked CancellationTokenSource), enabled at 120s for the metadata-extractor. A wedged PutObject now surfaces as a bounded TimeoutException instead of hanging the book-upload pipeline indefinitely. Large-file buckets (ai-models, lesson-videos) stay unbounded by design. 38/38 Storage tests green; CI passed.

Still open for Phase 0 (infra/ops — need the infrastructure repo + MinIO admin):

  • Attach storage-touching services to an internal MinIO route and point Storage:ServiceUrl at http://minio:9000 (removes the public minio.spikersoft.com hairpin through Traefik).
  • Verify metadata-svc (and other service creds) can read and write the ebooks bucket; fix bucket policy/IAM (ref #613).

Then Phases 1–4 (write-primary → backfill → serving cutover → decommission). The wedged workflow 86bfade3… remains preserved as a live specimen.

**Phase 0 (partial) landed:** backend PR #391 merged to `master` (merge `6ad002e2`). Added an opt-in per-operation timeout to `S3ObjectStore` (linked `CancellationTokenSource`), enabled at 120s for the metadata-extractor. A wedged `PutObject` now surfaces as a bounded `TimeoutException` instead of hanging the book-upload pipeline indefinitely. Large-file buckets (`ai-models`, `lesson-videos`) stay unbounded by design. 38/38 Storage tests green; CI passed. **Still open for Phase 0 (infra/ops — need the infrastructure repo + MinIO admin):** - Attach storage-touching services to an internal MinIO route and point `Storage:ServiceUrl` at `http://minio:9000` (removes the public `minio.spikersoft.com` hairpin through Traefik). - Verify `metadata-svc` (and other service creds) can read *and* write the `ebooks` bucket; fix bucket policy/IAM (ref #613). Then Phases 1–4 (write-primary → backfill → serving cutover → decommission). The wedged workflow `86bfade3…` remains preserved as a live specimen.
spikerj commented 2026-07-18 18:01:29 +00:00
Author
Owner
Copy Link
Copy Source

Epic #413 accuracy pass (2026-07-18) — Phase 0 status verified against masters; this ticket now owns the whole ebooks slice.

Verified:

  • Timeout half of Phase 0: LANDED (backend #391 on master) — S3ObjectStore per-operation timeout with tests (PutFileAsync_WhenOperationExceedsTimeout_ThrowsTimeoutException etc.); the mirror call in MetadataExtractionService.cs:114/296 is still inline-awaited, but now bounded (120s for the extractor) instead of hangs-forever. Phase 1 (write-primary with real error handling) still restructures this.
  • Internal-route half of Phase 0: NOT started — minio/docker-stack.yml attaches only traefik-public; embeddings + quiz-generation stacks explicitly point Storage__ServiceUrl at https://minio.spikersoft.com, and every other service defaults to the public URL. The hairpin (also #538's subject) remains.
  • Creds verification: open — #613 documents the metadata-svc policy gap pattern (quarantine proven; the ebooks write worked 2026-07-12 evidence, but a full read+write policy audit per key is the Phase 0 line).

Scope consolidation from the accuracy pass: #529's ebooks bind-drop and #531's ebooks serving cutover now explicitly defer to this ticket's phases (comments posted there). nginx already denies /spikersoft/ebooks (books are API-served with BookAccess authz since #618), so Phase 3 here is API/read-seam work, not an nginx-proxy question — and anonymous bucket download on ebooks must never be enabled (would bypass BookAccess).

**Epic #413 accuracy pass (2026-07-18) — Phase 0 status verified against masters; this ticket now owns the whole ebooks slice.** Verified: - **Timeout half of Phase 0: LANDED** (backend #391 on master) — `S3ObjectStore` per-operation timeout with tests (`PutFileAsync_WhenOperationExceedsTimeout_ThrowsTimeoutException` etc.); the mirror call in `MetadataExtractionService.cs:114/296` is still inline-awaited, but now bounded (120s for the extractor) instead of hangs-forever. Phase 1 (write-primary with real error handling) still restructures this. - **Internal-route half of Phase 0: NOT started** — `minio/docker-stack.yml` attaches only `traefik-public`; embeddings + quiz-generation stacks explicitly point `Storage__ServiceUrl` at `https://minio.spikersoft.com`, and every other service defaults to the public URL. The hairpin (also #538's subject) remains. - **Creds verification: open** — #613 documents the `metadata-svc` policy gap pattern (quarantine proven; the ebooks write worked 2026-07-12 evidence, but a full read+write policy audit per key is the Phase 0 line). Scope consolidation from the accuracy pass: #529's ebooks bind-drop and #531's ebooks serving cutover now explicitly defer to this ticket's phases (comments posted there). nginx already denies `/spikersoft/ebooks` (books are API-served with BookAccess authz since #618), so Phase 3 here is API/read-seam work, not an nginx-proxy question — and **anonymous bucket download on `ebooks` must never be enabled** (would bypass BookAccess).
spikerj commented 2026-07-29 15:51:17 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master. Correcting a stale note first: the 2026-07-18 comment saying the internal route was "NOT started" is wrong as of today.

Landed since that comment:

  • minio/docker-stack.yml now joins the dedicated minio external overlay, and roughly 20 stacks set Storage__ServiceUrl=http://minio:9000 — e.g. spikersoft-metadata-extractor/docker-stack.yml:62, spikersoft-upload-coordinator:49, spikersoft-embeddings:34, spikersoft-quiz-generation:49. No minio.spikersoft.com hairpin remains in those stacks, so the #538 NAT-hairpin path is off the table for them.
  • The S3 timeout half landed via backend PR #391.

Still open — the acceptance items:

  1. The dual-write seam is still in place. IEbookArtifactMirror / MetadataExtractor/Services/EbookArtifactMirror.cs still exist, and the mirror is still called inline from MetadataExtractionService.cs:58,64,71. Phases 1–4 of the sole-store cutover have not started.
  2. The filesystem is still authoritative. /mnt/fusionio/spikersoft/ebooks is still bound read-write in the backend, upload-coordinator, metadata-extractor and file-movement stacks (tracked in detail on #529, which is also still at phase 1).
  3. The mc mirror ebooks backfill is unverifiable from git. Only the recipes exist (docs/minio-storage-migration.md:83, docs/uploads-batch-cutover-runbook.md:39); no execution record anywhere. The one data point is #529's phase-0 comment claiming "ebooks 1.6 GiB ✓" from 2026-07-14 — a size note, not a verified object count or checksum. Settling evidence would be mc ls --recursive spiker/ebooks | wc -l compared against the fusionio tree.

The "ebooks must never be anonymous" invariant is holding, and I checked rather than assumed: git grep -rn 'mc anonymous|policy set' origin/master returns zero hits across spikersoft-infrastructure, and no anonymous grant on ebooks exists anywhere. Stating that as a verified absence, since it's the kind of thing that gets added casually during a cutover — worth re-checking after each phase.

Given the backfill is the gate on everything else here, that count/checksum is the next concrete step.

Audited against `origin/master`. **Correcting a stale note first: the 2026-07-18 comment saying the internal route was "NOT started" is wrong as of today.** **Landed since that comment:** - `minio/docker-stack.yml` now joins the dedicated `minio` external overlay, and roughly **20 stacks** set `Storage__ServiceUrl=http://minio:9000` — e.g. `spikersoft-metadata-extractor/docker-stack.yml:62`, `spikersoft-upload-coordinator:49`, `spikersoft-embeddings:34`, `spikersoft-quiz-generation:49`. No `minio.spikersoft.com` hairpin remains in those stacks, so the #538 NAT-hairpin path is off the table for them. - The S3 timeout half landed via backend PR #391. **Still open — the acceptance items:** 1. **The dual-write seam is still in place.** `IEbookArtifactMirror` / `MetadataExtractor/Services/EbookArtifactMirror.cs` still exist, and the mirror is still called inline from `MetadataExtractionService.cs:58,64,71`. Phases 1–4 of the sole-store cutover have not started. 2. **The filesystem is still authoritative.** `/mnt/fusionio/spikersoft/ebooks` is still bound read-write in the backend, upload-coordinator, metadata-extractor and file-movement stacks (tracked in detail on #529, which is also still at phase 1). 3. **The `mc mirror` ebooks backfill is unverifiable from git.** Only the recipes exist (`docs/minio-storage-migration.md:83`, `docs/uploads-batch-cutover-runbook.md:39`); no execution record anywhere. The one data point is #529's phase-0 comment claiming "ebooks 1.6 GiB ✓" from 2026-07-14 — a size note, not a verified object count or checksum. Settling evidence would be `mc ls --recursive spiker/ebooks | wc -l` compared against the fusionio tree. **The "ebooks must never be anonymous" invariant is holding**, and I checked rather than assumed: `git grep -rn 'mc anonymous|policy set' origin/master` returns **zero hits** across spikersoft-infrastructure, and no anonymous grant on `ebooks` exists anywhere. Stating that as a verified absence, since it's the kind of thing that gets added casually during a cutover — worth re-checking after each phase. Given the backfill is the gate on everything else here, that count/checksum is the next concrete step.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#696