[Dev][MinIO] Mac local stack has no MinIO — local dev exercises the disk-only path prod no longer uses (epic #413) #908

New Issue

2026-08-04T13:37:26Z

spikerj commented

2026-08-04 13:37:26 +00:00

Problem

Every real environment sets Storage__UseS3=true — 86 docker-stack.yml files in spikersoft-infrastructure, including the API's own. The Mac local stack sets it nowhere, and Storage:UseS3 is read as configuration.GetValue("Storage:UseS3", false), so local dev silently runs the legacy disk-only storage path that production no longer uses.

Confirmed empirically: a freshly brought-up local API logs zero Storage/S3/MinIO/objectstore lines, and there is no minio service anywhere in docker-compose*.yml — not even for Windows dev.

This violates the "practice how you play" rule in .claude/rules/secrets-in-openbao.md: local dev is supposed to run the same consumption path with obviously-bogus credentials, not a bypass.

Secondary footgun

ObjectStorageServiceCollectionExtensions.AddS3Client defaults Storage:ServiceUrl to https://minio.spikersoft.com — production. So the obvious "just turn the flag on locally" move aims a laptop at the real cluster. It fails on credentials rather than silently writing, but it is one env var away from being wrong.

Fix

Add MinIO to docker-compose.mac.yml and turn Storage__UseS3 on for both sides of the host/container boundary:

minio (S3 :9000, console :9001) + one-shot minio-init that creates all 17 buckets any service resolves via AddS3ObjectStore / AddKeyedS3ObjectStore.
Native API → http://localhost:9000 (via scripts/mac-dev-up.sh); pipeline containers → http://minio:9000 (via a shared YAML anchor).
mac-dev-up.sh gates bring-up on minio-init exit code, so a provisioning failure stops the script instead of surfacing later as a NoSuchBucket mid-upload.
Single bogus root credential (localdev-minio) rather than prod's per-service accounts — replicating that IAM on a laptop is ceremony, not fidelity; what's exercised is the S3 code path, not the authorization matrix.

Deliberately excluded

book-management — not an S3 service (no Storage__UseS3 in its stack file).
Native quiz/embeddings workers — they resolve models from ai-models in prod, but locally AI__*Path points at multi-GB host GGUFs from mac-dev-models.sh.

Blast radius

Storage:UseS3 is a single global flag, so this also moves GeoIP, 3D-model assets, profile images and media delivery onto buckets. Safe because #413 is still dual-run: disk stays authoritative, S3 mirrors on write and only answers what disk misses (S3MediaFallbackMiddleware sits behind the static-file mounts; S3ThreeDModelAssetStore.TryOpenAsync returns null rather than throwing). Empty buckets degrade rather than break.

Verified: API startup stays at 0 WRN / 0 ERR with the flag on, and a disk-miss → S3 read returns 200 with the right bytes through /blog-pictures.

Also fixed here

mac-dev-up.sh's art_pipe preflight die is gated on --fake-executor alone, not --no-worker — so --no-worker by itself aborts before compose up on any machine without an art_pipe sibling. Documented in the runbook.

## Problem Every real environment sets `Storage__UseS3=true` — 86 `docker-stack.yml` files in `spikersoft-infrastructure`, including the API's own. The Mac local stack sets it nowhere, and `Storage:UseS3` is read as `configuration.GetValue("Storage:UseS3", false)`, so local dev silently runs the **legacy disk-only storage path that production no longer uses**. Confirmed empirically: a freshly brought-up local API logs zero `Storage`/S3/MinIO/objectstore lines, and there is no `minio` service anywhere in `docker-compose*.yml` — not even for Windows dev. This violates the "practice how you play" rule in `.claude/rules/secrets-in-openbao.md`: local dev is supposed to run the same consumption path with obviously-bogus credentials, not a bypass. ### Secondary footgun `ObjectStorageServiceCollectionExtensions.AddS3Client` defaults `Storage:ServiceUrl` to `https://minio.spikersoft.com` — **production**. So the obvious "just turn the flag on locally" move aims a laptop at the real cluster. It fails on credentials rather than silently writing, but it is one env var away from being wrong. ## Fix Add MinIO to `docker-compose.mac.yml` and turn `Storage__UseS3` on for both sides of the host/container boundary: - `minio` (S3 :9000, console :9001) + one-shot `minio-init` that creates all 17 buckets any service resolves via `AddS3ObjectStore` / `AddKeyedS3ObjectStore`. - Native API → `http://localhost:9000` (via `scripts/mac-dev-up.sh`); pipeline containers → `http://minio:9000` (via a shared YAML anchor). - `mac-dev-up.sh` gates bring-up on `minio-init` **exit code**, so a provisioning failure stops the script instead of surfacing later as a `NoSuchBucket` mid-upload. - Single bogus root credential (`localdev-minio`) rather than prod's per-service accounts — replicating that IAM on a laptop is ceremony, not fidelity; what's exercised is the S3 code path, not the authorization matrix. ### Deliberately excluded - `book-management` — not an S3 service (no `Storage__UseS3` in its stack file). - Native quiz/embeddings workers — they resolve models from `ai-models` in prod, but locally `AI__*Path` points at multi-GB host GGUFs from `mac-dev-models.sh`. ## Blast radius `Storage:UseS3` is a single global flag, so this also moves GeoIP, 3D-model assets, profile images and media delivery onto buckets. Safe because #413 is still **dual-run**: disk stays authoritative, S3 mirrors on write and only answers what disk misses (`S3MediaFallbackMiddleware` sits behind the static-file mounts; `S3ThreeDModelAssetStore.TryOpenAsync` returns null rather than throwing). Empty buckets degrade rather than break. Verified: API startup stays at **0 WRN / 0 ERR** with the flag on, and a disk-miss → S3 read returns 200 with the right bytes through `/blog-pictures`. ## Also fixed here `mac-dev-up.sh`'s art_pipe preflight `die` is gated on `--fake-executor` **alone**, not `--no-worker` — so `--no-worker` by itself aborts before `compose up` on any machine without an `art_pipe` sibling. Documented in the runbook.

spikerj commented

2026-08-04 13:52:15 +00:00

Resolved in spikersoft-backend PR #514 (merged to master as 5f7de302).

The Mac local stack now runs its own MinIO with Storage__UseS3=true on both sides of the host/container boundary — native API at http://localhost:9000, pipeline containers at http://minio:9000 via a shared compose anchor. minio-init provisions all 17 buckets and bring-up gates on its exit code.

Verified: API startup holds at 0 WRN / 0 ERR, and a disk-miss read through /blog-pictures returns 200 from the bucket — which also confirms the S3 client constructs with no region set against plain-HTTP path-style, matching prod.

Closing.

Resolved in spikersoft-backend PR #514 (merged to `master` as 5f7de302). The Mac local stack now runs its own MinIO with `Storage__UseS3=true` on both sides of the host/container boundary — native API at `http://localhost:9000`, pipeline containers at `http://minio:9000` via a shared compose anchor. `minio-init` provisions all 17 buckets and bring-up gates on its exit code. Verified: API startup holds at 0 WRN / 0 ERR, and a disk-miss read through `/blog-pictures` returns 200 from the bucket — which also confirms the S3 client constructs with no region set against plain-HTTP path-style, matching prod. Closing.

spikerj closed this issue

2026-08-04 13:52:18 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#908