spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra][MinIO] file-movement's metadata-svc key can't write the quarantine bucket (Access Denied) — blocks #526/#529 cutover (epic #413) #613

New Issue
Open
opened 2026-07-16 06:25:36 +00:00 by spikerj · 4 comments
spikerj commented 2026-07-16 06:25:36 +00:00
Owner
Copy Link
Copy Source

Parent epic: #413. Surfaced while diagnosing #612.

Problem

The dual-run quarantine mirror throws Access Denied writing to the quarantine bucket — the scoped key file-movement uses (metadata-svc) has no write policy for that bucket.

Evidence (live, Seq, 2026-07-16 06:12:28Z)

FileMovement WARN  Quarantine mirror failed for /app/quarantine/..._advanced-game-narrative-toolbox-2nd.pdf — disk copy remains authoritative
                   Amazon.S3.AmazonS3Exception: Access Denied

Why it matters

It is non-fatal today (best-effort mirror; the disk copy stays authoritative, logged as a Warning), so quarantine still works on the filesystem. But the #529/#526 cutover removes the /app/quarantine bind — at which point quarantine writes have nowhere to go and fail outright. The uploads-batch-cutover-runbook.md (#529) already flagged exactly this:

Note: file-movement reuses metadata-svc today — widen that policy or (better) issue a dedicated movement-svc and update its stack env.

Fix (MinIO policy, no code)

Per the #529 batch table, the key file-movement uses needs rw on: ebooks, quarantine, dlls, assets, blogs. Either:

  • widen the metadata-svc policy to include those buckets, or
  • issue a dedicated movement-svc key with that policy and point spikersoft-file-movement's stack env at it (cleaner least-privilege split).

Then verify: EICAR upload → object appears in spiker/quarantine (the #529 Phase-2 verification step).

Blocks

#526 (quarantine bucket) and the #529 uploads-batch cutover.

Parent epic: #413. Surfaced while diagnosing #612. ## Problem The dual-run **quarantine mirror throws `Access Denied` writing to the `quarantine` bucket** — the scoped key `file-movement` uses (`metadata-svc`) has no write policy for that bucket. ## Evidence (live, Seq, 2026-07-16 06:12:28Z) ``` FileMovement WARN Quarantine mirror failed for /app/quarantine/..._advanced-game-narrative-toolbox-2nd.pdf — disk copy remains authoritative Amazon.S3.AmazonS3Exception: Access Denied ``` ## Why it matters It is **non-fatal today** (best-effort mirror; the disk copy stays authoritative, logged as a Warning), so quarantine still works on the filesystem. But the **#529/#526 cutover removes the `/app/quarantine` bind** — at which point quarantine writes have nowhere to go and fail outright. The `uploads-batch-cutover-runbook.md` (#529) already flagged exactly this: > **Note:** file-movement reuses `metadata-svc` today — widen that policy or (better) issue a dedicated `movement-svc` and update its stack env. ## Fix (MinIO policy, no code) Per the #529 batch table, the key file-movement uses needs **rw on: ebooks, quarantine, dlls, assets, blogs**. Either: - widen the `metadata-svc` policy to include those buckets, or - issue a dedicated `movement-svc` key with that policy and point `spikersoft-file-movement`'s stack env at it (cleaner least-privilege split). Then verify: EICAR upload → object appears in `spiker/quarantine` (the #529 Phase-2 verification step). ## Blocks #526 (quarantine bucket) and the #529 uploads-batch cutover.
spikerj commented 2026-07-18 18:14:51 +00:00
Author
Owner
Copy Link
Copy Source

Fix authored as the runbook-preferred dedicated key — two paired PRs up:

  • infra #126 (GATED): extends openbao/provision-minio-svc-users.sh with movement-svc (movement-svc-storage: rw ebooks/quarantine/dlls/assets/blogs — the exact #529 table row, DeleteObject included for the #499 purge; key → secret/ci/backend/minio/movement), plus two things this ticket's class of bug demanded: an attach-only widening of uploads-svc with ebooks-ro (#598's BookImageLocator would otherwise 403 silently) and a closing policy audit that prints accessKey → attached policies for all 8 svc users, so "does the key actually have the policy?" stops being unverifiable. Also swaps the file-movement stack env to movement-svc/MOVEMENT_S3_SECRET_KEY.
  • backend #415: file-movement workflow fetches secret/ci/backend/minio/movement (guard message points at the script).

Ops sequence (Joey / a SERVER-side session):

  1. merge infra bug: dungeon crawler menu (#126)
  2. on dreamstream1: export BAO_TOKEN=…; cd /mnt/infrastructure/openbao && sudo -E bash provision-minio-svc-users.sh (idempotent; existing users untouched; audit printed at the end)
  3. merge backend #415 → next file-movement deploy picks up the key
  4. verify + close this ticket: EICAR upload → object appears in spiker/quarantine

metadata-svc deliberately keeps its original scope (write ebooks, read uploads/ai-models) for extractor/embeddings/quiz — least privilege preserved.

**Fix authored as the runbook-preferred dedicated key — two paired PRs up:** - **infra #126** (GATED): extends `openbao/provision-minio-svc-users.sh` with `movement-svc` (`movement-svc-storage`: rw ebooks/quarantine/dlls/assets/blogs — the exact #529 table row, DeleteObject included for the #499 purge; key → `secret/ci/backend/minio/movement`), plus two things this ticket's class of bug demanded: an attach-only widening of `uploads-svc` with `ebooks-ro` (#598's BookImageLocator would otherwise 403 silently) and a closing **policy audit** that prints accessKey → attached policies for all 8 svc users, so "does the key actually have the policy?" stops being unverifiable. Also swaps the file-movement stack env to `movement-svc`/`MOVEMENT_S3_SECRET_KEY`. - **backend #415**: file-movement workflow fetches `secret/ci/backend/minio/movement` (guard message points at the script). **Ops sequence (Joey / a SERVER-side session):** 1. merge infra #126 2. on dreamstream1: `export BAO_TOKEN=…; cd /mnt/infrastructure/openbao && sudo -E bash provision-minio-svc-users.sh` (idempotent; existing users untouched; audit printed at the end) 3. merge backend #415 → next file-movement deploy picks up the key 4. **verify + close this ticket:** EICAR upload → object appears in `spiker/quarantine` `metadata-svc` deliberately keeps its original scope (write ebooks, read uploads/ai-models) for extractor/embeddings/quiz — least privilege preserved.
spikerj commented 2026-07-18 18:23:10 +00:00
Author
Owner
Copy Link
Copy Source

2026-07-18 provision run (Joey, dreamstream1) — MinIO side DONE, one recovery step left.

The audit output is the evidence this ticket wanted:

movement-svc: movement-svc-storage          ← created + attached ✓
uploads-svc:  ebooks-ro,uploads-rw          ← #598 widening applied ✓
blogs-svc:    blogs-svc-storage             ← #522 write question SETTLED ✓
metadata-svc: ai-models-ro,metadata-ebooks-rw  ← confirms this ticket's root cause:
                                               no quarantine/dlls/assets/blogs anywhere in its scope

The one failure: the OpenBao write 403'd (permission denied on secret/ci/backend/minio/movement) — the BAO_TOKEN used lacked write on secret/data/ci/backend/minio/*. Since the script never prints secrets and MinIO secret keys are returned exactly once, movement-svc's generated secret is unrecoverable. That's the exact scenario rotate-minio-ci-keys.sh documents; recovery is one command with an admin-capable token:

export BAO_TOKEN=<token with write on secret/ci/backend/minio/*>
cd /mnt/infrastructure/openbao && ONLY_SVCS="movement" sudo -E bash rotate-minio-ci-keys.sh

(the rotate loop is generic — works with the current checkout; infra PR #128 adds movement to its default set)

Then: dispatch/redeploy the file-movement workflow (backend #415 is merged — it now fetches the movement path; until the rotate runs, its guard refuses loudly, which is the designed fail-safe). Close this ticket on: EICAR upload → object in spiker/quarantine.

**2026-07-18 provision run (Joey, dreamstream1) — MinIO side DONE, one recovery step left.** The audit output is the evidence this ticket wanted: ``` movement-svc: movement-svc-storage ← created + attached ✓ uploads-svc: ebooks-ro,uploads-rw ← #598 widening applied ✓ blogs-svc: blogs-svc-storage ← #522 write question SETTLED ✓ metadata-svc: ai-models-ro,metadata-ebooks-rw ← confirms this ticket's root cause: no quarantine/dlls/assets/blogs anywhere in its scope ``` **The one failure:** the OpenBao write 403'd (`permission denied` on `secret/ci/backend/minio/movement`) — the `BAO_TOKEN` used lacked write on `secret/data/ci/backend/minio/*`. Since the script never prints secrets and MinIO secret keys are returned exactly once, **movement-svc's generated secret is unrecoverable**. That's the exact scenario `rotate-minio-ci-keys.sh` documents; recovery is one command with an admin-capable token: ``` export BAO_TOKEN=<token with write on secret/ci/backend/minio/*> cd /mnt/infrastructure/openbao && ONLY_SVCS="movement" sudo -E bash rotate-minio-ci-keys.sh ``` (the rotate loop is generic — works with the current checkout; infra PR #128 adds `movement` to its default set) **Then:** dispatch/redeploy the file-movement workflow (backend #415 is merged — it now fetches the movement path; until the rotate runs, its guard refuses loudly, which is the designed fail-safe). **Close this ticket on:** EICAR upload → object in `spiker/quarantine`.
spikerj commented 2026-07-22 03:05:21 +00:00
Author
Owner
Copy Link
Copy Source

Board-sweep finding (2026-07-22): HALF-SHIPPED — the movement-svc cutover is fully merged (infra stack Storage__AccessKey=movement-svc + the file-movement workflow's MOVEMENT secret plumbing), but the RUNNING service still carries Storage__AccessKey=metadata-svc — the env change never rolled to the swarm. Dispatched the file-movement workflow just now to redeploy properly (Bao-fetched secret); once the service env shows movement-svc and a quarantine write succeeds, this closes. Will follow up after the rollout.

Board-sweep finding (2026-07-22): HALF-SHIPPED — the movement-svc cutover is fully merged (infra stack Storage__AccessKey=movement-svc + the file-movement workflow's MOVEMENT secret plumbing), but the RUNNING service still carries Storage__AccessKey=metadata-svc — the env change never rolled to the swarm. Dispatched the file-movement workflow just now to redeploy properly (Bao-fetched secret); once the service env shows movement-svc and a quarantine write succeeds, this closes. Will follow up after the rollout.
spikerj commented 2026-07-29 16:01:22 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — all the code and provisioning landed; only the one-time key rotation remains. Staying open on that single step.

Committed and complete:

  • spikersoft-file-movement/docker-stack.yml:32-33 — Storage__AccessKey=movement-svc, Storage__SecretKey=${MOVEMENT_S3_SECRET_KEY}. The borrowed metadata-svc key is gone from the stack, and the header at :18-20 documents why (movement-svc gets rw on ebooks/quarantine/dlls/assets/blogs — the buckets metadata-svc's policy could not write).
  • openbao/provision-minio-svc-users.sh:20 provisions movement-svc with exactly that policy. The note at :47-50 records that this resolved the runbook's open decision the way it recommended, and that the stack env swap plus the backend workflow's secret/ci/backend/minio/movement path shipped in the paired PRs.
  • The recovery path is supported: both provision-minio-svc-users.sh:60 and rotate-minio-ci-keys.sh:55 honour ONLY_SVCS, so ONLY_SVCS="movement" ./rotate-minio-ci-keys.sh is a real, scoped command rather than an all-or-nothing rotation.

Remaining — one action, unrecorded: run ONLY_SVCS="movement" rotate-minio-ci-keys.sh with an admin-capable Bao token, so the secret/ci/backend/minio/movement path actually holds a key. Per comment 10415 the state was HALF-SHIPPED — merged stack env, running service still on metadata-svc — and no infra commit since then records the rotation. Git can't prove it either way; the settling evidence is a successful File Movement create_manifest against the quarantine bucket.

This is blocking more than it looks. #529's phase-0 items and #783's follow-up both name this rotation as outstanding, and #529's whole 7-service cutover sits behind it. Given the fix is a single scoped command against tooling that already exists, this is probably the highest ratio of unblocked-work to effort currently on the board.

Audited against `origin/master` — **all the code and provisioning landed; only the one-time key rotation remains.** Staying open on that single step. **Committed and complete:** - `spikersoft-file-movement/docker-stack.yml:32-33` — `Storage__AccessKey=movement-svc`, `Storage__SecretKey=${MOVEMENT_S3_SECRET_KEY}`. The borrowed `metadata-svc` key is gone from the stack, and the header at `:18-20` documents why (`movement-svc` gets rw on ebooks/quarantine/dlls/assets/blogs — the buckets `metadata-svc`'s policy could not write). - `openbao/provision-minio-svc-users.sh:20` provisions `movement-svc` with exactly that policy. The note at `:47-50` records that this resolved the runbook's open decision the way it recommended, and that the stack env swap plus the backend workflow's `secret/ci/backend/minio/movement` path shipped in the paired PRs. - The recovery path is supported: both `provision-minio-svc-users.sh:60` and `rotate-minio-ci-keys.sh:55` honour `ONLY_SVCS`, so `ONLY_SVCS="movement" ./rotate-minio-ci-keys.sh` is a real, scoped command rather than an all-or-nothing rotation. **Remaining — one action, unrecorded:** run `ONLY_SVCS="movement" rotate-minio-ci-keys.sh` with an admin-capable Bao token, so the `secret/ci/backend/minio/movement` path actually holds a key. Per comment 10415 the state was HALF-SHIPPED — merged stack env, running service still on `metadata-svc` — and no infra commit since then records the rotation. Git can't prove it either way; the settling evidence is a successful File Movement `create_manifest` against the quarantine bucket. **This is blocking more than it looks.** #529's phase-0 items and #783's follow-up both name this rotation as outstanding, and #529's whole 7-service cutover sits behind it. Given the fix is a single scoped command against tooling that already exists, this is probably the highest ratio of unblocked-work to effort currently on the board.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#613