spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][MinIO] art-asset-artifacts: the ACTUAL last GridFS bucket → MinIO (ArtStudio stage outputs, ~4.5GB) (epic #413) #783

New Issue
Open
opened 2026-07-22 01:49:51 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-22 01:49:51 +00:00
Owner
Copy Link
Copy Source

Discovered during #778's post-drop sweep: the ArtStudio artifact store (art-asset-artifacts GridFS bucket — stage outputs referenced by ArtAssetArtifactRef.gridFsId: aligned frames, fused composites, touchup bases, 16-bit TIFFs) was never on the GridFS holdout list. 74 files, ~4.5 GB, actively read/written by the artpipe workers and served to Photo Stack / Touch-Up via the artifact endpoints.

Same recipe as #771/#773/#778, with two wrinkles:

  1. Writers are the artpipe WORKERS (find every GridFSBucket("art-asset-artifacts") user — likely ArtPipeProcessor and the ArtStudio domain services) — each host needs the S3 impl + creds/bucket grant (a new artpipe-svc or extend an existing svc user).
  2. 4.5 GB migration — the generic tools/MigrateGridFsBucketToMinio handles it (SHA-256 read-back per object), but run it with time budget and consider Storage:OperationTimeoutSeconds guidance for large TIFFs (unbounded per the #696 doctrine).

After this one: Mongo GridFS is genuinely empty.

Discovered during #778's post-drop sweep: the ArtStudio artifact store (`art-asset-artifacts` GridFS bucket — stage outputs referenced by `ArtAssetArtifactRef.gridFsId`: aligned frames, fused composites, touchup bases, 16-bit TIFFs) was never on the GridFS holdout list. 74 files, ~4.5 GB, actively read/written by the artpipe workers and served to Photo Stack / Touch-Up via the artifact endpoints. Same recipe as #771/#773/#778, with two wrinkles: 1. Writers are the artpipe WORKERS (find every GridFSBucket("art-asset-artifacts") user — likely ArtPipeProcessor and the ArtStudio domain services) — each host needs the S3 impl + creds/bucket grant (a new artpipe-svc or extend an existing svc user). 2. 4.5 GB migration — the generic tools/MigrateGridFsBucketToMinio handles it (SHA-256 read-back per object), but run it with time budget and consider Storage:OperationTimeoutSeconds guidance for large TIFFs (unbounded per the #696 doctrine). After this one: Mongo GridFS is genuinely empty.
spikerj commented 2026-07-22 02:10:08 +00:00
Author
Owner
Copy Link
Copy Source

Data migration COMPLETE: 74/74 artifacts (~4.5GB incl. the 1.1GB TIFFs) copied to MinIO art-asset-artifacts with per-object SHA-256 read-back verification, 0 failures, temp-file streaming (nothing held in memory). Both PRs up: spikersoft-backend #451 (seams + both hosts + CI secret plumbing + migration tool) and spikersoft-infrastructure #138 (buckets+policies live-applied, 7 artpipe stacks wired). Order-independence held: current prod (GridFS path) unaffected; after merge+deploy the S3 path finds every legacy id. Remaining: merge both → artpipe deploy rollout → verify a Photo Stack artifact load + Touch-Up frame strip → archive+drop art-asset-artifacts.* GridFS (operator go) → collection-sweep proof that Mongo GridFS is empty.

Data migration COMPLETE: 74/74 artifacts (~4.5GB incl. the 1.1GB TIFFs) copied to MinIO art-asset-artifacts with per-object SHA-256 read-back verification, 0 failures, temp-file streaming (nothing held in memory). Both PRs up: spikersoft-backend #451 (seams + both hosts + CI secret plumbing + migration tool) and spikersoft-infrastructure #138 (buckets+policies live-applied, 7 artpipe stacks wired). Order-independence held: current prod (GridFS path) unaffected; after merge+deploy the S3 path finds every legacy id. Remaining: merge both → artpipe deploy rollout → verify a Photo Stack artifact load + Touch-Up frame strip → archive+drop art-asset-artifacts.* GridFS (operator go) → collection-sweep proof that Mongo GridFS is empty.
spikerj commented 2026-07-22 03:51:12 +00:00
Author
Owner
Copy Link
Copy Source

Rollout verification (2026-07-22 ~03:50): checked every workflow run on the #451 merge commit — all deploy legs green (API + the artpipe-processor tiers included), with one exception that is NOT part of this rollout: File Movement create_manifest fails fetching secret/ci/backend/minio/movement from Bao (no value at path) — that's #613's known unrecoverable movement-svc secret (the 07-18 provision run 403'd on the Bao write; recovery is ONLY_SVCS="movement" rotate-minio-ci-keys.sh on dreamstream1 with an admin-capable token). The 03:05 workflow re-dispatch hit the same wall — the rotate has to happen first.

Also: the serving-side follow-up this ticket's migration unlocked is now filed and implemented as #784 (presigned artifact URLs + caching — backend PR #452 merged, angular PR #518 in CI).

Remaining here per the earlier comment (operator steps): prod verify Photo Stack / Touch-Up loads → archive+drop art-asset-artifacts.* GridFS → empty-GridFS sweep proof.

Rollout verification (2026-07-22 ~03:50): checked every workflow run on the #451 merge commit — **all deploy legs green** (API + the artpipe-processor tiers included), with one exception that is NOT part of this rollout: **File Movement** `create_manifest` fails fetching `secret/ci/backend/minio/movement` from Bao (`no value at path`) — that's #613's known unrecoverable movement-svc secret (the 07-18 provision run 403'd on the Bao write; recovery is `ONLY_SVCS="movement" rotate-minio-ci-keys.sh` on dreamstream1 with an admin-capable token). The 03:05 workflow re-dispatch hit the same wall — the rotate has to happen first. Also: the serving-side follow-up this ticket's migration unlocked is now filed and implemented as **#784** (presigned artifact URLs + caching — backend PR #452 merged, angular PR #518 in CI). Remaining here per the earlier comment (operator steps): prod verify Photo Stack / Touch-Up loads → archive+drop `art-asset-artifacts.*` GridFS → empty-GridFS sweep proof.
spikerj commented 2026-07-29 15:51:58 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — code, infra and the data move are all done and recorded; three operator steps remain. Staying open on those.

Done:

  • SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs:735-742 — keyed art-asset-artifacts S3 store + S3ArtAssetArtifactService when UseS3; same on the worker host at ArtPipeProcessor/Program.cs:40-45.
  • tools/MigrateArtAssetArtifactsToMinio/Program.cs (backend PR #451).
  • Infra PR #138 (85c430e) — buckets/policies plus S3 env and minio network on 7 artpipe stacks.
  • The migration itself is recorded as executed (comment 10177): 74/74 objects, ~4.5 GB, per-object SHA-256 read-back, 0 failures. Comment 10448 records all deploy legs green.

One thing worth stating so it isn't mistaken for incompleteness: GridFsArtAssetArtifactService.cs is still on master, but that's the by-design local-dev path taken when UseS3 is off — see the else-branch at :742. It is not residue.

Remaining — the three operator steps this ticket's own comments list, with nothing recorded since 2026-07-22:

  1. Prod verification of a Photo Stack artifact load + the Touch-Up frame strip.
  2. Archive and drop the art-asset-artifacts.* GridFS collections (needs operator go-ahead).
  3. A collection-sweep proving Mongo GridFS is actually empty. This ticket's closing claim — "after this one, Mongo GridFS is genuinely empty" — is unproven until that sweep runs, and it's the claim the whole #413 epic leans on.

Also from comment 10448, explicitly not part of this rollout but still blocking elsewhere: File Movement create_manifest still fails on secret/ci/backend/minio/movement (#613). The recovery is ONLY_SVCS="movement" rotate-minio-ci-keys.sh with an admin-capable Bao token, and no infra commit since then shows it having been run.

Audited against `origin/master` — **code, infra and the data move are all done and recorded; three operator steps remain.** Staying open on those. **Done:** - `SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs:735-742` — keyed `art-asset-artifacts` S3 store + `S3ArtAssetArtifactService` when `UseS3`; same on the worker host at `ArtPipeProcessor/Program.cs:40-45`. - `tools/MigrateArtAssetArtifactsToMinio/Program.cs` (backend PR #451). - Infra PR #138 (`85c430e`) — buckets/policies plus S3 env and minio network on 7 artpipe stacks. - **The migration itself is recorded as executed** (comment 10177): 74/74 objects, ~4.5 GB, per-object SHA-256 read-back, 0 failures. Comment 10448 records all deploy legs green. One thing worth stating so it isn't mistaken for incompleteness: `GridFsArtAssetArtifactService.cs` is still on master, but that's the by-design local-dev path taken when `UseS3` is off — see the else-branch at `:742`. It is not residue. **Remaining — the three operator steps this ticket's own comments list, with nothing recorded since 2026-07-22:** 1. Prod verification of a Photo Stack artifact load + the Touch-Up frame strip. 2. Archive and drop the `art-asset-artifacts.*` GridFS collections (needs operator go-ahead). 3. A collection-sweep proving Mongo GridFS is actually empty. **This ticket's closing claim — "after this one, Mongo GridFS is genuinely empty" — is unproven until that sweep runs**, and it's the claim the whole #413 epic leans on. Also from comment 10448, explicitly *not* part of this rollout but still blocking elsewhere: File Movement `create_manifest` still fails on `secret/ci/backend/minio/movement` (#613). The recovery is `ONLY_SVCS="movement" rotate-minio-ci-keys.sh` with an admin-capable Bao token, and no infra commit since then shows it having been run.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#783