spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][MinIO] Final GridFS slices: vault-files + photograph-previews → MinIO sole store (completes epic #413's GridFS retirement) #778

New Issue
Closed
opened 2026-07-21 20:08:16 +00:00 by spikerj · 4 comments
spikerj commented 2026-07-21 20:08:16 +00:00
Owner
Copy Link
Copy Source

The last two GridFS holdouts, same recipe as #771/#773:

vault-files (IGridFsVaultFileService — info-vault uploads): bucket is EMPTY in prod (0 files) → pure code swap, no data migration.
photograph-previews (IGridFsPhotographPreviewService — #718 gallery preview/thumbnail derivatives): 22 files / 11.2MB. Written by BOTH the API and the PhotographProcessor worker (which already runs Storage__UseS3=true with photos-svc creds), so the S3 impl registers in both hosts.

Plan:

  1. Rename seams (IVaultFileStorageService / IPhotographPreviewStorageService), S3 impls in SpikerSoft.Data (shared base with owner metadata; GridFS impls stay as local-dev fallback), new Data→Storage project ref.
  2. Buckets vault-files + photograph-previews; api-svc rw both, photos-svc rw photograph-previews (live + provisioning-script parity).
  3. Generic migration tool (parameterized GridFS→S3 bucket copy w/ SHA-256 read-back) run for photograph-previews.
  4. Tests for the S3 stores; full suites green.
  5. After deploy + gallery spot-check: drop both GridFS collections (previews archived first; vault has nothing to archive).

Completes the GridFS retirement — after this, Mongo GridFS holds nothing and epic #413's object-storage story covers all binary media.

The last two GridFS holdouts, same recipe as #771/#773: **vault-files** (IGridFsVaultFileService — info-vault uploads): bucket is EMPTY in prod (0 files) → pure code swap, no data migration. **photograph-previews** (IGridFsPhotographPreviewService — #718 gallery preview/thumbnail derivatives): 22 files / 11.2MB. Written by BOTH the API and the PhotographProcessor worker (which already runs Storage__UseS3=true with photos-svc creds), so the S3 impl registers in both hosts. Plan: 1. Rename seams (IVaultFileStorageService / IPhotographPreviewStorageService), S3 impls in SpikerSoft.Data (shared base with owner metadata; GridFS impls stay as local-dev fallback), new Data→Storage project ref. 2. Buckets vault-files + photograph-previews; api-svc rw both, photos-svc rw photograph-previews (live + provisioning-script parity). 3. Generic migration tool (parameterized GridFS→S3 bucket copy w/ SHA-256 read-back) run for photograph-previews. 4. Tests for the S3 stores; full suites green. 5. After deploy + gallery spot-check: drop both GridFS collections (previews archived first; vault has nothing to archive). Completes the GridFS retirement — after this, Mongo GridFS holds nothing and epic #413's object-storage story covers all binary media.
spikerj commented 2026-07-21 20:14:51 +00:00
Author
Owner
Copy Link
Copy Source

Implementation up: spikersoft-backend PR #445 + spikersoft-infrastructure PR #136. Ops executed: buckets created, api-svc rw both + photos-svc rw previews (all verified with each service's own creds), photograph-previews 22/22 migrated with SHA-256 read-back (vault-files empty — code swap only). GridFS previews pre-archived to ~/backups/gridfs-photograph-previews-20260721. After merge+deploy: gallery thumbnail spot-check → drop both GridFS collections → Mongo GridFS is empty and the epic's GridFS retirement is complete.

Implementation up: spikersoft-backend PR #445 + spikersoft-infrastructure PR #136. Ops executed: buckets created, api-svc rw both + photos-svc rw previews (all verified with each service's own creds), photograph-previews 22/22 migrated with SHA-256 read-back (vault-files empty — code swap only). GridFS previews pre-archived to ~/backups/gridfs-photograph-previews-20260721. After merge+deploy: gallery thumbnail spot-check → drop both GridFS collections → Mongo GridFS is empty and the epic's GridFS retirement is complete.
spikerj commented 2026-07-21 20:49:21 +00:00
Author
Owner
Copy Link
Copy Source

Deploy of #445 verified CONVERGED (UpdateStatus=completed, task stable 50+ min) — note: two transient exit-139 (segfault) task failures during rollout before stabilizing; watching for recurrence on future deploys. API healthy (healthz 200, avatar-from-MinIO 200, 0 errors last 10m). Awaiting operator gallery-thumbnail spot-check, then the final GridFS drop.

Deploy of #445 verified CONVERGED (UpdateStatus=completed, task stable 50+ min) — note: two transient exit-139 (segfault) task failures during rollout before stabilizing; watching for recurrence on future deploys. API healthy (healthz 200, avatar-from-MinIO 200, 0 errors last 10m). Awaiting operator gallery-thumbnail spot-check, then the final GridFS drop.
spikerj commented 2026-07-22 01:49:51 +00:00
Author
Owner
Copy Link
Copy Source

Operator-approved final drop executed: photograph-previews.files/.chunks and vault-files.files/.chunks are gone (previews archived at ~/backups/gridfs-photograph-previews-20260721; vault was empty). The photography/vault/sensitive-docs/profile-images GridFS retirement is COMPLETE. Honest discovery during the post-drop sweep: ONE more GridFS bucket exists that was never on the holdout list — art-asset-artifacts (ArtStudio stage outputs: aligned frames, fused TIFFs, touchup bases; 74 files, ~4.5 GB, actively read by Photo Stack/Touch-Up). Filed as the true final slice: #782. Closing this ticket as its scoped slices are done.

Operator-approved final drop executed: photograph-previews.files/.chunks and vault-files.files/.chunks are gone (previews archived at ~/backups/gridfs-photograph-previews-20260721; vault was empty). The photography/vault/sensitive-docs/profile-images GridFS retirement is COMPLETE. Honest discovery during the post-drop sweep: ONE more GridFS bucket exists that was never on the holdout list — art-asset-artifacts (ArtStudio stage outputs: aligned frames, fused TIFFs, touchup bases; 74 files, ~4.5 GB, actively read by Photo Stack/Touch-Up). Filed as the true final slice: #782. Closing this ticket as its scoped slices are done.
spikerj commented 2026-07-22 01:49:59 +00:00
Author
Owner
Copy Link
Copy Source

Correction: the art-asset-artifacts slice ticket is #783 (not #782).

Correction: the art-asset-artifacts slice ticket is #783 (not #782).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#778