Art Studio MinIO migration: reader cutover preceded writer cutover → gap-window artifacts orphaned (500 on download) #825

New Issue

2026-07-24T04:50:00Z

spikerj commented

2026-07-24 04:50:00 +00:00

Summary

The ProArt Art Studio GridFS→MinIO migration (#783, epic #413) cut over the reader to MinIO ~5.5 hours before the art-pipeline writer. Every artifact generated in that window was written to GridFS but read from MinIO → HTTP 500 on download/display (FileNotFoundException: File <id> not found in bucket art-asset-artifacts).

Timeline (2026-07-23/24 UTC)

20:31 spikersoft-backend (API / download reader) redeployed with Storage__UseS3=true → download reads MinIO.
20:50 backfill migrate-783 (MigrateArtAssetArtifactsToMinio) ran — copied the then-existing GridFS artifacts.
00:51–01:26 art asset 6a62b6fdd453b392fdff4f13 ("Pawn : Chess") generated → artifacts written to GridFS (writer still on GridFS), e.g. blender_export_1de22c8e.glb (6a62bf32257e389508ba0be4) at 01:26.
02:06 spikersoft-artpipe-modeling (writer) redeployed with Storage__UseS3=true → writes now go to MinIO.
03:37 user download → API reads MinIO → object absent → 500.

Root cause

Read cutover preceded write cutover. During the gap the two halves disagreed on the backing store. Additionally the backfill ran before the writers flipped, so gap-window artifacts were never a candidate for that pass.

Impact

Any art asset whose stages ran between the reader cutover (20:31) and each writer's cutover (artpipe-modeling 02:06; verify other stage services) is undownloadable until backfilled. Confirmed at least the "Pawn : Chess" asset; a full sweep is being run.

Remediation done (2026-07-24)

Re-ran the idempotent MigrateArtAssetArtifactsToMinio backfill: 108 migrated, 44 already verified, 0 FAILED of 152 GridFS files. The chess-set .glb and all gap-window artifacts are now in MinIO; download path resolves.

Prevention (the actual ask)

Order every remaining store cutover writer-first, reader-last — flip all writer services to Storage:UseS3=true, confirm, THEN flip the reader. Never the reverse.
Run the backfill AFTER all writers have flipped (and make a final backfill pass part of the cutover checklist), so no gap-window objects are missed.
Add a post-cutover verification sweep to the runbook: cross-check referenced artifact keys against the target bucket; alert on any miss.
Applies to the remaining #413 service/bucket cutovers (quarantine bucket, other stage services, other domains still on GridFS).

## Summary The ProArt Art Studio GridFS→MinIO migration (#783, epic #413) cut over the **reader** to MinIO ~5.5 hours **before** the art-pipeline **writer**. Every artifact generated in that window was written to GridFS but read from MinIO → **HTTP 500 on download/display** (`FileNotFoundException: File <id> not found in bucket art-asset-artifacts`). ## Timeline (2026-07-23/24 UTC) - **20:31** `spikersoft-backend` (API / download reader) redeployed with `Storage__UseS3=true` → download reads MinIO. - **20:50** backfill `migrate-783` (MigrateArtAssetArtifactsToMinio) ran — copied the then-existing GridFS artifacts. - **00:51–01:26** art asset `6a62b6fdd453b392fdff4f13` ("Pawn : Chess") generated → artifacts written to **GridFS** (writer still on GridFS), e.g. `blender_export_1de22c8e.glb` (`6a62bf32257e389508ba0be4`) at 01:26. - **02:06** `spikersoft-artpipe-modeling` (writer) redeployed with `Storage__UseS3=true` → writes now go to MinIO. - **03:37** user download → API reads MinIO → object absent → 500. ## Root cause Read cutover preceded write cutover. During the gap the two halves disagreed on the backing store. Additionally the backfill ran *before* the writers flipped, so gap-window artifacts were never a candidate for that pass. ## Impact Any art asset whose stages ran between the reader cutover (20:31) and each writer's cutover (artpipe-modeling 02:06; verify other stage services) is undownloadable until backfilled. Confirmed at least the "Pawn : Chess" asset; a full sweep is being run. ## Remediation done (2026-07-24) Re-ran the idempotent `MigrateArtAssetArtifactsToMinio` backfill: **108 migrated, 44 already verified, 0 FAILED of 152** GridFS files. The chess-set `.glb` and all gap-window artifacts are now in MinIO; download path resolves. ## Prevention (the actual ask) 1. **Order every remaining store cutover writer-first, reader-last** — flip all writer services to `Storage:UseS3=true`, confirm, THEN flip the reader. Never the reverse. 2. **Run the backfill AFTER all writers have flipped** (and make a final backfill pass part of the cutover checklist), so no gap-window objects are missed. 3. **Add a post-cutover verification sweep** to the runbook: cross-check referenced artifact keys against the target bucket; alert on any miss. 4. Applies to the remaining #413 service/bucket cutovers (quarantine bucket, other stage services, other domains still on GridFS).

spikerj commented

2026-07-29 16:03:52 +00:00

Audited against origin/master — the incident was remediated; none of the four prevention items landed. Staying open, and the prevention half is the part that matters.

The remediation is recorded and complete (108 migrated, 44 already verified, 0 failed of 152). But this ticket says the prevention items are "the actual ask", and:

Item 1 — writer-first, reader-last ordering: not documented anywhere. git grep -niE "writer.first|writer first|reader.last|writers have flipped" across docs/*.md → zero hits.
Item 3 — post-cutover verification sweep: absent. git grep -niE "verification sweep|cross-check|post-cutover verif" across docs/*.md → zero hits.
No runbook mentions this ticket or the gap-window lesson at all — git grep -n "825\|gap-window\|gap window" origin/master -- 'docs/*.md' returns only unrelated OIDC callback URLs.
Item 2 is partially covered by accident: docs/uploads-batch-cutover-runbook.md:55 makes "Final delta backfill" step 1 of Phase 2, which does put a backfill after dual-run. But Phase 2 (:53-67) never states the writer/reader ordering constraint — it lists bind removal and placement relaxation, then "deploy the seven stacks". Nothing tells the operator that flipping the reader first is the thing that caused a production outage.

Why this is worth doing rather than closing on the remediation. The same failure mode recurred two days later as #856: the #853 objectKey rename deployed to the reader but not the artpipe writer, producing GET /api/artstudio 500s in production. Different migration, identical shape — reader ahead of writer. That's two incidents from one unwritten rule.

And it's about to matter again. #529 has seven services still to cut over, #696 has ebooks, and both have live backfill steps. Right now the uploads runbook would let an operator repeat this exactly.

Remaining: add the ordering constraint and the verification sweep to docs/uploads-batch-cutover-runbook.md and docs/minio-storage-migration.md, with a pointer back to this ticket and #856 as the two worked examples. That's a documentation change measured in minutes against a failure mode that has already cost two production incidents.

Audited against `origin/master` — **the incident was remediated; none of the four prevention items landed.** Staying open, and the prevention half is the part that matters. The remediation is recorded and complete (108 migrated, 44 already verified, 0 failed of 152). But this ticket says the prevention items are "the actual ask", and: - **Item 1 — writer-first, reader-last ordering: not documented anywhere.** `git grep -niE "writer.first|writer first|reader.last|writers have flipped"` across `docs/*.md` → **zero hits**. - **Item 3 — post-cutover verification sweep: absent.** `git grep -niE "verification sweep|cross-check|post-cutover verif"` across `docs/*.md` → **zero hits**. - **No runbook mentions this ticket or the gap-window lesson at all** — `git grep -n "825\|gap-window\|gap window" origin/master -- 'docs/*.md'` returns only unrelated OIDC callback URLs. - **Item 2 is partially covered by accident**: `docs/uploads-batch-cutover-runbook.md:55` makes "Final delta backfill" step 1 of Phase 2, which does put a backfill after dual-run. But Phase 2 (`:53-67`) never states the writer/reader ordering constraint — it lists bind removal and placement relaxation, then "deploy the seven stacks". Nothing tells the operator that flipping the reader first is the thing that caused a production outage. **Why this is worth doing rather than closing on the remediation.** The same failure mode recurred two days later as **#856**: the #853 `objectKey` rename deployed to the reader but not the artpipe writer, producing `GET /api/artstudio` 500s in production. Different migration, identical shape — reader ahead of writer. That's two incidents from one unwritten rule. And it's about to matter again. **#529** has seven services still to cut over, **#696** has ebooks, and both have live backfill steps. Right now the uploads runbook would let an operator repeat this exactly. **Remaining:** add the ordering constraint and the verification sweep to `docs/uploads-batch-cutover-runbook.md` and `docs/minio-storage-migration.md`, with a pointer back to this ticket and #856 as the two worked examples. That's a documentation change measured in minutes against a failure mode that has already cost two production incidents.

spikerj referenced this issue

2026-07-29 16:11:42 +00:00

ArtStudio/photo-gallery downloads 404: artpipe workers write GridFS while API reads MinIO (stale pre-#783 deploy) #820

spikerj referenced this issue

2026-07-29 17:14:14 +00:00

[Infra][Architecture] Provider-agnostic object storage (MinIO -> S3/Azure Blob) to replace GlusterFS/host-path model #413

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#825