spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

ArtStudio/photo-gallery downloads 404: artpipe workers write GridFS while API reads MinIO (stale pre-#783 deploy) #820

New Issue
Open
opened 2026-07-23 17:34:40 +00:00 by spikerj · 1 comment
spikerj commented 2026-07-23 17:34:40 +00:00
Owner
Copy Link
Copy Source

Summary

Art Studio / photo-gallery artifact downloads 404 for every newly generated asset: the API serves artifacts exclusively from MinIO (bucket art-asset-artifacts, Storage__UseS3=true), but the running artpipe-* pipeline workers still write generated artifacts to GridFS. Split-brain storage → every download of a fresh artifact fails.

Evidence (live prod, 2026-07-23)

  • API (spikersoft-backend, redeployed 15:04 UTC, Storage__UseS3=true):
    System.IO.FileNotFoundException: File 6a624d9e3f451a3e2c09ef4c not found in bucket art-asset-artifacts
    from DownloadArtifactQueryHandler → ArtStudioController.DownloadArtifact.
  • Writer (spikersoft-artpipe-photostack, asset 6a624d3aa63b0a42fd5581b8, 17:21:35 UTC):
    Uploaded art asset artifact aligned_003.jpg (align_fuse/image-jpeg) to **GridFS** with id 6a624d9f...
  • Asset delete swept 0 objects from art-asset-artifacts (bytes were never in MinIO).

Root cause

The running artpipe-photostack and artpipe-modeling services carry a stale, pre-#783 stack spec: their container env has no Storage__* keys (so UseS3Storage() defaults to GridFS) and they are not attached to the minio overlay network. The Storage__UseS3=true env was added to the infra stack files in #783 (commit 85c430e, 2026-07-22 01:56 UTC), but the ArtPipeProcessor deploy job only triggers on spikersoft-backend pushes — the #783 change landed in spikersoft-infrastructure, so the deploy job never re-ran to pick it up. (A #584-class stale-deploy.)

Fix

Redeploy the six artpipe stacks so they pick up the MinIO env + minio network (triggered via ArtPipeProcessor workflow_dispatch).

Follow-ups

  1. Orphan-asset window: any asset generated between the API's MinIO cutover and this redeploy has bytes stranded in GridFS (refs in Mongo, API reads MinIO → 404). The #783 migration covered only pre-cutover GridFS objects. Need a catch-up GridFS→MinIO migration for that window, or regenerate affected assets. Bound the window by when the API env flipped.
  2. Deploy-trigger gap: infra-repo stack changes don't trigger the app-repo deploy jobs that consume them — the exact reason this regressed. Consider an infra-repo path trigger or a redeploy hook.
  3. Dedicated artpipe-svc MinIO user (currently reuses photos-svc).
## Summary Art Studio / photo-gallery artifact downloads 404 for every **newly generated** asset: the API serves artifacts exclusively from MinIO (bucket `art-asset-artifacts`, `Storage__UseS3=true`), but the running `artpipe-*` pipeline workers still write generated artifacts to **GridFS**. Split-brain storage → every download of a fresh artifact fails. ## Evidence (live prod, 2026-07-23) - API (`spikersoft-backend`, redeployed 15:04 UTC, `Storage__UseS3=true`): `System.IO.FileNotFoundException: File 6a624d9e3f451a3e2c09ef4c not found in bucket art-asset-artifacts` from `DownloadArtifactQueryHandler` → `ArtStudioController.DownloadArtifact`. - Writer (`spikersoft-artpipe-photostack`, asset `6a624d3aa63b0a42fd5581b8`, 17:21:35 UTC): `Uploaded art asset artifact aligned_003.jpg (align_fuse/image-jpeg) to **GridFS** with id 6a624d9f...` - Asset delete swept **0** objects from `art-asset-artifacts` (bytes were never in MinIO). ## Root cause The running `artpipe-photostack` and `artpipe-modeling` services carry a **stale, pre-#783 stack spec**: their container env has **no `Storage__*` keys** (so `UseS3Storage()` defaults to GridFS) and they are **not attached to the `minio` overlay network**. The `Storage__UseS3=true` env was added to the infra stack files in #783 (commit `85c430e`, 2026-07-22 01:56 UTC), but the `ArtPipeProcessor` deploy job only triggers on **spikersoft-backend** pushes — the #783 change landed in **spikersoft-infrastructure**, so the deploy job never re-ran to pick it up. (A #584-class stale-deploy.) ## Fix Redeploy the six artpipe stacks so they pick up the MinIO env + `minio` network (triggered via `ArtPipeProcessor` workflow_dispatch). ## Follow-ups 1. **Orphan-asset window**: any asset generated between the API's MinIO cutover and this redeploy has bytes stranded in GridFS (refs in Mongo, API reads MinIO → 404). The #783 migration covered only *pre-cutover* GridFS objects. Need a catch-up GridFS→MinIO migration for that window, or regenerate affected assets. Bound the window by when the API env flipped. 2. **Deploy-trigger gap**: infra-repo stack changes don't trigger the app-repo deploy jobs that consume them — the exact reason this regressed. Consider an infra-repo path trigger or a redeploy hook. 3. Dedicated `artpipe-svc` MinIO user (currently reuses `photos-svc`).
spikerj commented 2026-07-29 16:11:42 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — still live, and NOT resolved by #783/#852/#853 closing. Recording the mechanism, because it's easy to assume the migration covered this.

The root cause is a deploy-trigger gap, not the migration. Confirmed structurally:

  • The artpipe worker's Storage__UseS3=true lives in spikersoft-infrastructure — spikersoft-artpipe-modeling/docker-stack-gpu.yml:70.
  • The deploy job that would apply it lives in spikersoft-backend — .gitea/workflows/spikersoft-artpipe-processor.yml, and its push: trigger paths (:5-16) are all backend source directories plus the workflow file itself: SpikerSoft.EventHandlers.ArtPipeProcessor/**, SpikerSoft.Common/**, SpikerSoft.Contracts.Workers/**, SpikerSoft.Data/**, SpikerSoft.Storage/**, SpikerSoft.EventHandlers.Infrastructure/**.

A workflow in the backend repo cannot trigger on an infrastructure-repo push. So an infra-only change to that env is inert until some unrelated backend commit happens to touch one of those paths and incidentally redeploys the workers. That's the defect, and #783's migration completing doesn't touch it.

So closing #783, #852 and #853 does not close this. Those addressed the data move and the key naming; this is about whether the running workers ever picked up the config.

Three follow-ups, all independently checkable and none of which I could confirm as done:

  1. Whether the deploy trigger now accounts for infra changes (it does not, per the paths above).
  2. Whether a catch-up GridFS→MinIO script exists for the orphan window — assets generated between the API cutover and the eventual worker redeploy. These are stranded regardless of #783's backfill, which ran at a fixed point in time. Note this is the same shape as #825's gap-window orphans, and #825's prevention items (writer-first ordering, post-cutover verification sweep) are also unlanded.
  3. Whether an artpipe-svc MinIO user exists.

Same defect as #841, which is also open — that ticket describes the ArtPipeProcessor deploy job shipping a stale checkout. While that's unresolved, this mechanism stays live.

Git cannot prove which env the running containers actually have. Settling evidence: docker service inspect spikersoft-artpipe-modeling_artpipe-modeling showing Storage__UseS3=true in the live task spec.

Audited against `origin/master` — **still live, and NOT resolved by #783/#852/#853 closing.** Recording the mechanism, because it's easy to assume the migration covered this. **The root cause is a deploy-trigger gap, not the migration.** Confirmed structurally: - The artpipe worker's `Storage__UseS3=true` lives in **spikersoft-infrastructure** — `spikersoft-artpipe-modeling/docker-stack-gpu.yml:70`. - The deploy job that would apply it lives in **spikersoft-backend** — `.gitea/workflows/spikersoft-artpipe-processor.yml`, and its `push:` trigger paths (`:5-16`) are all backend source directories plus the workflow file itself: `SpikerSoft.EventHandlers.ArtPipeProcessor/**`, `SpikerSoft.Common/**`, `SpikerSoft.Contracts.Workers/**`, `SpikerSoft.Data/**`, `SpikerSoft.Storage/**`, `SpikerSoft.EventHandlers.Infrastructure/**`. A workflow in the backend repo **cannot** trigger on an infrastructure-repo push. So an infra-only change to that env is inert until some unrelated backend commit happens to touch one of those paths and incidentally redeploys the workers. That's the defect, and #783's migration completing doesn't touch it. **So closing #783, #852 and #853 does not close this.** Those addressed the data move and the key naming; this is about whether the running workers ever picked up the config. **Three follow-ups, all independently checkable and none of which I could confirm as done:** 1. Whether the deploy trigger now accounts for infra changes (it does not, per the paths above). 2. Whether a catch-up GridFS→MinIO script exists for the **orphan window** — assets generated between the API cutover and the eventual worker redeploy. These are stranded regardless of #783's backfill, which ran at a fixed point in time. Note this is the same shape as **#825**'s gap-window orphans, and #825's prevention items (writer-first ordering, post-cutover verification sweep) are also unlanded. 3. Whether an `artpipe-svc` MinIO user exists. **Same defect as #841**, which is also open — that ticket describes the ArtPipeProcessor deploy job shipping a stale checkout. While that's unresolved, this mechanism stays live. Git cannot prove which env the running containers actually have. Settling evidence: `docker service inspect spikersoft-artpipe-modeling_artpipe-modeling` showing `Storage__UseS3=true` in the live task spec.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#820