spikerj/spikersoft-issues
1
0 Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

ArtStudio/photo-gallery downloads 404: artpipe workers write GridFS while API reads MinIO (stale pre-#783 deploy) #820

New Issue
Closed
opened 2026-07-23 17:34:40 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-23 17:34:40 +00:00
Owner
Copy link
Copy source

Summary

Art Studio / photo-gallery artifact downloads 404 for every newly generated asset: the API serves artifacts exclusively from MinIO (bucket art-asset-artifacts, Storage__UseS3=true), but the running artpipe-* pipeline workers still write generated artifacts to GridFS. Split-brain storage → every download of a fresh artifact fails.

Evidence (live prod, 2026-07-23)

  • API (spikersoft-backend, redeployed 15:04 UTC, Storage__UseS3=true):
    System.IO.FileNotFoundException: File 6a624d9e3f451a3e2c09ef4c not found in bucket art-asset-artifacts
    from DownloadArtifactQueryHandler → ArtStudioController.DownloadArtifact.
  • Writer (spikersoft-artpipe-photostack, asset 6a624d3aa63b0a42fd5581b8, 17:21:35 UTC):
    Uploaded art asset artifact aligned_003.jpg (align_fuse/image-jpeg) to **GridFS** with id 6a624d9f...
  • Asset delete swept 0 objects from art-asset-artifacts (bytes were never in MinIO).

Root cause

The running artpipe-photostack and artpipe-modeling services carry a stale, pre-#783 stack spec: their container env has no Storage__* keys (so UseS3Storage() defaults to GridFS) and they are not attached to the minio overlay network. The Storage__UseS3=true env was added to the infra stack files in #783 (commit 85c430e, 2026-07-22 01:56 UTC), but the ArtPipeProcessor deploy job only triggers on spikersoft-backend pushes — the #783 change landed in spikersoft-infrastructure, so the deploy job never re-ran to pick it up. (A #584-class stale-deploy.)

Fix

Redeploy the six artpipe stacks so they pick up the MinIO env + minio network (triggered via ArtPipeProcessor workflow_dispatch).

Follow-ups

  1. Orphan-asset window: any asset generated between the API's MinIO cutover and this redeploy has bytes stranded in GridFS (refs in Mongo, API reads MinIO → 404). The #783 migration covered only pre-cutover GridFS objects. Need a catch-up GridFS→MinIO migration for that window, or regenerate affected assets. Bound the window by when the API env flipped.
  2. Deploy-trigger gap: infra-repo stack changes don't trigger the app-repo deploy jobs that consume them — the exact reason this regressed. Consider an infra-repo path trigger or a redeploy hook.
  3. Dedicated artpipe-svc MinIO user (currently reuses photos-svc).
## Summary Art Studio / photo-gallery artifact downloads 404 for every **newly generated** asset: the API serves artifacts exclusively from MinIO (bucket `art-asset-artifacts`, `Storage__UseS3=true`), but the running `artpipe-*` pipeline workers still write generated artifacts to **GridFS**. Split-brain storage → every download of a fresh artifact fails. ## Evidence (live prod, 2026-07-23) - API (`spikersoft-backend`, redeployed 15:04 UTC, `Storage__UseS3=true`): `System.IO.FileNotFoundException: File 6a624d9e3f451a3e2c09ef4c not found in bucket art-asset-artifacts` from `DownloadArtifactQueryHandler` → `ArtStudioController.DownloadArtifact`. - Writer (`spikersoft-artpipe-photostack`, asset `6a624d3aa63b0a42fd5581b8`, 17:21:35 UTC): `Uploaded art asset artifact aligned_003.jpg (align_fuse/image-jpeg) to **GridFS** with id 6a624d9f...` - Asset delete swept **0** objects from `art-asset-artifacts` (bytes were never in MinIO). ## Root cause The running `artpipe-photostack` and `artpipe-modeling` services carry a **stale, pre-#783 stack spec**: their container env has **no `Storage__*` keys** (so `UseS3Storage()` defaults to GridFS) and they are **not attached to the `minio` overlay network**. The `Storage__UseS3=true` env was added to the infra stack files in #783 (commit `85c430e`, 2026-07-22 01:56 UTC), but the `ArtPipeProcessor` deploy job only triggers on **spikersoft-backend** pushes — the #783 change landed in **spikersoft-infrastructure**, so the deploy job never re-ran to pick it up. (A #584-class stale-deploy.) ## Fix Redeploy the six artpipe stacks so they pick up the MinIO env + `minio` network (triggered via `ArtPipeProcessor` workflow_dispatch). ## Follow-ups 1. **Orphan-asset window**: any asset generated between the API's MinIO cutover and this redeploy has bytes stranded in GridFS (refs in Mongo, API reads MinIO → 404). The #783 migration covered only *pre-cutover* GridFS objects. Need a catch-up GridFS→MinIO migration for that window, or regenerate affected assets. Bound the window by when the API env flipped. 2. **Deploy-trigger gap**: infra-repo stack changes don't trigger the app-repo deploy jobs that consume them — the exact reason this regressed. Consider an infra-repo path trigger or a redeploy hook. 3. Dedicated `artpipe-svc` MinIO user (currently reuses `photos-svc`).
spikerj commented 2026-07-29 16:11:42 +00:00
Author
Owner
Copy link
Copy source

Audited against origin/master — still live, and NOT resolved by #783/#852/#853 closing. Recording the mechanism, because it's easy to assume the migration covered this.

The root cause is a deploy-trigger gap, not the migration. Confirmed structurally:

  • The artpipe worker's Storage__UseS3=true lives in spikersoft-infrastructure — spikersoft-artpipe-modeling/docker-stack-gpu.yml:70.
  • The deploy job that would apply it lives in spikersoft-backend — .gitea/workflows/spikersoft-artpipe-processor.yml, and its push: trigger paths (:5-16) are all backend source directories plus the workflow file itself: SpikerSoft.EventHandlers.ArtPipeProcessor/**, SpikerSoft.Common/**, SpikerSoft.Contracts.Workers/**, SpikerSoft.Data/**, SpikerSoft.Storage/**, SpikerSoft.EventHandlers.Infrastructure/**.

A workflow in the backend repo cannot trigger on an infrastructure-repo push. So an infra-only change to that env is inert until some unrelated backend commit happens to touch one of those paths and incidentally redeploys the workers. That's the defect, and #783's migration completing doesn't touch it.

So closing #783, #852 and #853 does not close this. Those addressed the data move and the key naming; this is about whether the running workers ever picked up the config.

Three follow-ups, all independently checkable and none of which I could confirm as done:

  1. Whether the deploy trigger now accounts for infra changes (it does not, per the paths above).
  2. Whether a catch-up GridFS→MinIO script exists for the orphan window — assets generated between the API cutover and the eventual worker redeploy. These are stranded regardless of #783's backfill, which ran at a fixed point in time. Note this is the same shape as #825's gap-window orphans, and #825's prevention items (writer-first ordering, post-cutover verification sweep) are also unlanded.
  3. Whether an artpipe-svc MinIO user exists.

Same defect as #841, which is also open — that ticket describes the ArtPipeProcessor deploy job shipping a stale checkout. While that's unresolved, this mechanism stays live.

Git cannot prove which env the running containers actually have. Settling evidence: docker service inspect spikersoft-artpipe-modeling_artpipe-modeling showing Storage__UseS3=true in the live task spec.

Audited against `origin/master` — **still live, and NOT resolved by #783/#852/#853 closing.** Recording the mechanism, because it's easy to assume the migration covered this. **The root cause is a deploy-trigger gap, not the migration.** Confirmed structurally: - The artpipe worker's `Storage__UseS3=true` lives in **spikersoft-infrastructure** — `spikersoft-artpipe-modeling/docker-stack-gpu.yml:70`. - The deploy job that would apply it lives in **spikersoft-backend** — `.gitea/workflows/spikersoft-artpipe-processor.yml`, and its `push:` trigger paths (`:5-16`) are all backend source directories plus the workflow file itself: `SpikerSoft.EventHandlers.ArtPipeProcessor/**`, `SpikerSoft.Common/**`, `SpikerSoft.Contracts.Workers/**`, `SpikerSoft.Data/**`, `SpikerSoft.Storage/**`, `SpikerSoft.EventHandlers.Infrastructure/**`. A workflow in the backend repo **cannot** trigger on an infrastructure-repo push. So an infra-only change to that env is inert until some unrelated backend commit happens to touch one of those paths and incidentally redeploys the workers. That's the defect, and #783's migration completing doesn't touch it. **So closing #783, #852 and #853 does not close this.** Those addressed the data move and the key naming; this is about whether the running workers ever picked up the config. **Three follow-ups, all independently checkable and none of which I could confirm as done:** 1. Whether the deploy trigger now accounts for infra changes (it does not, per the paths above). 2. Whether a catch-up GridFS→MinIO script exists for the **orphan window** — assets generated between the API cutover and the eventual worker redeploy. These are stranded regardless of #783's backfill, which ran at a fixed point in time. Note this is the same shape as **#825**'s gap-window orphans, and #825's prevention items (writer-first ordering, post-cutover verification sweep) are also unlanded. 3. Whether an `artpipe-svc` MinIO user exists. **Same defect as #841**, which is also open — that ticket describes the ArtPipeProcessor deploy job shipping a stale checkout. While that's unresolved, this mechanism stays live. Git cannot prove which env the running containers actually have. Settling evidence: `docker service inspect spikersoft-artpipe-modeling_artpipe-modeling` showing `Storage__UseS3=true` in the live task spec.
spikerj commented 2026-08-07 13:52:31 +00:00
Author
Owner
Copy link
Copy source

Verified 2026-08-07 — the reported bug is fixed live, but two structural follow-ups remain, so this migrates rather than closes clean.

Fixed: docker service inspect spikersoft-artpipe-modeling_artpipe-modeling shows the running task spec carrying Storage__UseS3=true / Storage__ServiceUrl=http://minio:9000, so writer and reader now agree on MinIO and the split-brain 404s are gone. The orphan window was swept by the idempotent MigrateArtAssetArtifactsToMinio re-run on 2026-07-24 (108 migrated, 44 already verified, 0 failed of 152).

Still open — migrated to spikerj/spikersoft-infrastructure#187:

  1. The deploy-trigger gap that caused this is unchanged. .gitea/workflows/spikersoft-artpipe-processor.yml lives in spikersoft-backend and its push: paths (:5-16) are all backend source dirs — a backend workflow cannot trigger on an infra-repo push, so stack-file changes stay inert until an unrelated backend commit incidentally redeploys. Same class as #841. Fixing it from the infra side (a dispatch-on-change workflow) is why it routed there.
  2. No dedicated artpipe-svc MinIO user — the live spec still shows Storage__AccessKey=photos-svc.

Status: partially done — the storage split-brain and the orphan window are resolved; the trigger gap and the service account are not.

Closing here. Work now lives in the repo that holds the fix, so fixes #187 in a PR will auto-close it on merge. The umbrella tracker keeps cross-repo epics only.

— Opus 5 Agent

Verified 2026-08-07 — the reported bug is fixed live, but two structural follow-ups remain, so this migrates rather than closes clean. **Fixed:** `docker service inspect spikersoft-artpipe-modeling_artpipe-modeling` shows the running task spec carrying `Storage__UseS3=true` / `Storage__ServiceUrl=http://minio:9000`, so writer and reader now agree on MinIO and the split-brain 404s are gone. The orphan window was swept by the idempotent `MigrateArtAssetArtifactsToMinio` re-run on 2026-07-24 (108 migrated, 44 already verified, 0 failed of 152). **Still open** — migrated to **spikerj/spikersoft-infrastructure#187**: 1. The deploy-trigger gap that caused this is unchanged. `.gitea/workflows/spikersoft-artpipe-processor.yml` lives in spikersoft-backend and its `push:` paths (`:5-16`) are all backend source dirs — a backend workflow cannot trigger on an infra-repo push, so stack-file changes stay inert until an unrelated backend commit incidentally redeploys. Same class as #841. Fixing it from the infra side (a dispatch-on-change workflow) is why it routed there. 2. No dedicated `artpipe-svc` MinIO user — the live spec still shows `Storage__AccessKey=photos-svc`. Status: partially done — the storage split-brain and the orphan window are resolved; the trigger gap and the service account are not. Closing here. Work now lives in the repo that holds the fix, so `fixes #187` in a PR will auto-close it on merge. The umbrella tracker keeps cross-repo epics only. — Opus 5 Agent
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 epic
Cross-repo epic — children live in per-repo trackers
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#820