PROD: GET /api/artstudio 500s — #853 objectKey rename deployed to the reader but not the artpipe writer #856

New Issue

2026-07-26T14:06:58Z

spikerj commented

2026-07-26 14:06:58 +00:00

Symptom

GET /api/artstudio (ListMyArtAssets) returns 500 for the owner of any asset
written since the #853 API deploy:

System.Text.Json.JsonException: JSON deserialization for type
'SpikerSoft.Data.Mongos.ArtStudio.ArtAssetArtifactRef' was missing required
properties including: 'objectKey'.
  at ...ListMyArtAssetsQueryHandler.Handle(...):line 32

The Art Studio asset list is the picker's data source, so this also blocks
assigning art in the game playgrounds.

Root cause: deploy skew across the #853 breaking rename

#853 renamed the artifact ref's storage key gridFsId → objectKey and marked
it [JsonRequired] so a missing key fails loudly instead of silently yielding
"" (the #852 bug). Sound on its own — but the writer was never redeployed:

service	role	image updated	has #853
`spikersoft-backend`	reader (API)	2026-07-26 03:57	yes
`spikersoft-artpipe-modeling`	writer (ArtPipeProcessor)	2026-07-25 22:56	no

#853 merged 2026-07-26 02:08. The reader was redeployed after it; the writer was
not. So the processor still persists the pre-#853 shape while the API requires
the new one.

Evidence

Asset 6a6611b7022ed7387cbe3564 ("Game Command Center"), created
2026-07-26 13:55:03 — ~10 hours after the rename merged — has both its
artifact refs in the legacy shape:

{"Stage":"concept","gridFsId":"6a6611b7022ed7387cbe3564__d24c0b8ab64143...",
 "StageRunId":"...","Kind":"image-png","FileName":"sdxl_lightning_6ebf034b.png",
 ...,"sourceArtifactGridFsId":null}

Note the value in gridFsId is already the new {assetId}__{guid} object-key
format, not a GridFS ObjectId — so the writer is post-MinIO-cutover but
pre-#853-rename. This is purely a field-name skew, not a storage-layout problem.

PascalCase (Stage, Kind) is not the issue — case-insensitive matching handles
those. gridFsId → objectKey is a genuine rename and the only broken field.

Blast radius

Ongoing. Every asset the processor writes from here on is unreadable by the
API. Currently 2 bad refs across 1 asset (94 refs total in the collection);
that asset is still mid-pipeline (texturing), so texturing/export/enrichment
will each append another.
One bad ref anywhere in a returned page fails the whole query, so the list
breaks for that user, not just for that asset.

Fix options

A. Redeploy the writer. Correct and needs no code change, but the
ArtPipeProcessor ships inside the 36 GiB artpipe-model-prodstages monolith, so
this means a full image rebuild via model-env-images.yml. Already-written docs
still need repair.

B. Accept gridFsId as a read-side alias for objectKey. Ships in a normal
backend deploy, and also repairs reading of the docs already written. Preserves
#853's intent — fail loudly when the key is genuinely absent, rather than
defaulting to "" — by treating the legacy spelling as the same value, not as a
missing one.

Recommend B plus A: B unblocks now and is the right defensive posture for a
breaking rename whose writer deploys independently; A stops the bad writes at
source. A read-side alias should survive at least one full release cycle after
the writer is confirmed updated.

Also needed

Repair the 2 existing legacy refs (a one-shot $set renaming the field, or
covered for free by option B).
Guard against recurrence: a breaking rename of a field shared by two
independently-deployed services needs the tolerant-reader window as part of the
change, not as a follow-up. #853 removed the #852 shim in the same release that
introduced the new name.

## Symptom `GET /api/artstudio` (ListMyArtAssets) returns 500 for the owner of any asset written since the #853 API deploy: ``` System.Text.Json.JsonException: JSON deserialization for type 'SpikerSoft.Data.Mongos.ArtStudio.ArtAssetArtifactRef' was missing required properties including: 'objectKey'. at ...ListMyArtAssetsQueryHandler.Handle(...):line 32 ``` The Art Studio asset list is the picker's data source, so this also blocks assigning art in the game playgrounds. ## Root cause: deploy skew across the #853 breaking rename #853 renamed the artifact ref's storage key `gridFsId` → `objectKey` and marked it `[JsonRequired]` so a missing key fails loudly instead of silently yielding `""` (the #852 bug). Sound on its own — but the **writer was never redeployed**: | service | role | image updated | has #853 | |---|---|---|---| | `spikersoft-backend` | reader (API) | 2026-07-26 **03:57** | yes | | `spikersoft-artpipe-modeling` | **writer** (ArtPipeProcessor) | 2026-07-25 **22:56** | **no** | #853 merged 2026-07-26 02:08. The reader was redeployed after it; the writer was not. So the processor still persists the pre-#853 shape while the API requires the new one. ## Evidence Asset `6a6611b7022ed7387cbe3564` ("Game Command Center"), created **2026-07-26 13:55:03** — ~10 hours after the rename merged — has both its artifact refs in the legacy shape: ```json {"Stage":"concept","gridFsId":"6a6611b7022ed7387cbe3564__d24c0b8ab64143...", "StageRunId":"...","Kind":"image-png","FileName":"sdxl_lightning_6ebf034b.png", ...,"sourceArtifactGridFsId":null} ``` Note the value in `gridFsId` is already the **new** `{assetId}__{guid}` object-key format, not a GridFS ObjectId — so the writer is post-MinIO-cutover but pre-#853-rename. This is purely a field-name skew, not a storage-layout problem. PascalCase (`Stage`, `Kind`) is not the issue — case-insensitive matching handles those. `gridFsId` → `objectKey` is a genuine rename and the only broken field. ## Blast radius - **Ongoing.** Every asset the processor writes from here on is unreadable by the API. Currently 2 bad refs across 1 asset (94 refs total in the collection); that asset is still mid-pipeline (`texturing`), so texturing/export/enrichment will each append another. - One bad ref anywhere in a returned page fails the **whole** query, so the list breaks for that user, not just for that asset. ## Fix options **A. Redeploy the writer.** Correct and needs no code change, but the ArtPipeProcessor ships inside the 36 GiB `artpipe-model-prodstages` monolith, so this means a full image rebuild via `model-env-images.yml`. Already-written docs still need repair. **B. Accept `gridFsId` as a read-side alias for `objectKey`.** Ships in a normal backend deploy, and also repairs reading of the docs already written. Preserves #853's intent — fail loudly when the key is genuinely absent, rather than defaulting to `""` — by treating the legacy spelling as the same value, not as a missing one. Recommend **B plus A**: B unblocks now and is the right defensive posture for a breaking rename whose writer deploys independently; A stops the bad writes at source. A read-side alias should survive at least one full release cycle after the writer is confirmed updated. ## Also needed - Repair the 2 existing legacy refs (a one-shot `$set` renaming the field, or covered for free by option B). - **Guard against recurrence:** a breaking rename of a field shared by two independently-deployed services needs the tolerant-reader window as part of the change, not as a follow-up. #853 removed the #852 shim in the same release that introduced the new name.

spikerj commented

2026-07-26 14:29:12 +00:00

Why the rebuilds kept failing (run 18574)

Not disk — the pinned ubuntu-amd64-4090 runner had 1.5 TB free. Two
compounding problems:

1. A push-triggered run rebuilds all TEN model images. DEFAULT_IMAGES is
safetycheck prodstages photostack triposr triposg shape instantmesh sf3d hunyuan3dpaint textto3d, and a push passes no inputs.images, so it takes that
path — ~200 GB of pulls and pushes per run.

2. Any single image failing blocks the deploy of all the others. The build
step ends if [ -n "${FAILED_IMAGES}" ]; then exit 1, and deploy has
needs: [build_and_publish]. Run 18574 died on safetycheck — the FIRST of the
ten — with unexpected EOF pulling a layer from the registry (#538 flakiness).
prodstages, the only image needed to fix this ticket, was never attempted.
The runner container then vanished and the docker daemon stopped responding; the
retry timestamps tell the story — 05:12, 05:20, then 06:00, 07:05, 08:10.

Fix applied

Dispatched with images: prodstages — the recovery move the build step's own
comment prescribes. One image instead of ten, its env base already cached on the
runner, and deploy is unconditional on which images built (it redeploys every
artpipe stack including spikersoft-artpipe-modeling), so it ships.

Because #486 merged first, this single rebuild also carries the export-stage
target_faces fix from #855.

What was NOT changed, and why

My first instinct was to narrow DEFAULT_IMAGES so pushes stopped rebuilding
all ten. That would have been a bug. Nine of the ten back services running
at 1/1 right now (the #357 per-model modeling lane, #718 photostack, #839
triposr), so a change to SpikerSoft.Data must reach all of them. Narrowing it
would silently stop updating nine live workers — which is precisely how this
ticket happened: the reader shipped a renamed field and the writer never got it.

The stale part was the workflow_dispatch description, which still claimed
"Empty = the DEPLOYED set only (safetycheck prodstages)" from before those three
expansions. Corrected on docs/artpipe-processor-dispatch-description, which
also documents the partial-failure recovery move.

That branch's PR is deliberately not open yet: the workflow lists its own
file in paths, so merging it triggers a full ten-image rebuild, and a critical
prodstages build is in flight.

Remaining recurrence risk (not addressed here)

Fail-closed-on-any-image plus #538 registry flakiness means a ten-image run
rarely completes, and one flaky pull strands nine good images. The build step's
comment defends this ("silently deploying a half-built set is worse than an
obvious red job") — a deliberate call, so it should be changed on purpose rather
than as a side effect of this ticket. Worth its own issue.

Also worth noting: the runner carries ~1.8 TB of reclaimable Docker cruft
(769 GB images, 976 GB volumes, 60 GB build cache). Not a factor today — the
workflow's env-image eviction only fires at 85% and the runner is at 57% — but a
40 GB image build is one bad week from it mattering.

## Why the rebuilds kept failing (run 18574) Not disk — the pinned `ubuntu-amd64-4090` runner had 1.5 TB free. Two compounding problems: **1. A push-triggered run rebuilds all TEN model images.** `DEFAULT_IMAGES` is `safetycheck prodstages photostack triposr triposg shape instantmesh sf3d hunyuan3dpaint textto3d`, and a push passes no `inputs.images`, so it takes that path — ~200 GB of pulls and pushes per run. **2. Any single image failing blocks the deploy of all the others.** The build step ends `if [ -n "${FAILED_IMAGES}" ]; then exit 1`, and `deploy` has `needs: [build_and_publish]`. Run 18574 died on `safetycheck` — the FIRST of the ten — with `unexpected EOF` pulling a layer from the registry (#538 flakiness). **`prodstages`, the only image needed to fix this ticket, was never attempted.** The runner container then vanished and the docker daemon stopped responding; the retry timestamps tell the story — 05:12, 05:20, then 06:00, 07:05, 08:10. ## Fix applied Dispatched with `images: prodstages` — the recovery move the build step's own comment prescribes. One image instead of ten, its env base already cached on the runner, and `deploy` is unconditional on which images built (it redeploys every artpipe stack including `spikersoft-artpipe-modeling`), so it ships. Because #486 merged first, this single rebuild also carries the export-stage `target_faces` fix from #855. ## What was NOT changed, and why My first instinct was to narrow `DEFAULT_IMAGES` so pushes stopped rebuilding all ten. **That would have been a bug.** Nine of the ten back services running at 1/1 right now (the #357 per-model modeling lane, #718 photostack, #839 triposr), so a change to `SpikerSoft.Data` must reach all of them. Narrowing it would silently stop updating nine live workers — which is precisely how this ticket happened: the reader shipped a renamed field and the writer never got it. The stale part was the `workflow_dispatch` **description**, which still claimed "Empty = the DEPLOYED set only (safetycheck prodstages)" from before those three expansions. Corrected on `docs/artpipe-processor-dispatch-description`, which also documents the partial-failure recovery move. That branch's PR is deliberately **not open yet**: the workflow lists its own file in `paths`, so merging it triggers a full ten-image rebuild, and a critical `prodstages` build is in flight. ## Remaining recurrence risk (not addressed here) Fail-closed-on-any-image plus #538 registry flakiness means a ten-image run rarely completes, and one flaky pull strands nine good images. The build step's comment defends this ("silently deploying a half-built set is worse than an obvious red job") — a deliberate call, so it should be changed on purpose rather than as a side effect of this ticket. Worth its own issue. Also worth noting: the runner carries ~1.8 TB of reclaimable Docker cruft (769 GB images, 976 GB volumes, 60 GB build cache). Not a factor today — the workflow's env-image eviction only fires at 85% and the runner is at 57% — but a 40 GB image build is one bad week from it mattering.

spikerj referenced this issue

2026-07-26 19:25:54 +00:00

SECURITY: container registry serves anonymous pull tokens over the public internet #857

spikerj referenced this issue

2026-07-26 19:26:14 +00:00

artpipe env images: two RUN loops produce 17 GB and 16 GB single layers — split per model #858

spikerj referenced this issue

2026-07-26 19:31:01 +00:00

artpipe: 33 GB of immutable weights/venvs are baked into every CI build — make them data, not image layers #859

spikerj referenced this issue

2026-07-29 15:30:58 +00:00

artpipe: decimation to 6k faces tore 60% of the mesh surface open — relax it, and replace the simplifier #861

spikerj commented

2026-07-29 15:32:01 +00:00

Audited against origin/master. The prod 500 was resolved operationally, not by a code change — recording that clearly, because the PR trail is misleading.

Per comment 13152, the actual fix was dispatching a prodstages-only rebuild so the writer image picked up #853. Backend PRs #488/#489 reference this ticket but are CI registry perf only — they are not the fix, and shouldn't be read as one.
Master's source was never defective: writer and reader bind the same class. SpikerSoft.Data/Mongos/ArtStudio/ArtAsset.cs:428 ArtAssetArtifactRef, with ObjectKey at ~:447-449 still a bare [BsonElement("objectKey")] [JsonPropertyName("objectKey")] [JsonRequired]. The skew was purely a deploy-ordering artifact.

Still open — the two hardening items:

Option B, the gridFsId read-side alias (the ticket recommends "B plus A"). No alias, converter, or fallback exists anywhere: git grep -i gridFsId origin/master -- '*.cs' hits only unrelated Sponsor/Photography/Profile fields. Without it, the same reader-before-writer ordering reproduces this outage exactly.
The tolerant-reader-window practice to guard against recurrence — nothing captured in docs or CI.

Also worth flagging: mongodb-scripts/2026-07-migrate-art-assets-object-key.js exists but shipped with #853 (e2fe0346), i.e. before the 2026-07-26 13:55 bad writes. Nothing in the repo evidences a re-run to repair the documents written during the skew window. If those assets still 500 on download, that re-run is outstanding.

Given [JsonRequired] on ObjectKey makes this failure mode hard-fail rather than degrade, item 1 is the one worth doing.

Audited against `origin/master`. **The prod 500 was resolved operationally, not by a code change** — recording that clearly, because the PR trail is misleading. - Per comment 13152, the actual fix was dispatching a `prodstages`-only rebuild so the writer image picked up #853. Backend PRs #488/#489 reference this ticket but are **CI registry perf only** — they are not the fix, and shouldn't be read as one. - Master's source was never defective: writer and reader bind the same class. `SpikerSoft.Data/Mongos/ArtStudio/ArtAsset.cs:428` `ArtAssetArtifactRef`, with `ObjectKey` at ~`:447-449` still a bare `[BsonElement("objectKey")] [JsonPropertyName("objectKey")] [JsonRequired]`. The skew was purely a deploy-ordering artifact. **Still open — the two hardening items:** 1. **Option B, the `gridFsId` read-side alias** (the ticket recommends "B plus A"). No alias, converter, or fallback exists anywhere: `git grep -i gridFsId origin/master -- '*.cs'` hits only unrelated Sponsor/Photography/Profile fields. Without it, the same reader-before-writer ordering reproduces this outage exactly. 2. **The tolerant-reader-window practice** to guard against recurrence — nothing captured in docs or CI. Also worth flagging: `mongodb-scripts/2026-07-migrate-art-assets-object-key.js` exists but shipped **with #853** (`e2fe0346`), i.e. *before* the 2026-07-26 13:55 bad writes. Nothing in the repo evidences a re-run to repair the documents written during the skew window. If those assets still 500 on download, that re-run is outstanding. Given `[JsonRequired]` on `ObjectKey` makes this failure mode hard-fail rather than degrade, item 1 is the one worth doing.

spikerj referenced this issue

2026-07-29 16:03:52 +00:00

Art Studio MinIO migration: reader cutover preceded writer cutover → gap-window artifacts orphaned (500 on download) #825

spikerj referenced this issue

2026-07-29 16:04:22 +00:00

[CI][Backend] Tier-3 artpipe finals builds keep getting superseded/cancelled before pushing — dino fix (#702) never reached the registry #717

spikerj referenced this issue

2026-07-29 17:14:14 +00:00

[Infra][Architecture] Provider-agnostic object storage (MinIO -> S3/Azure Blob) to replace GlusterFS/host-path model #413

Sign in to join this conversation.