artpipe-model-blender:latest missing from registry — service Reject-loops every ~5s since the 04:11Z artpipe redeploy #887

New Issue

2026-07-28T14:24:32Z

spikerj commented

2026-07-28 14:24:32 +00:00

Since the artpipe stack redeploy at 2026-07-28 04:11:43Z (ArtPipeProcessor CI run), spikersoft-artpipe-model-blender_artpipe-model-blender has been stuck 0/1 with the scheduler retrying every ~5s:

Rejected ... "No such image: git.spikersoft.com/spikerj/artpipe-model-blender:latest"

docker manifest inspect git.spikersoft.com/spikerj/artpipe-model-blender:latest fails (manifest missing) and no local copy exists on the 4090 (the current artpipe-gpu node). Every other artpipe model image pulled fine in the same deploy — so either the blender image build/push was skipped/failed in the pipeline (cf. #877 ModelEnvImages cache/canceled-run fragility, #839 one-bad-model-blocks-the-set), or the tag was GC'd from the registry.

Impact: any pipeline stage needing the blender container (export/assembly work) can't run — jobs will stall the way #839 describes; plus perpetual scheduler churn. Not related to the #885 broker outage (this task never gets far enough to open a connection — it predates it by 8 hours).

Fix: re-run the blender image build/push (or the ModelEnvImages workflow with the dispatch escape hatch from #839), confirm the manifest exists for the node arch, then the service resolves on its own. Consider a post-push docker manifest inspect gate in CI so a deploy can't reference a tag that didn't land.

Since the artpipe stack redeploy at 2026-07-28 04:11:43Z (ArtPipeProcessor CI run), `spikersoft-artpipe-model-blender_artpipe-model-blender` has been stuck 0/1 with the scheduler retrying every ~5s: ``` Rejected ... "No such image: git.spikersoft.com/spikerj/artpipe-model-blender:latest" ``` `docker manifest inspect git.spikersoft.com/spikerj/artpipe-model-blender:latest` fails (manifest missing) and no local copy exists on the 4090 (the current `artpipe-gpu` node). Every other artpipe model image pulled fine in the same deploy — so either the blender image build/push was skipped/failed in the pipeline (cf. #877 ModelEnvImages cache/canceled-run fragility, #839 one-bad-model-blocks-the-set), or the tag was GC'd from the registry. Impact: any pipeline stage needing the blender container (export/assembly work) can't run — jobs will stall the way #839 describes; plus perpetual scheduler churn. Not related to the #885 broker outage (this task never gets far enough to open a connection — it predates it by 8 hours). Fix: re-run the blender image build/push (or the ModelEnvImages workflow with the dispatch escape hatch from #839), confirm the manifest exists for the node arch, then the service resolves on its own. Consider a post-push `docker manifest inspect` gate in CI so a deploy can't reference a tag that didn't land.

spikerj referenced this issue

2026-07-28 14:45:57 +00:00

OUTAGE: dreamstream4 down → RabbitMQ backbone unreachable since 12:17Z — consumers can't connect, redis node lost #885

spikerj closed this issue

2026-07-29 01:30:19 +00:00

spikerj referenced this issue

2026-07-29 13:40:28 +00:00

2026-07-29 power outage: RabbitMQ mnesia corruption took down all consumers (recovered) #895

Sign in to join this conversation.