[Bug][Infra][Deploy] quiz-generation rolling update races the image push — 'No such image' reject loops + UpdateStatus=paused #511

New Issue

2026-07-12T18:32:28Z

spikerj commented

2026-07-12 18:32:28 +00:00

Summary

quiz-generation (final book-pipeline stage) keeps tripping its rolling update with No such image on freshly-pushed digests, then self-heals a minute or two later. Observed 3 different digests fail then recover in ~10 min (2026-07-12 ~18:18–18:31 UTC):

Rejected  ...@sha256:5f25f967...  "No such image"   (~18:03)
Rejected  ...@sha256:17d60d8b...  "No such image"   (~18:23)  -> later converged 1/1
Rejected  ...@sha256:fc0b0731...  "No such image"   (~18:29)  -> later converged 1/1

Current state: service is 1/1 Running on the fc0b0731 digest, and docker manifest inspect now finds it in the registry — but UpdateStatus: paused — update paused due to failure or early termination of task ....

Root cause (very likely): CI updates the service spec BEFORE the image push completes

The pattern — spec pinned to @sha256:X → swarm rejects with No such image: ...@sha256:X → the same digest becomes pullable a bit later and the task converges — is the classic push/deploy ordering race: the pipeline runs docker service update --image ...@sha256:X (or stack deploy) while the docker push of that digest is still in flight / not yet committed on the registry. Swarm can't pull it, tasks get Rejected, and after enough failures the rolling update flips to paused (which can leave the service stuck on a bad revision if the push never lands).

Impact

Transient outages of the final pipeline stage (quiz generation) on every deploy — books completing during the window don't get their quiz created until it converges.
UpdateStatus=paused means a genuinely-bad push (digest that never lands) would leave quiz-generation stuck rather than rolling forward — a deploy that silently half-applies.
Noise: repeated No such image reject loops make real failures harder to spot.

Fix options

Order the pipeline correctly — ensure docker push (all layers + manifest) fully completes and the digest is verifiable in the registry before the service update/stack deploy step. A simple gate: after push, docker manifest inspect <registry>/<repo>@sha256:<digest> must succeed before updating the service.
Deploy by immutable tag, then update — push a unique tag (e.g. commit SHA), verify it's pullable, then update the service to that tag/digest. Avoids racing :latest.
Registry write consistency — if the registry (Gitea) returns success on push before the manifest is queryable, add a short verify-with-retry after push.
Consider --update-failure-action rollback (or fixing the paused state handling) so a bad digest rolls back cleanly instead of parking paused.

Repro / evidence

docker service ps spikersoft-quiz-generation_quiz-generation → interleaved Rejected "No such image: ...@sha256:..." then Running across 3 digests.
docker service inspect ... --format '{{.UpdateStatus.State}} {{.UpdateStatus.Message}}' → paused — update paused due to failure or early termination of task ....
docker manifest inspect .../spikersoft-quiz-generation:latest → EXISTS now (i.e. the digest that was "No such image" minutes ago is present now → timing, not a permanently-missing image).

Suggested test / guard

CI: after build+push, a mandatory docker manifest inspect <repo>@<digest> (with brief retry) gate that must pass before the deploy step. That single check would have prevented all three reject loops above.

## Summary `quiz-generation` (final book-pipeline stage) keeps tripping its rolling update with `No such image` on freshly-pushed digests, then self-heals a minute or two later. Observed **3 different digests fail then recover** in ~10 min (2026-07-12 ~18:18–18:31 UTC): ``` Rejected ...@sha256:5f25f967... "No such image" (~18:03) Rejected ...@sha256:17d60d8b... "No such image" (~18:23) -> later converged 1/1 Rejected ...@sha256:fc0b0731... "No such image" (~18:29) -> later converged 1/1 ``` Current state: service is `1/1 Running` on the fc0b0731 digest, and `docker manifest inspect` now finds it in the registry — but `UpdateStatus: paused — update paused due to failure or early termination of task ...`. ## Root cause (very likely): CI updates the service spec BEFORE the image push completes The pattern — spec pinned to `@sha256:X` → swarm rejects with `No such image: ...@sha256:X` → the *same* digest becomes pullable a bit later and the task converges — is the classic **push/deploy ordering race**: the pipeline runs `docker service update --image ...@sha256:X` (or `stack deploy`) while the `docker push` of that digest is still in flight / not yet committed on the registry. Swarm can't pull it, tasks get Rejected, and after enough failures the rolling update flips to `paused` (which can leave the service stuck on a bad revision if the push never lands). ## Impact - Transient outages of the **final pipeline stage** (quiz generation) on every deploy — books completing during the window don't get their quiz created until it converges. - `UpdateStatus=paused` means a genuinely-bad push (digest that never lands) would leave quiz-generation stuck rather than rolling forward — a deploy that silently half-applies. - Noise: repeated `No such image` reject loops make real failures harder to spot. ## Fix options 1. **Order the pipeline correctly** — ensure `docker push` (all layers + manifest) fully completes and the digest is verifiable in the registry *before* the `service update`/`stack deploy` step. A simple gate: after push, `docker manifest inspect <registry>/<repo>@sha256:<digest>` must succeed before updating the service. 2. **Deploy by immutable tag, then update** — push a unique tag (e.g. commit SHA), verify it's pullable, then update the service to that tag/digest. Avoids racing `:latest`. 3. **Registry write consistency** — if the registry (Gitea) returns success on push before the manifest is queryable, add a short verify-with-retry after push. 4. Consider `--update-failure-action rollback` (or fixing the paused state handling) so a bad digest rolls back cleanly instead of parking `paused`. ## Repro / evidence - `docker service ps spikersoft-quiz-generation_quiz-generation` → interleaved `Rejected "No such image: ...@sha256:..."` then `Running` across 3 digests. - `docker service inspect ... --format '{{.UpdateStatus.State}} {{.UpdateStatus.Message}}'` → `paused — update paused due to failure or early termination of task ...`. - `docker manifest inspect .../spikersoft-quiz-generation:latest` → EXISTS now (i.e. the digest that was "No such image" minutes ago is present now → timing, not a permanently-missing image). ## Related - #510 — resilience exploration, **item C** (verify pushed digest is pullable before updating the spec). This ticket is the concrete, actionable instance of that item. - #508 — image-description phantom deploy (same family: service spec references an image the registry doesn't have at update time). ## Suggested test / guard - CI: after build+push, a mandatory `docker manifest inspect <repo>@<digest>` (with brief retry) gate that must pass before the deploy step. That single check would have prevented all three reject loops above.

spikerj commented

2026-07-12 18:46:08 +00:00

Both guards up: backend PR #238 (post-push docker manifest inspect retry gate — the deploy step can no longer start until the digest is actually servable; a push that never lands fails the BUILD) and infra PR #44 (update_config: failure_action: rollback, order: start-first — residual bad rollouts self-revert instead of parking paused). Scoped to quiz-generation per this ticket; the same 8-line gate is copy-paste for the other ~25 workflows — recommend a follow-up sweep (happy to do it as one PR on request). Timing note: tonight's three occurrences coincided with the registry's disk/MinIO stress — exactly when push-success ≠ immediately-servable, which is why the gate polls the registry rather than trusting the push exit code.

Both guards up: **backend PR #238** (post-push `docker manifest inspect` retry gate — the deploy step can no longer start until the digest is actually servable; a push that never lands fails the BUILD) and **infra PR #44** (`update_config: failure_action: rollback, order: start-first` — residual bad rollouts self-revert instead of parking paused). Scoped to quiz-generation per this ticket; the same 8-line gate is copy-paste for the other ~25 workflows — recommend a follow-up sweep (happy to do it as one PR on request). Timing note: tonight's three occurrences coincided with the registry's disk/MinIO stress — exactly when push-success ≠ immediately-servable, which is why the gate polls the registry rather than trusting the push exit code.

spikerj commented

2026-07-12 18:49:42 +00:00

Fleet sweep up: PR #239 gates all 28 image-publishing workflows (also re-lands quiz-generation's gate — #238 had merged with only the accidental empty commit). With infra #44's rollback policy merged, both halves of this ticket are done pending the sweep merge; #510's item C is covered fleet-wide by the same PR. Will close on merge.

Fleet sweep up: **PR #239** gates all 28 image-publishing workflows (also re-lands quiz-generation's gate — #238 had merged with only the accidental empty commit). With infra #44's rollback policy merged, both halves of this ticket are done pending the sweep merge; #510's item C is covered fleet-wide by the same PR. Will close on merge.

spikerj commented

2026-07-12 19:23:07 +00:00

Closing with evidence: backend PR #239 merged (deploy-race gate — post-push docker manifest inspect retry, 12×5s — now guards all 28 image-publishing workflows incl. quiz-generation) and infra PR #44 merged (update_config: failure_action: rollback, order: start-first). Both halves of the fix are on master; subsequent deploys have not reproduced the No such image reject loop. Fleet-wide coverage also satisfies #510 item C.

Closing with evidence: **backend PR #239 merged** (deploy-race gate — post-push `docker manifest inspect` retry, 12×5s — now guards all 28 image-publishing workflows incl. quiz-generation) and **infra PR #44 merged** (`update_config: failure_action: rollback, order: start-first`). Both halves of the fix are on master; subsequent deploys have not reproduced the `No such image` reject loop. Fleet-wide coverage also satisfies #510 item C.

spikerj closed this issue

2026-07-12 19:23:07 +00:00

spikerj referenced this issue

2026-07-12 21:40:14 +00:00

[Exploration][Infra] Resilience to SERVER node flaps — restart-policy windows, auto-reconcile sweep, de-SPOF the SERVER-pinned stateful tier #510

spikerj referenced this issue

2026-07-13 00:40:18 +00:00

[ArtPipe] Baked images Phase 0: build scaffolding (Dockerfiles, CI, setup flags) + disk/limits go-no-go gates (epic #515) #516

spikerj referenced this issue

2026-07-13 21:07:28 +00:00

[CI][Backend] Multi-arch create_manifest amends MUTABLE :amd64/:arm64v8 tags — out-of-order arch pushes can stitch a mixed-commit :latest (all backend multi-arch workflows) #548

spikerj referenced this issue