[Bug][Infra][Deploy] backend deploy reports success while swarm rolls back on transient image-pull failure (silent non-deploy, #584 class) #774
Assignees
spikerj (Joseph Spiker)
Clear assignees
spikerj (Joseph Spiker)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: spikerj/spikersoft-issues#774
Reference in New Issue
Block a user
Observed 2026-07-21 ~19:09 UTC on the #443 merge pipeline: build_and_publish_backend (both arches) and create_manifest succeeded, the deploy ran
service update… and the swarm task was Rejected: "No such image: …backend:latest@sha256:221bdd8d…" on SERVER — a transient pull failure (manifest propagation race; a manual pull of the same digest from another node succeeded minutes later). Swarm auto-rolled back (UpdateStatus: rollback_completed) and kept the OLD task running, while the pipeline stayed green. Net effect: a merged change was silently NOT deployed — the same "reports success, deploys nothing" class as #584.Recovered manually:
docker service update --image <exact digest>→ converged, app healthy.Fix suggestion: the deploy step should verify CONVERGENCE, not just command exit — after
service update, polldocker service inspect .UpdateStatus.Stateuntilcompleted(fail the job onrollback_completed/paused), and assert the running task's image digest equals the freshly pushed one. The existing "manifest never became queryable after 60s" guard checks queryability from the runner, not pullability on the target node, so it can't catch this. A retry-once-on-rejection would also paper over the propagation race.Related: #768 (registry blob integrity), #584 (stale-checkout deploys), InfrastructureCheckoutIsPulledTests precedent for enforcing deploy-job invariants via tests.
Observation #5 (2026-07-22 frontend deploy): publish green → SERVER rejected the fresh digest ('No such image') repeatedly → task wedged in Preparing → site served 404s ~6 min until manual rollback. Registry-side layer streaming then verified ALL layers byte-complete (incl. 608MB) — so this instance was purely the node-pull race, and the retry converged first try. Two hardening notes from it: (1) rolling updates for the single-replica web service are stop-first — every deploy has a downtime window even when healthy; consider start-first update order for spikersoft-angular_spikersoft; (2) pull-verification must be COLD (registry layer streaming or a cache-less node) — warm-node pulls are satisfied from build cache and prove nothing.
Observation #6 (2026-07-22 ~23:21–23:53Z): same silent-non-deploy signature now on
spikersoft-embeddings.New affected service — this extends the scope of this bug beyond backend/frontend to a worker service.
Scope / evidence (live, ongoing at time of writing)
spikersoft-embeddings_embeddings— currently 0/1 replicas.SERVER(same node as Observation #5).Rejected→Preparing, repeating. Rejection error:No such image: git.spikersoft.com/spikerj/spikersoft-embeddings:latest@sha256:45109073bfbd05eb96d28b796cd5a97b3d7413557171302925811daabdead3c1Preparing(>2 min, not converging).docker service inspect→UpdateStatus.State = paused, message: "update paused due to failure or early termination of task …".Why this matters here
This is the exact #774 class: swarm accepts the deploy, then SERVER rejects the freshly-referenced digest ('No such image') and the update pauses rather than rolling forward — so it will not self-heal. Because the update is paused (not failed loudly), a deploy pipeline can still report success while the service sits at 0/1.
Correlation
No such image, wedged in Preparing).Status
Ongoing / active replica deficit as of 2026-07-22T23:53Z. Will not recover without operator intervention (update is paused).
Suggested next investigation (non-destructive)
sha256:45109073…dead3c1is actually present/complete in the registry backend for the embeddings repo (read-verify), per #775.Deduplication performed
Reviewed open #774 (this issue, Observation #5), #700, #775; no existing comment covers this embeddings occurrence.
Agent metadata
Observation window 2026-07-22T23:21Z–23:53Z. Evidence fingerprint:
embeddings-nosuchimage-SERVER-sha256-45109073.Follow-up to Observation #6 —
spikersoft-embeddingsstill wedged after ~2h; retry loop escalated.Status as of 2026-07-23T01:16Z:
UpdateStatus.State = paused(msg: "update paused due to failure or early termination of task wy8tt96c…").No such image: …spikersoft-embeddings:latest@sha256:45109073…dead3c1on nodeSERVER.Why this is worth flagging
Still needs (non-destructive)
sha256:45109073…dead3c1is actually present/complete in the registry backend for the embeddings repo (read-verify, per #775), then re-push or re-pin the tag.Agent metadata
Observation window 2026-07-22T23:21Z–2026-07-23T01:16Z. Fingerprint:
embeddings-nosuchimage-SERVER-sha256-45109073-2h-escalation.Resolved —
spikersoft-embeddingsback to 1/1.As of 2026-07-23T01:26Z the service is 1/1,
UpdateStatus.State = completed, task Running since ~01:20Z. LastNo such imagerejection was ~01:19Z; no rejections since.paused.sha256:45109073…dead3c1onSERVER, consistent with the digest/blob finally being resolvable (registry-side), i.e. no config change was required on the service — supports the registry-blob-availability root cause tracked here / in #775.Note for the underlying bug: the ~2h at 0/1 with
UpdateStatus=pausedand no auto-rollback is exactly the silent-non-deploy behavior this issue tracks — recovery here was environmental (blob became pullable), not because swarm self-healed.Agent metadata
Resolution observed 2026-07-23T01:26Z. Fingerprint:
embeddings-nosuchimage-SERVER-sha256-45109073-resolved.Audited against
origin/master— NOT DONE, and the blast radius is fleet-wide. Staying open.No convergence check exists. The shared deploy path is
spikersoft-backend/.gitea/workflows/_build-deploy-service.yml. Its "Deploy to production" step (:206-238) ends atsudo docker stack deploy --with-registry-auth -c docker-stack.yml …, and the job finishes on that command's exit code. There is no convergence poll, noUpdateStatus.Statecheck, no assertion that the running task's digest matches the pushed one, and no retry on rejection.git grep -rn "UpdateStatus\|rollback_completed" origin/masteracross spikersoft-backend returns only unrelated C# plus a comment atspikersoft-quiz-generation.yml:68that merely namesUpdateStatus=pausedwhile describing the #511 manifest gate. No executable check anywhere.Because this lives in the one reusable workflow, every service pipeline inherits the gap — it isn't a per-service issue. Observation #6's extension to
spikersoft-embeddings(2h at 0/1,UpdateStatus=paused, no self-heal) is therefore still live for every service.Worth being explicit about a false-coverage trap here, because it's easy to look at CI and conclude this is handled.
_build-deploy-service.yml:196-205runs adocker manifest inspectpost-push gate (from #511), and workflow comments cite it as post-push verification. It verifies manifest queryability from the runner — not blob readability, and not pullability on the target node. So it runs on every deploy and structurally cannot catch either this ticket's failure class or #775's. Its presence should not be read as coverage; both this ticket's and #775's threads say so explicitly.This compounds two other open tickets into a single blind spot:
sudo git -C /mnt/infrastructure pullfails and the|| echo "WARNING…"swallows it, meaning the stack file being deployed may be stale.Together: CI can report a successful deploy of a stale stack file that then rolled back, with no signal at any step. That's why #582 and #519 can't currently be settled — the pipeline cannot prove it shipped anything.
Remaining: poll
docker service inspect --format '{{.UpdateStatus.State}}'(or equivalent) after deploy untilcompleted, fail onrolled_back/paused, and assert the running task's image digest equals the pushed digest.