[Ops] book-management worker sat at 0/1 for 12 h after a registry 500 during pull — no alert fires for a service with zero replicas #1197

Open
opened 2026-10-10 04:17:21 +00:00 by spikerj · 0 comments
Owner

What happened

spikersoft-book-management_spikersoft-book-management was at 0/1 from 2026-10-09 15:38Z until 2026-10-10 04:12Z (recon found it). Task history:

  • SERVER: Rejected — "received unexpected HTTP status: 500 Internal Server Error" (image pull from the Gitea registry failed during the 15:24Z master fan-out, the same window the RustFS-backed registry was under the burst).
  • 4090: the retried task shows Complete then nothing further — the restart policy gave up and the service stayed empty.

docker service update --force at 04:12Z brought it back to 1/1 immediately (pull succeeded).

Why it matters

The system-remediation alerts fire for nodes going down (xavier-3/xavier-7 alerts arrived within seconds today) but nothing watches services with fewer replicas than desired. A worker can silently stop consuming its queue for half a day; book-management's bus messages were presumably unacked/redelivered the whole time (not verified).

Ask

  1. Add a replica-shortfall check to the docker-monitor/system-remediation sweep: any non-job, non-zero-desired service at running < desired for > N minutes → alert (same severity path as node-down). Exclude one-shot *-init / *initiator services.
  2. Consider --restart-max-attempts / longer restart-window on the worker stacks so a transient registry 500 does not exhaust the retry budget.

Found by the 2026-10-10 04:10Z swarm recon.

## What happened `spikersoft-book-management_spikersoft-book-management` was at **0/1** from 2026-10-09 15:38Z until 2026-10-10 04:12Z (recon found it). Task history: - SERVER: `Rejected — "received unexpected HTTP status: 500 Internal Server Error"` (image pull from the Gitea registry failed during the 15:24Z master fan-out, the same window the RustFS-backed registry was under the burst). - 4090: the retried task shows `Complete` then nothing further — the restart policy gave up and the service stayed empty. `docker service update --force` at 04:12Z brought it back to 1/1 immediately (pull succeeded). ## Why it matters The system-remediation alerts fire for **nodes** going down (xavier-3/xavier-7 alerts arrived within seconds today) but nothing watches **services with fewer replicas than desired**. A worker can silently stop consuming its queue for half a day; book-management's bus messages were presumably unacked/redelivered the whole time (not verified). ## Ask 1. Add a replica-shortfall check to the docker-monitor/system-remediation sweep: any non-job, non-zero-desired service at `running < desired` for > N minutes → alert (same severity path as node-down). Exclude one-shot `*-init` / `*initiator` services. 2. Consider `--restart-max-attempts` / longer `restart-window` on the worker stacks so a transient registry 500 does not exhaust the retry budget. Found by the 2026-10-10 04:10Z swarm recon.
Sign in to join this conversation.