[Bug][Infra][Observability][QA] node-agent shows a green 7/7 while missing on ds1/ds4/ds6 — global-service desired-count hides a 30% monitoring blind spot on exactly the degraded nodes #603

New Issue

2026-07-14T22:38:08Z

spikerj commented

2026-07-14 22:38:08 +00:00

QA Team — a monitoring blind spot that is invisible precisely because the dashboard reports it as healthy.

spikersoft-node-agent is a global service (one task per node). docker service ls shows 7/7 — green. But three nodes have no running task:

node-agent  dreamstream1  Rejected 18h ago  "No such image: git.spikersoft.com/spikerj/spikersoft-node-agent:latest"
node-agent  dreamstream4  Rejected 11h ago  "No such image: ...node-agent:latest"
node-agent  dreamstream6  Rejected 14h ago  "No such image: ...node-agent:latest"

Why the count lies

Swarm global-mode desired-count only counts nodes where a task was successfully placed. A node that rejects every task is silently dropped from the denominator, so 7/7 means "7 of the 7 nodes we managed to place on", not "7 of 10 nodes". We have a 30% monitoring blind spot on ds1/ds4/ds6 and every dashboard reads green.

This is not academic: node-agent is what feeds the journal/system telemetry that the ops log-watch loops and #555 (node-outage alerting) rely on. The three nodes with no agent are exactly the three nodes that are already degraded (see #587/#579) — i.e. we are blind on the nodes that most need watching.

Root cause

The same registry :latest failure as #587 — node-agent:latest returns manifest unknown; ds1/4/6 lack a stale cached copy, so they reject. Fixing #587 will fix the placement, but the lying 7/7 is an independent observability bug worth its own guard.

Suggested fix / acceptance

Alert on global-service coverage, not replica ratio: compare running tasks against docker node ls node count, and alert when a global service is absent from any Ready node. desired == running is not sufficient for global mode.
Acceptance: with a global service failing on N nodes, the monitor reports coverage < 100% and names the missing nodes, rather than showing a green N/N.

Related: #587 (registry :latest root cause), #555 (node-outage alerting), #579 (ds1/4/6 degraded).

**QA Team** — a monitoring blind spot that is invisible precisely because the dashboard reports it as healthy. `spikersoft-node-agent` is a **global** service (one task per node). `docker service ls` shows **`7/7`** — green. But three nodes have **no running task**: ``` node-agent dreamstream1 Rejected 18h ago "No such image: git.spikersoft.com/spikerj/spikersoft-node-agent:latest" node-agent dreamstream4 Rejected 11h ago "No such image: ...node-agent:latest" node-agent dreamstream6 Rejected 14h ago "No such image: ...node-agent:latest" ``` ## Why the count lies Swarm **global-mode desired-count only counts nodes where a task was successfully placed.** A node that rejects every task is silently dropped from the denominator, so `7/7` means "7 of the 7 nodes we managed to place on", not "7 of 10 nodes". **We have a 30% monitoring blind spot on ds1/ds4/ds6 and every dashboard reads green.** This is not academic: node-agent is what feeds the journal/system telemetry that the ops log-watch loops and #555 (node-outage alerting) rely on. **The three nodes with no agent are exactly the three nodes that are already degraded** (see #587/#579) — i.e. we are blind on the nodes that most need watching. ## Root cause The same registry `:latest` failure as #587 — `node-agent:latest` returns `manifest unknown`; ds1/4/6 lack a stale cached copy, so they reject. Fixing #587 will fix the placement, but the **lying `7/7`** is an independent observability bug worth its own guard. ## Suggested fix / acceptance - Alert on **global-service coverage**, not replica ratio: compare running tasks against `docker node ls` node count, and alert when a global service is absent from any `Ready` node. `desired == running` is not sufficient for global mode. - Acceptance: with a global service failing on N nodes, the monitor reports coverage `< 100%` and names the missing nodes, rather than showing a green N/N. Related: #587 (registry `:latest` root cause), #555 (node-outage alerting), #579 (ds1/4/6 degraded).

spikerj commented

2026-07-29 16:00:34 +00:00

Audited against origin/master — NOT DONE. No implementing change of any kind.

git log origin/master --grep='#603' across all repos: zero commits.
No global-service coverage guard exists. git grep -iE 'global.?service|global-mode|coverage' over infra *.sh|*.py|*.yml|*.md returns only Redis --cluster-require-full-coverage lines and one prose mention at docs/swarm-resilience-audit-510.md:66. Searching backend SpikerSoft.EventHandlers.NodeAgent/* and SystemRemediation/* for NodeCoverage|docker node ls|global.*service returns only a docstring at NodeAgent/Program.cs:10.
The acceptance criterion — a monitor comparing running global tasks against the docker node ls count and naming the missing nodes — has no implementation anywhere.

One thing worth separating, since it could otherwise look like this ticket got fixed indirectly: the stated root cause (#587, registry :latest) is a different problem. Even if #587 is fully resolved and node-agent starts everywhere, the reporting defect this ticket is about — a global service showing 7/7 when it is running on 4 of 7 nodes — remains, because desired-count for a global service is derived from where tasks were scheduled, not from the node inventory. That's why this needs its own guard rather than waiting on #587.

This also compounds two things I've found elsewhere this week. It's the same family as #579 (docker service ls reports 1/1 for a service crash-looping every 20s) — swarm's own summary views are not trustworthy liveness signals, and we keep discovering that one service at a time. And with #756 (ops alerting silently dead), a monitoring blind spot has nothing behind it.

Remaining: all of it. The concrete shape is a check that enumerates docker node ls, compares against running tasks per global service, and reports the names of nodes with no task — plus a decision about where that lands (node-agent, docker-monitor, or a Prometheus rule alongside the exporter #698 also needs).

Audited against `origin/master` — **NOT DONE. No implementing change of any kind.** - `git log origin/master --grep='#603'` across all repos: **zero commits.** - No global-service coverage guard exists. `git grep -iE 'global.?service|global-mode|coverage'` over infra `*.sh|*.py|*.yml|*.md` returns only Redis `--cluster-require-full-coverage` lines and one prose mention at `docs/swarm-resilience-audit-510.md:66`. Searching backend `SpikerSoft.EventHandlers.NodeAgent/*` and `SystemRemediation/*` for `NodeCoverage|docker node ls|global.*service` returns only a docstring at `NodeAgent/Program.cs:10`. - The acceptance criterion — a monitor comparing running global tasks against the `docker node ls` count and **naming** the missing nodes — has no implementation anywhere. One thing worth separating, since it could otherwise look like this ticket got fixed indirectly: the stated root cause (#587, registry `:latest`) is a **different** problem. Even if #587 is fully resolved and node-agent starts everywhere, the reporting defect this ticket is about — a global service showing `7/7` when it is running on 4 of 7 nodes — remains, because desired-count for a global service is derived from where tasks were scheduled, not from the node inventory. That's why this needs its own guard rather than waiting on #587. This also compounds two things I've found elsewhere this week. It's the same family as **#579** (`docker service ls` reports `1/1` for a service crash-looping every 20s) — swarm's own summary views are not trustworthy liveness signals, and we keep discovering that one service at a time. And with **#756** (ops alerting silently dead), a monitoring blind spot has nothing behind it. **Remaining:** all of it. The concrete shape is a check that enumerates `docker node ls`, compares against running tasks per global service, and reports the *names* of nodes with no task — plus a decision about where that lands (node-agent, docker-monitor, or a Prometheus rule alongside the exporter #698 also needs).

spikerj referenced this issue

2026-07-29 16:07:05 +00:00

storybook: static build broken on master (addon-actions import, node built-ins, worker tsconfig) #728

spikerj referenced this issue

2026-07-29 18:31:48 +00:00

[Bug][Infra][Alerting] Node outages are invisible to ops alerting — no node-down alerts, kernel alerts arrive after recovery and carry no kernel message, cooldown hides escalation #555

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#603