spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra][Alerting] Node outages are invisible to ops alerting — no node-down alerts, kernel alerts arrive after recovery and carry no kernel message, cooldown hides escalation #555

New Issue
Open
opened 2026-07-14 02:53:00 +00:00 by spikerj · 4 comments
spikerj commented 2026-07-14 02:53:00 +00:00
Owner
Copy Link
Copy Source

QA Team — found during QA log watch 2026-07-14 ~03:05Z, distilled from tonight's dreamstream4 incident (#552).

The core gap: in the last 5h the only ops emails were HIGH: Kernel error on <node> (ds4, ds1, 4090), HIGH: OOM kill on 4090, and one crash-loop. Nothing alerted on the outage itself: dreamstream4 — a swarm manager — was Down/Unreachable ~01:15–02:40Z and the first related email landed 02:32Z, essentially as the node came back. Root cause of the blindness: kernel/OOM detection lives in the node-agent on the node — when the node dies, its reporter dies with it, and nothing on the manager side alerts on swarm node state.

What that blindness cost tonight (all found by QA log-reading, none alerted):

  • InfluxDB (pinned to ds4) hard-down ~85 min while docker service ls showed a stale 1/1 → fleet-wide telemetry blackout (#554)
  • keycloak-postgres forced onto a broken mount → durable pg corruption (#552)
  • 6 nodes churning jetson-influx containers every 10s for ~90 min (#554)

Secondary gaps:

  1. Alert content: Kernel error on dreamstream1 contains no dmesg line — diagnosing requires SSH to the node (which LAN DNS makes awkward, #538). The node-agent has the raw kernel message when it fires the incident; put it (and the occurrence counter) in the email body.
  2. Cooldown hides escalation: dreamstream1 is at occurrences=53 and climbing with alerts suppressed ('Alert on cooldown'). dreamstream4 died at occurrences=33. A pure time-based cooldown means the more a node deteriorates, the quieter alerting gets. Re-alert on escalation (e.g. each doubling of the counter), not just elapsed time.

Suggested fix shape: manager-side node-state alerting — docker-monitor already subscribes to swarm node events (once #541 un-junks them, node down/unreachable is exactly the event to page on), or system-remediation can poll docker node ls from a manager. Plus the two secondary fixes above.

Refs: #552 (the incident), #554 (silent InfluxDB outage), #541 (node-event stream this could be built on), #538 (DNS friction for manual diagnosis).

**QA Team** — found during QA log watch 2026-07-14 ~03:05Z, distilled from tonight's dreamstream4 incident (#552). **The core gap:** in the last 5h the only ops emails were `HIGH: Kernel error on <node>` (ds4, ds1, 4090), `HIGH: OOM kill on 4090`, and one crash-loop. **Nothing alerted on the outage itself**: dreamstream4 — a swarm *manager* — was Down/Unreachable ~01:15–02:40Z and the first related email landed **02:32Z**, essentially as the node came back. Root cause of the blindness: kernel/OOM detection lives in the node-agent *on the node* — when the node dies, its reporter dies with it, and nothing on the manager side alerts on swarm node state. **What that blindness cost tonight (all found by QA log-reading, none alerted):** - InfluxDB (pinned to ds4) hard-down ~85 min while `docker service ls` showed a stale 1/1 → fleet-wide telemetry blackout (#554) - keycloak-postgres forced onto a broken mount → durable pg corruption (#552) - 6 nodes churning jetson-influx containers every 10s for ~90 min (#554) **Secondary gaps:** 1. **Alert content:** `Kernel error on dreamstream1` contains no dmesg line — diagnosing requires SSH to the node (which LAN DNS makes awkward, #538). The node-agent has the raw kernel message when it fires the incident; put it (and the occurrence counter) in the email body. 2. **Cooldown hides escalation:** dreamstream1 is at **occurrences=53 and climbing with alerts suppressed** ('Alert on cooldown'). dreamstream4 died at occurrences=33. A pure time-based cooldown means the more a node deteriorates, the quieter alerting gets. Re-alert on escalation (e.g. each doubling of the counter), not just elapsed time. **Suggested fix shape:** manager-side node-state alerting — docker-monitor already subscribes to swarm node events (once #541 un-junks them, `node down/unreachable` is exactly the event to page on), or system-remediation can poll `docker node ls` from a manager. Plus the two secondary fixes above. **Refs:** #552 (the incident), #554 (silent InfluxDB outage), #541 (node-event stream this could be built on), #538 (DNS friction for manual diagnosis).
spikerj commented 2026-07-14 02:59:09 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — alert-tuning addendum 2026-07-14 ~03:15Z: the fleet kernel errors turn out to be PCIe AER corrected errors (see #552 for the full diagnosis). AER corrected errors can burst by the hundreds while harming nothing, so the kernel-error detector should distinguish severity: Corrected → count and warn on rate/trend; Uncorrected/Fatal → page immediately. Treating every corrected line as HIGH both inflates counters (ds1 'occurrences=53') and guarantees the cooldown suppression this ticket already flags — the two problems compound: noisy corrected errors trigger cooldown, then a real uncorrected error lands inside the suppression window.

**QA Team** — alert-tuning addendum 2026-07-14 ~03:15Z: the fleet kernel errors turn out to be **PCIe AER corrected errors** (see #552 for the full diagnosis). AER corrected errors can burst by the hundreds while harming nothing, so the kernel-error detector should distinguish severity: `Corrected` → count and warn on *rate/trend*; `Uncorrected/Fatal` → page immediately. Treating every corrected line as HIGH both inflates counters (ds1 'occurrences=53') and guarantees the cooldown suppression this ticket already flags — the two problems compound: noisy corrected errors trigger cooldown, then a real uncorrected error lands inside the suppression window.
spikerj commented 2026-07-14 04:14:41 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — live deploy watch 2026-07-14 ~04:15Z: the new node-agent build (with JournalTailService — looks like this ticket's kernel-message-in-alert fix) is rolling out, but the rollout is stuck: UpdateStatus=paused since 04:11:31Z at 7/10 running (laptop-server, ds5 + one more without agents at check time). Needs a resume/re-deploy to converge — until then those nodes have no monitoring agent at all.

Two observations from the rollout for the record:

  1. dreamstream4 hit No such image: spikersoft-node-agent:latest rejects again before recovering to Preparing — third occurrence of the #511-signature pull race tonight (postgres→ds6, traefik→ds1 historic, now node-agent→ds4). Whatever makes fresh pulls flaky on the dreamstreams right after updates keeps costing rollout time; may deserve its own investigation once the night's fires are out.
  2. The new JournalTailService logged 2x TaskCanceledException publishing journal.dreamstream2 events at startup (04:12:48Z) — RabbitMQ shows no alarms/flow-control, so it looks like a startup race, but worth watching once all 10 agents are tailing journals: if journal lines are published per-line fleet-wide, mind the #541 lesson about event volume.
**QA Team** — live deploy watch 2026-07-14 ~04:15Z: the new node-agent build (with `JournalTailService` — looks like this ticket's kernel-message-in-alert fix) is rolling out, but the **rollout is stuck: UpdateStatus=paused since 04:11:31Z at 7/10 running** (laptop-server, ds5 + one more without agents at check time). Needs a resume/re-deploy to converge — until then those nodes have no monitoring agent at all. Two observations from the rollout for the record: 1. **dreamstream4 hit `No such image: spikersoft-node-agent:latest` rejects again** before recovering to Preparing — third occurrence of the #511-signature pull race tonight (postgres→ds6, traefik→ds1 historic, now node-agent→ds4). Whatever makes fresh pulls flaky on the dreamstreams right after updates keeps costing rollout time; may deserve its own investigation once the night's fires are out. 2. The new JournalTailService logged 2x `TaskCanceledException` publishing `journal.dreamstream2` events at startup (04:12:48Z) — RabbitMQ shows no alarms/flow-control, so it looks like a startup race, but worth watching once all 10 agents are tailing journals: if journal lines are published per-line fleet-wide, mind the #541 lesson about event volume.
spikerj commented 2026-07-14 13:33:45 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — a perfect concrete specimen for this ticket's severity-classification ask, 2026-07-14 ~13:35Z: system-remediation just dispatched a HIGH 'Kernel error on 4090' ops email (occurrences 28-29). The actual kernel lines:

uvcvideo 2-1.1.1:1.1: Failed to set UVC probe control : -32 (exp. 26)

That is a USB webcam stalling on probe — zero platform impact (node healthy: 56Gi RAM free, GPU idle at 35°C). Yet it pages at the same severity, with the same empty alert body, as the PCIe/NVMe corrected-error bursts that preceded real node crashes (#552). Two asks this crystallizes: (1) include the raw kernel line in the alert so recipients can triage from the email; (2) classify by subsystem/severity — uvcvideo/USB probe noise should be info-tier at most, AER corrected = warn-on-rate, AER uncorrected/panic/OOM = page. Right now the signal:noise of the kernel-error alert class trains people to ignore exactly the alert that matters.

**QA Team** — a perfect concrete specimen for this ticket's severity-classification ask, 2026-07-14 ~13:35Z: system-remediation just dispatched a **HIGH 'Kernel error on 4090'** ops email (occurrences 28-29). The actual kernel lines: ``` uvcvideo 2-1.1.1:1.1: Failed to set UVC probe control : -32 (exp. 26) ``` That is a **USB webcam stalling on probe** — zero platform impact (node healthy: 56Gi RAM free, GPU idle at 35°C). Yet it pages at the same severity, with the same empty alert body, as the PCIe/NVMe corrected-error bursts that preceded real node crashes (#552). Two asks this crystallizes: (1) include the raw kernel line in the alert so recipients can triage from the email; (2) classify by subsystem/severity — `uvcvideo`/USB probe noise should be info-tier at most, AER corrected = warn-on-rate, AER uncorrected/panic/OOM = page. Right now the signal:noise of the kernel-error alert class trains people to ignore exactly the alert that matters.
spikerj commented 2026-07-29 18:31:48 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — NOT DONE. All three gaps outstanding.

Fix 1 — manager-side node-down alerting: not built. git grep -rniE "node.?down|node.?unreachable|docker node ls" across DockerMonitor and SystemRemediation returns exactly one hit, and it's a comment: DockerClusterEventSource.cs:167 — "publishing them buried real node-down transitions in ~30k junk". So the #541 junk-filtering context is documented, but no alert is raised on the filtered event, and nothing polls docker node ls from a manager.

Fix 2 — kernel message in the alert body: not done. The only dmesg reference is RemediationRuleEngine.cs:280, and it's the recommendation text telling a human to go look ("Inspect dmesg/journal for hardware, driver, or filesystem faults"). The raw kernel line the node-agent already holds when it fires is still not carried into the alert, so diagnosis still requires SSH to the node.

Fix 3 — escalation-aware cooldown: not done. No cooldown logic exists in the remediation path at all; the only Cooldown hits are unrelated (auth throttling, game weapon benchmarks). So the pure time-based suppression this ticket describes is unchanged — a deteriorating node still gets quieter as it worsens.

Worth stating plainly: this ticket currently has no effect even if fixed. #756 (ops alerting silently dead — every system-remediation-alert email fails SMTP AUTH 535) is still open and unfixed. So node-down alerts would be generated and then dropped. And per the #895 post-mortem (2026-07-29), alerting was still blind during that outage too — "nobody was paged for this either."

That makes the ordering matter: #756's fix 2 — a non-email escape hatch — is the prerequisite for this ticket producing any value. I verified on #756 that no such path exists (git grep -rniE "webhook|seq signal|slack" across those services → zero non-test hits), and that its own fix 3 is currently impossible because the notifications DLQ config is never bound.

So the alerting cluster is: #756 (channel is dead, and its DLQ prerequisite is unwired) → this ticket (nothing detects node-down to send) → #603 (global services report healthy while missing from nodes) → #579 (service ls reports 1/1 for a service crash-looping every 20s). Four tickets, one theme: the signals that would tell you something is wrong are individually broken, so each fix alone changes nothing.

Notes here are accurate; nothing has moved since filing.

Audited against `origin/master` — **NOT DONE. All three gaps outstanding.** **Fix 1 — manager-side node-down alerting: not built.** `git grep -rniE "node.?down|node.?unreachable|docker node ls"` across DockerMonitor and SystemRemediation returns exactly one hit, and it's a **comment**: `DockerClusterEventSource.cs:167` — *"publishing them buried real node-down transitions in ~30k junk"*. So the #541 junk-filtering context is documented, but no alert is raised on the filtered event, and nothing polls `docker node ls` from a manager. **Fix 2 — kernel message in the alert body: not done.** The only `dmesg` reference is `RemediationRuleEngine.cs:280`, and it's the *recommendation text* telling a human to go look ("Inspect dmesg/journal for hardware, driver, or filesystem faults"). The raw kernel line the node-agent already holds when it fires is still not carried into the alert, so diagnosis still requires SSH to the node. **Fix 3 — escalation-aware cooldown: not done.** No cooldown logic exists in the remediation path at all; the only `Cooldown` hits are unrelated (auth throttling, game weapon benchmarks). So the pure time-based suppression this ticket describes is unchanged — a deteriorating node still gets quieter as it worsens. **Worth stating plainly: this ticket currently has no effect even if fixed.** **#756** (ops alerting silently dead — every `system-remediation-alert` email fails SMTP AUTH 535) is still open and unfixed. So node-down alerts would be generated and then dropped. And per the **#895** post-mortem (2026-07-29), alerting was still blind during that outage too — *"nobody was paged for this either."* That makes the ordering matter: **#756's fix 2 — a non-email escape hatch — is the prerequisite for this ticket producing any value.** I verified on #756 that no such path exists (`git grep -rniE "webhook|seq signal|slack"` across those services → zero non-test hits), and that its own fix 3 is currently impossible because the notifications DLQ config is never bound. So the alerting cluster is: **#756** (channel is dead, and its DLQ prerequisite is unwired) → **this ticket** (nothing detects node-down to send) → **#603** (global services report healthy while missing from nodes) → **#579** (`service ls` reports 1/1 for a service crash-looping every 20s). Four tickets, one theme: the signals that would tell you something is wrong are individually broken, so each fix alone changes nothing. Notes here are accurate; nothing has moved since filing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#555