spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Critical][Ops][Notifications] Ops alerting is silently dead — every system-remediation-alert email fails SMTP AUTH 535; critical disk/OOM alerts never delivered #756

New Issue
Open
opened 2026-07-21 02:54:05 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-21 02:54:05 +00:00
Owner
Copy Link
Copy Source

Found during 2026-07-21 swarm recon (Seq sweep 07-20T14:00Z → 07-21T02:50Z).

Evidence

  • SpikerSoft.EventHandlers.Notifications fails every email send with MailKit.Security.AuthenticationException: 535: 5.7.8 Error: authentication failed: (reason unavailable) at SmtpClient.AuthenticateAsync → SpikerSoft.Business.Services.EmailService.SendEmailAsync (/src/SpikerSoft.Workers.Notifications/Services/EmailService.cs:308).
  • 22 failures in the 12h window, 27 in the prior 24h — ongoing since at least 2026-07-19T14:00Z, steady rate.
  • Survived the 01:42Z 2026-07-21 notifications redeploy (failures at 02:22:51 and 02:24:17 on the new container 77828eb16cb9) — so it is config/credentials, not a stale image.

Impact (why this is Critical)

The undelivered system-remediation-alert emails to team@spikersoft.com included:

  • CRITICAL: Disk critically low on SERVER:/ — climbing hourly 93% (14:21Z) → 99% (02:22Z). Nobody was paged; found only by log sweep. (Disk since remediated — see the SERVER-disk ticket.)
  • HIGH: OOM kill on SERVER at 20:32, 22:58, 00:27, 02:24Z.
  • HIGH: Kernel error on dreamstream1 at 16:42, 20:50, 22:08, 01:31Z (+ one on 4090 14:05Z).

Circular-failure design flaw: the alert channel for infrastructure failures is itself email, so a mail failure blinds ops to everything including itself. The deep SMTP health check from #227 exists but its failure also only surfaces… via email/status endpoints nobody watched.

Correlation / suspected cause

No recent PR touches SMTP sending (checked backend PRs #408–#439). Notifications SMTP creds now come from OpenBao (Phase 2 #546; bootstrap-notifications-secrets.sh). Most likely a rotated/never-provisioned SMTP password in Bao for the notifications sender account (compare the MailProvisioning:ConnectionString empty-string trap found during backend PR #431 work — blank appsettings value sourced into Bao). First step: bao kv get -format=json the notifications mail secret and verify field names only, then test AUTH against mailserver with the account.

Suggested fixes

  1. Restore working SMTP creds (verify against mailserver_mailserver, port 587).
  2. Add a non-email escape hatch for system-remediation alerts (Seq signal/webhook, RabbitMQ→dashboard, anything not SMTP).
  3. Alert on the notifications DLQ / failure rate so a dead mail pipeline is itself visible.
Found during 2026-07-21 swarm recon (Seq sweep 07-20T14:00Z → 07-21T02:50Z). ## Evidence - `SpikerSoft.EventHandlers.Notifications` fails **every** email send with `MailKit.Security.AuthenticationException: 535: 5.7.8 Error: authentication failed: (reason unavailable)` at `SmtpClient.AuthenticateAsync` → `SpikerSoft.Business.Services.EmailService.SendEmailAsync` (`/src/SpikerSoft.Workers.Notifications/Services/EmailService.cs:308`). - 22 failures in the 12h window, 27 in the prior 24h — ongoing since **at least 2026-07-19T14:00Z**, steady rate. - **Survived the 01:42Z 2026-07-21 notifications redeploy** (failures at 02:22:51 and 02:24:17 on the new container `77828eb16cb9`) — so it is config/credentials, not a stale image. ## Impact (why this is Critical) The undelivered `system-remediation-alert` emails to team@spikersoft.com included: - `CRITICAL: Disk critically low on SERVER:/` — climbing hourly 93% (14:21Z) → **99% (02:22Z)**. Nobody was paged; found only by log sweep. (Disk since remediated — see the SERVER-disk ticket.) - `HIGH: OOM kill on SERVER` at 20:32, 22:58, 00:27, 02:24Z. - `HIGH: Kernel error on dreamstream1` at 16:42, 20:50, 22:08, 01:31Z (+ one on 4090 14:05Z). Circular-failure design flaw: the alert channel for infrastructure failures is itself email, so a mail failure blinds ops to everything including itself. The deep SMTP health check from #227 exists but its failure also only surfaces… via email/status endpoints nobody watched. ## Correlation / suspected cause No recent PR touches SMTP sending (checked backend PRs #408–#439). Notifications SMTP creds now come from OpenBao (Phase 2 #546; `bootstrap-notifications-secrets.sh`). Most likely a rotated/never-provisioned SMTP password in Bao for the notifications sender account (compare the `MailProvisioning:ConnectionString` empty-string trap found during backend PR #431 work — blank appsettings value sourced into Bao). First step: `bao kv get -format=json` the notifications mail secret and verify field names only, then test AUTH against mailserver with the account. ## Suggested fixes 1. Restore working SMTP creds (verify against mailserver_mailserver, port 587). 2. Add a non-email escape hatch for system-remediation alerts (Seq signal/webhook, RabbitMQ→dashboard, anything not SMTP). 3. Alert on the notifications DLQ / failure rate so a dead mail pipeline is itself visible.
spikerj commented 2026-07-22 00:32:49 +00:00
Author
Owner
Copy Link
Copy Source

Concrete blast radius from the dead alerting: learn.spikersoft.com was fully down (Traefik 404, frontend 0/1 replicas) from 2026-07-21 22:01Z to ~08:0xZ 2026-07-22 — ~10 hours, zero pages. Found only by a human asking 'is the site down?'. See #775 (corrupt registry blob) for the cause chain.

Concrete blast radius from the dead alerting: learn.spikersoft.com was fully down (Traefik 404, frontend 0/1 replicas) from 2026-07-21 22:01Z to ~08:0xZ 2026-07-22 — ~10 hours, zero pages. Found only by a human asking 'is the site down?'. See #775 (corrupt registry blob) for the cause chain.
spikerj commented 2026-07-22 01:45:42 +00:00
Author
Owner
Copy Link
Copy Source

Correction to my earlier comment: the learn.spikersoft.com outage was ~2 hours (22:01Z→~00:10Z), not ~10 — my time math was wrong. Still zero pages for a full site outage.

Correction to my earlier comment: the learn.spikersoft.com outage was ~2 hours (22:01Z→~00:10Z), not ~10 — my time math was wrong. Still zero pages for a full site outage.
spikerj commented 2026-07-29 16:02:56 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — NOT DONE, and fix 3 as written is currently impossible. That's the useful finding, so leading with it.

Fix 3 — "alert on the notifications DLQ / failure rate" — has no DLQ to alert on. SpikerSoft.Workers.Notifications/README.md:88-105 documents the defect on master, in the repo's own words:

Failure handling — known defect: no retry, no DLQ … on a handler exception the consumer builds a best-effort Success=false error envelope (RPC routes only) and then acks the message anyway — there is no retry ladder and none of these queues is bound to a dead-letter exchange. On the fire-and-forget email routes that is a silent data-loss path: a transactional email that throws (password reset, verification code) is logged and dropped, and the caller already received its JobAccepted.

And the config that looks like it covers this is inert: EventHandlers.Notifications/appsettings.json:27-28 has a populated RabbitMQ:DeadLetterQueue block (Enabled: true, MaxRetries: 3, RetryDelays: [5, 30, 300], 7-day TTL, max length 10 000), but per README.md:104-105 NotificationsRpcConsumerHostedService never binds any of it.

So the ordering here is wrong in the ticket: you cannot alert on a DLQ until the DLQ is wired. That's prerequisite work, not part of the alerting fix.

Worth noting this is the third instance of the same shape I've hit this week — config that reads as a shipped feature and is wired to nothing. #589 has AutoRetryTransientErrors / AutoRetryDelayHours "configured everywhere and read nowhere"; #698 has a consumer-liveness registry that nothing registers with; #847 has a GenerationParams:Enabled flag no code path reads. Might be worth a sweep for inert config as its own piece of work.

Fix 2 — non-email escape hatch: NOT DONE. git grep -rniE "webhook|seq signal|slack|escape hatch" origin/master -- '*SystemRemediation*' '*Notifications*' returns zero non-test hits. The circular-failure design flaw this ticket identifies — the alert channel for infrastructure failures is itself email — is completely unaddressed.

Fix 1 — SMTP credentials: not verifiable from git (OpenBao). The failing call is unchanged at SpikerSoft.Workers.Notifications/Services/EmailService.cs:308 (client.AuthenticateAsync(_smtpUsername, _smtpPassword, …)).

A diagnostic lead for fix 1, found while auditing #610. spikersoft-infrastructure/mailserver/docker-stack.yml commits five literal plaintext credentials (:38 DBPASS, :39 RSPAMD_PASSWORD, :108 DBPASS, :177 MYSQL_RANDOM_ROOT_PASSWORD, :180 MYSQL_PASSWORD) — I've filed these on #633 for rotation. If the mail account's password was ever rotated in Bao without the mailserver side being updated (or vice versa), that mismatch would present exactly as a persistent 535 authentication failed surviving redeploys, which matches the symptom here precisely. Worth comparing the Bao notifications secret against the mailserver's account before assuming the Bao value is simply missing.

Still critical. This has been live since at least 2026-07-19, and the #895 post-mortem (2026-07-29) lists "alerting still blind (#756) — nobody was paged for this either" among its follow-ups. That's a second major incident that went unpaged.

Audited against `origin/master` — **NOT DONE, and fix 3 as written is currently impossible.** That's the useful finding, so leading with it. **Fix 3 — "alert on the notifications DLQ / failure rate" — has no DLQ to alert on.** `SpikerSoft.Workers.Notifications/README.md:88-105` documents the defect on master, in the repo's own words: > **Failure handling — known defect: no retry, no DLQ** … on a handler exception the consumer builds a best-effort `Success=false` error envelope (RPC routes only) and then **acks the message anyway** — there is no retry ladder and none of these queues is bound to a dead-letter exchange. On the fire-and-forget email routes that is a silent data-loss path: a transactional email that throws (password reset, verification code) is logged and dropped, and the caller already received its `JobAccepted`. And the config that looks like it covers this is inert: `EventHandlers.Notifications/appsettings.json:27-28` has a populated `RabbitMQ:DeadLetterQueue` block (`Enabled: true`, `MaxRetries: 3`, `RetryDelays: [5, 30, 300]`, 7-day TTL, max length 10 000), but per `README.md:104-105` **`NotificationsRpcConsumerHostedService` never binds any of it**. So the ordering here is wrong in the ticket: you cannot alert on a DLQ until the DLQ is wired. That's prerequisite work, not part of the alerting fix. Worth noting this is the *third* instance of the same shape I've hit this week — config that reads as a shipped feature and is wired to nothing. #589 has `AutoRetryTransientErrors` / `AutoRetryDelayHours` "configured everywhere and read nowhere"; #698 has a consumer-liveness registry that nothing registers with; #847 has a `GenerationParams:Enabled` flag no code path reads. Might be worth a sweep for inert config as its own piece of work. **Fix 2 — non-email escape hatch: NOT DONE.** `git grep -rniE "webhook|seq signal|slack|escape hatch" origin/master -- '*SystemRemediation*' '*Notifications*'` returns **zero** non-test hits. The circular-failure design flaw this ticket identifies — the alert channel for infrastructure failures is itself email — is completely unaddressed. **Fix 1 — SMTP credentials: not verifiable from git** (OpenBao). The failing call is unchanged at `SpikerSoft.Workers.Notifications/Services/EmailService.cs:308` (`client.AuthenticateAsync(_smtpUsername, _smtpPassword, …)`). **A diagnostic lead for fix 1, found while auditing #610.** `spikersoft-infrastructure/mailserver/docker-stack.yml` commits five literal plaintext credentials (`:38 DBPASS`, `:39 RSPAMD_PASSWORD`, `:108 DBPASS`, `:177 MYSQL_RANDOM_ROOT_PASSWORD`, `:180 MYSQL_PASSWORD`) — I've filed these on #633 for rotation. If the mail account's password was ever rotated in Bao without the mailserver side being updated (or vice versa), that mismatch would present exactly as a persistent `535 authentication failed` surviving redeploys, which matches the symptom here precisely. Worth comparing the Bao notifications secret against the mailserver's account before assuming the Bao value is simply missing. **Still critical.** This has been live since at least 2026-07-19, and the #895 post-mortem (2026-07-29) lists "alerting still blind (#756) — nobody was paged for this either" among its follow-ups. That's a second major incident that went unpaged.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#756