[Observability] Add deep SMTP health check (TLS + AUTH) to SpikerSoft.EventHandlers.Notifications #227

New Issue

2026-06-16T00:35:55Z

spikerj commented

2026-06-16 00:35:55 +00:00

Background

After #225 / #226, all transactional email sending is owned by the SpikerSoft.EventHandlers.Notifications worker (it holds MailKit and the real EmailService). The API edge only does a shallow TCP-reachability smoke test and explicitly defers deep liveness to the worker:

SpikerSoft.Api/Infrastructure/HealthChecks/SMTP_HealthCheck.cs:
"Deeper protocol-level liveness (TLS handshake, AUTH, open-relay probe) belongs in the notifications worker's own health check, where it can own the MailKit dependency without bloating the API image."

But the worker's health checks (EventHandlerHostBuilder.WithHealthChecks -> AddEventHandlerHealthChecks) only register MongoDB (and not even that here, since it's called with includeMongoDb: false). There is currently no SMTP health check anywhere that validates TLS or credentials.

Why this matters

The email sends are now fire-and-forget remote commands. If the worker's SMTP credentials are wrong, the relay is unreachable over TLS, or AUTH starts failing, emails are silently dropped on the worker side and there is no health signal — exactly the class of silent failure that caused #225, just relocated to the worker.

Proposed change

Add an IHealthCheck in SpikerSoft.Workers.Notifications that, against the same Email:* configuration used by EmailService:

Connects to Email:Host:Email:Port using the same port -> SecureSocketOptions mapping as EmailService (465 = SslOnConnect, 587 = StartTls, else StartTlsWhenAvailable) — i.e. a real TLS handshake.
Optionally authenticates with Email:Username / Email:Password (toggle Email:HealthCheck:VerifyAuth, default true).
Disconnects cleanly and reports latency.

To avoid hammering the SMTP provider (and risking account lockout / rate limiting) when /health is scraped frequently, the check caches its last result for Email:HealthCheck:CacheSeconds (default 60) and serializes probes with a semaphore.

Failure is reported as Degraded (not Unhealthy) so a flaky relay doesn't trigger pod restart loops — the worker can still consume and retry. Registered on /health (detailed/readiness); /healthz liveness is unaffected.

Acceptance criteria

GET /health on the Notifications worker includes an smtp check that performs a TLS handshake and (when enabled) an AUTH probe.
Wrong credentials / unreachable relay surface as Degraded with a useful description, not a silent drop.
Probe frequency to the real SMTP server is bounded by Email:HealthCheck:CacheSeconds.
Email:HealthCheck:VerifyAuth=false downgrades to connect + TLS only (no AUTH).
Unit coverage for config parsing / option toggles; existing tests still pass.

Follow-up to #225 / #226.

## Background After #225 / #226, **all** transactional email sending is owned by the `SpikerSoft.EventHandlers.Notifications` worker (it holds MailKit and the real `EmailService`). The API edge only does a shallow TCP-reachability smoke test and explicitly defers deep liveness to the worker: > `SpikerSoft.Api/Infrastructure/HealthChecks/SMTP_HealthCheck.cs`: > "Deeper protocol-level liveness (TLS handshake, AUTH, open-relay probe) belongs in the notifications worker's own health check, where it can own the MailKit dependency without bloating the API image." But the worker's health checks (`EventHandlerHostBuilder.WithHealthChecks` -> `AddEventHandlerHealthChecks`) only register MongoDB (and not even that here, since it's called with `includeMongoDb: false`). **There is currently no SMTP health check anywhere that validates TLS or credentials.** ## Why this matters The email sends are now fire-and-forget remote commands. If the worker's SMTP credentials are wrong, the relay is unreachable over TLS, or AUTH starts failing, emails are silently dropped on the worker side and there is **no health signal** — exactly the class of silent failure that caused #225, just relocated to the worker. ## Proposed change Add an `IHealthCheck` in `SpikerSoft.Workers.Notifications` that, against the same `Email:*` configuration used by `EmailService`: 1. Connects to `Email:Host`:`Email:Port` using the same port -> `SecureSocketOptions` mapping as `EmailService` (465 = `SslOnConnect`, 587 = `StartTls`, else `StartTlsWhenAvailable`) — i.e. a real TLS handshake. 2. Optionally authenticates with `Email:Username` / `Email:Password` (toggle `Email:HealthCheck:VerifyAuth`, default `true`). 3. Disconnects cleanly and reports latency. To avoid hammering the SMTP provider (and risking account lockout / rate limiting) when `/health` is scraped frequently, the check caches its last result for `Email:HealthCheck:CacheSeconds` (default 60) and serializes probes with a semaphore. Failure is reported as **Degraded** (not Unhealthy) so a flaky relay doesn't trigger pod restart loops — the worker can still consume and retry. Registered on `/health` (detailed/readiness); `/healthz` liveness is unaffected. ## Acceptance criteria - [ ] `GET /health` on the Notifications worker includes an `smtp` check that performs a TLS handshake and (when enabled) an AUTH probe. - [ ] Wrong credentials / unreachable relay surface as `Degraded` with a useful description, not a silent drop. - [ ] Probe frequency to the real SMTP server is bounded by `Email:HealthCheck:CacheSeconds`. - [ ] `Email:HealthCheck:VerifyAuth=false` downgrades to connect + TLS only (no AUTH). - [ ] Unit coverage for config parsing / option toggles; existing tests still pass. Follow-up to #225 / #226.

spikerj commented

2026-06-16 01:02:55 +00:00

Resolved in spikersoft-backend PR #10 (merged to master at 2026-06-16T00:58:01Z).

Added SpikerSoft.Workers.Notifications/HealthChecks/SmtpHealthCheck: a real TLS handshake (same port -> SecureSocketOptions mapping as EmailService) plus an optional AUTH probe (Email:HealthCheck:VerifyAuth, default true). Results are cached for Email:HealthCheck:CacheSeconds (default 60) and serialized with a semaphore to bound SMTP login frequency; failures map to Degraded (not Unhealthy) to avoid restart loops. Registered as a singleton on the worker's /health endpoint (tags smtp, email); /healthz liveness is unaffected. Unit coverage added in NotificationsSmtpHealthCheckTests.

Remaining acceptance items are deploy-time verifications (hit /health against the real relay; break Email:Password in non-prod and confirm Degraded). Closing.

Resolved in spikersoft-backend PR #10 (merged to `master` at 2026-06-16T00:58:01Z). Added `SpikerSoft.Workers.Notifications/HealthChecks/SmtpHealthCheck`: a real TLS handshake (same port -> `SecureSocketOptions` mapping as `EmailService`) plus an optional AUTH probe (`Email:HealthCheck:VerifyAuth`, default `true`). Results are cached for `Email:HealthCheck:CacheSeconds` (default 60) and serialized with a semaphore to bound SMTP login frequency; failures map to Degraded (not Unhealthy) to avoid restart loops. Registered as a singleton on the worker's `/health` endpoint (tags `smtp`, `email`); `/healthz` liveness is unaffected. Unit coverage added in `NotificationsSmtpHealthCheckTests`. Remaining acceptance items are deploy-time verifications (hit `/health` against the real relay; break `Email:Password` in non-prod and confirm Degraded). Closing.

spikerj closed this issue

2026-06-16 01:03:25 +00:00

spikerj referenced this issue

2026-07-12 06:21:26 +00:00

[Bug][Backend] Book deletion leaves all artifacts behind — on disk AND in the MinIO ebooks bucket (no purge path exists) #499

spikerj referenced this issue

2026-07-12 06:30:38 +00:00

[Bug][Backend] Book deletion leaves all artifacts behind — on disk AND in the MinIO ebooks bucket (no purge path exists) #499

spikerj referenced this issue

2026-07-21 02:54:05 +00:00

[Critical][Ops][Notifications] Ops alerting is silently dead — every system-remediation-alert email fails SMTP AUTH 535; critical disk/OOM alerts never delivered #756

spikerj referenced this issue