[health-checks] HTTP probes have no timeout → a down dependency hangs /health up to ~75s #422

New Issue

2026-07-05T22:33:24Z

spikerj commented

2026-07-05 22:33:24 +00:00

Verified finding (surfaced live in the local API log)

The HTTP-based health checks are registered via AddHttpClient<T>() with no .Timeout, and none of the AddCheck<>() registrations set a per-check timeout:. So a slow/unreachable dependency hangs the check on the HttpClient default (100s) — or the OS TCP connect timeout (~75s) hits first.

Observed live: TileServerGL health check ... Unhealthy completed after 75007.674ms ... 'Operation timed out (192.168.0.202:18081)' — a 75-second hang on one probe.

Affected (in `ServiceCollectionExtensions.AddHealthChecksConfiguration`)

Keycloak_HealthCheck, InfluxDB_HealthCheck, RabbitMQ_HealthCheck, TileServerGlHealthCheck, DnsServerHealthCheck — all AddHttpClient<T>() with no timeout. (DnsServerHealthCheck already bounds its own call internally, but its client was still unbounded.)

Impact

/health aggregates all checks, so one hung HTTP probe can stall the endpoint ~75s. At scale that trips orchestrator/load-balancer health probes and makes the API look down when only a peripheral dependency (tile server, InfluxDB) is degraded.

Fix (in flight)

Bound each health-check HttpClient to 10s in the registration, well under any orchestrator health-check timeout — a down dependency now reports Unhealthy in ~10s instead of ~75s. Adds a [Theory] regression test asserting all five clients are bounded. Fix build-clean + test green.

## Verified finding (surfaced live in the local API log) The HTTP-based health checks are registered via `AddHttpClient<T>()` with **no `.Timeout`**, and none of the `AddCheck<>()` registrations set a per-check `timeout:`. So a slow/unreachable dependency hangs the check on the HttpClient default (100s) — or the OS TCP connect timeout (~75s) hits first. **Observed live:** `TileServerGL health check ... Unhealthy completed after 75007.674ms ... 'Operation timed out (192.168.0.202:18081)'` — a **75-second** hang on one probe. ### Affected (in `ServiceCollectionExtensions.AddHealthChecksConfiguration`) `Keycloak_HealthCheck`, `InfluxDB_HealthCheck`, `RabbitMQ_HealthCheck`, `TileServerGlHealthCheck`, `DnsServerHealthCheck` — all `AddHttpClient<T>()` with no timeout. (`DnsServerHealthCheck` already bounds its own call internally, but its client was still unbounded.) ### Impact `/health` aggregates all checks, so one hung HTTP probe can stall the endpoint ~75s. At scale that trips orchestrator/load-balancer health probes and makes the API look down when only a peripheral dependency (tile server, InfluxDB) is degraded. ## Fix (in flight) Bound each health-check HttpClient to **10s** in the registration, well under any orchestrator health-check timeout — a down dependency now reports Unhealthy in ~10s instead of ~75s. Adds a `[Theory]` regression test asserting all five clients are bounded. Fix build-clean + test green.

spikerj commented

2026-07-05 22:34:14 +00:00

Resolved in spikersoft-backend PR #104 (merged to master). Bound all five HTTP health-check probes (Keycloak/InfluxDB/RabbitMQ/TileServerGL/DnsServer) to a 10s HttpClient.Timeout in AddHealthChecksConfiguration, so a down/slow dependency reports Unhealthy in ~10s instead of hanging /health ~75s. Added HealthCheckHttpClientTimeoutTests (Theory, 5/5 green) as a regression guard. Closing.

Resolved in spikersoft-backend PR #104 (merged to `master`). Bound all five HTTP health-check probes (`Keycloak`/`InfluxDB`/`RabbitMQ`/`TileServerGL`/`DnsServer`) to a 10s `HttpClient.Timeout` in `AddHealthChecksConfiguration`, so a down/slow dependency reports Unhealthy in ~10s instead of hanging `/health` ~75s. Added `HealthCheckHttpClientTimeoutTests` (Theory, 5/5 green) as a regression guard. Closing.

spikerj closed this issue

2026-07-05 22:34:21 +00:00

spikerj referenced this issue

2026-07-19 01:14:48 +00:00

[Epic][Blog][UX] Blog management & UX overhaul — findings from the 2026-07-18 full review #705

spikerj referenced this issue

2026-07-19 01:15:00 +00:00

[Bug][Blog][Angular] Media delete is broken end-to-end — placeholder handler AND a confirm button with no click binding (epic #705) #706

spikerj referenced this issue

2026-07-19 01:16:08 +00:00

[Epic][Blog][UX] Blog management & UX overhaul — findings from the 2026-07-18 full review #705

spikerj referenced this issue