[Bug] /api/Cluster/nodes 500s in prod — API container can't reach Docker daemon (EADDRNOTAVAIL) #100

New Issue

2026-05-11T01:43:38Z

spikerj commented

2026-05-11 01:43:38 +00:00

Symptom

In production, GET /api/Cluster/nodes returns 500 every time. Seq + container logs show:

System.Net.Http.HttpRequestException: Connection failed
 ---> System.Net.Sockets.SocketException (99): Cannot assign requested address
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(...)
   at Docker.DotNet.DockerClient.<>c__DisplayClass6_0.<<-ctor>b__1>d.MoveNext()
   at Docker.DotNet.SwarmOperations.Docker.DotNet.ISwarmOperations.ListNodesAsync(...)
   at SpikerSoft.Business.Services.DockerClusterTopologyService.GetTopologyAsync(...)

Root cause

The spikersoft-backend API container was wired with Docker.DotNet and configured to talk to the Docker daemon directly:

// SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs
if (string.IsNullOrEmpty(dockerUri))
{
    dockerUri = "tcp://172.17.0.1:2375";
}

In production:

The API container has no /var/run/docker.sock volume mount.
The API runs on a Swarm overlay network where 172.17.0.1 (the host's docker0 bridge gateway) and host.docker.internal (Desktop-only) are unroutable, hence EADDRNOTAVAIL.
Even if it could reach a daemon, exposing the Docker socket to the API replica would give every API RCE root-on-host privileges.

Meanwhile the spikersoft-docker-monitor service was healthy the whole time — it has the socket mounted, refreshes its cache every 30s, and publishes events to RabbitMQ. The API just wasn't using it.

Fix (architectural — option A from chat)

Make the docker-monitor the only service in the swarm with Docker socket access, and have the API consume snapshots through Redis.

SpikerSoft.EventHandlers.DockerMonitor
- New DockerClusterTopologyMapper static helper (Docker.DotNet → neutral ClusterTopologyDto); moved out of the API project.
- DockerCacheService gains Get/SetClusterTopologyAsync writing under SpikerSoft:docker:cluster:topology (5-min TTL, matching the existing services and daemon:info keys).
- DockerMonitorService.RefreshCacheAsync builds and caches a fresh snapshot on every 30s tick.
SpikerSoft.Business.Services.DockerClusterTopologyService
- Rewritten to read from IDistributedCache only — no IDockerClient dependency.
- Stale-snapshot threshold (2 min): logs loudly but still returns the data; better than going dark on a transient monitor outage.
- Throws new ClusterTopologyUnavailableException for missing / corrupt / Redis-down.
SpikerSoft.Api
- Registers IDistributedCache via AddStackExchangeRedisCache with InstanceName = "SpikerSoft:" to match the monitor's key namespace.
- ClusterController.GetTopology maps ClusterTopologyUnavailableException → HTTP 503 (so the dashboard can retry on cold-start instead of showing a hard error).
Tests
- DockerClusterTopologyMapperTests (5 tests) — moved mapping behavior coverage to its new home.
- DockerClusterTopologyServiceTests rewritten (5 tests) — cache hit, missing, corrupt, Redis exception, stale-but-served.
- All 10 pass; full Cluster/DockerSwarm/DockerPhase suite (127 tests) still green.

Effect

API replica no longer needs Docker socket access (no new attack surface).
One privileged consumer of /var/run/docker.sock cluster-wide.
503 for cold-starts is distinguishable from 500 for actual bugs.
Same Redis snapshot can serve a future Kubernetes monitor with no API changes.

Out of scope

The hard-coded tcp://172.17.0.1:2375 fallback in CreateDockerClient is still used by DockerSwarmService (legacy /api/DockerSwarm controller). That path also needs to be moved off of direct Docker.DotNet at some point, but it's not on the hot dashboard path so it isn't blocking.

## Symptom In production, `GET /api/Cluster/nodes` returns 500 every time. Seq + container logs show: ``` System.Net.Http.HttpRequestException: Connection failed ---> System.Net.Sockets.SocketException (99): Cannot assign requested address at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(...) at Docker.DotNet.DockerClient.<>c__DisplayClass6_0.<<-ctor>b__1>d.MoveNext() at Docker.DotNet.SwarmOperations.Docker.DotNet.ISwarmOperations.ListNodesAsync(...) at SpikerSoft.Business.Services.DockerClusterTopologyService.GetTopologyAsync(...) ``` ## Root cause The `spikersoft-backend` API container was wired with `Docker.DotNet` and configured to talk to the Docker daemon directly: ```csharp // SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs if (string.IsNullOrEmpty(dockerUri)) { dockerUri = "tcp://172.17.0.1:2375"; } ``` In production: 1. The API container has **no** `/var/run/docker.sock` volume mount. 2. The API runs on a Swarm overlay network where `172.17.0.1` (the host's `docker0` bridge gateway) and `host.docker.internal` (Desktop-only) are unroutable, hence `EADDRNOTAVAIL`. 3. Even if it could reach a daemon, exposing the Docker socket to the API replica would give every API RCE root-on-host privileges. Meanwhile the `spikersoft-docker-monitor` service was healthy the whole time — it has the socket mounted, refreshes its cache every 30s, and publishes events to RabbitMQ. The API just wasn't using it. ## Fix (architectural — option A from chat) Make the docker-monitor the *only* service in the swarm with Docker socket access, and have the API consume snapshots through Redis. 1. **`SpikerSoft.EventHandlers.DockerMonitor`** - New `DockerClusterTopologyMapper` static helper (Docker.DotNet → neutral `ClusterTopologyDto`); moved out of the API project. - `DockerCacheService` gains `Get/SetClusterTopologyAsync` writing under `SpikerSoft:docker:cluster:topology` (5-min TTL, matching the existing `services` and `daemon:info` keys). - `DockerMonitorService.RefreshCacheAsync` builds and caches a fresh snapshot on every 30s tick. 2. **`SpikerSoft.Business.Services.DockerClusterTopologyService`** - Rewritten to read from `IDistributedCache` only — no `IDockerClient` dependency. - Stale-snapshot threshold (2 min): logs loudly but still returns the data; better than going dark on a transient monitor outage. - Throws new `ClusterTopologyUnavailableException` for missing / corrupt / Redis-down. 3. **`SpikerSoft.Api`** - Registers `IDistributedCache` via `AddStackExchangeRedisCache` with `InstanceName = "SpikerSoft:"` to match the monitor's key namespace. - `ClusterController.GetTopology` maps `ClusterTopologyUnavailableException` → HTTP 503 (so the dashboard can retry on cold-start instead of showing a hard error). 4. **Tests** - `DockerClusterTopologyMapperTests` (5 tests) — moved mapping behavior coverage to its new home. - `DockerClusterTopologyServiceTests` rewritten (5 tests) — cache hit, missing, corrupt, Redis exception, stale-but-served. - All 10 pass; full Cluster/DockerSwarm/DockerPhase suite (127 tests) still green. ## Effect - API replica no longer needs Docker socket access (no new attack surface). - One privileged consumer of `/var/run/docker.sock` cluster-wide. - 503 for cold-starts is distinguishable from 500 for actual bugs. - Same Redis snapshot can serve a future Kubernetes monitor with no API changes. ## Out of scope - The hard-coded `tcp://172.17.0.1:2375` fallback in `CreateDockerClient` is still used by `DockerSwarmService` (legacy `/api/DockerSwarm` controller). That path also needs to be moved off of direct Docker.DotNet at some point, but it's not on the hot dashboard path so it isn't blocking.

spikerj closed this issue

2026-05-11 04:18:23 +00:00

spikerj referenced this issue

2026-07-02 15:06:55 +00:00

[Spike] Completion-ratio unlocks for anonymous users — contribute to see more; account unlocks everything #329

spikerj referenced this issue

2026-07-02 15:07:18 +00:00

[Ratio unlocks 2/2] Frontend: anon lesson mode, horizon locks, claim celebration, funnel events #332

spikerj referenced this issue

2026-07-05 11:46:34 +00:00

[NodeAgent/SystemRemediation] Host disk-space probe — detect/alert on low disk before it stalls services #381

spikerj referenced this issue