[Bug][Backend][DockerMonitor][Hardware live] RefreshCacheAsync UNLINKs docker:cluster:topology ~0.6 s before re-writing it — the API's 5 s hardware-live poll hits the hole ~11×/h and drops the swarm overlay #1190

Closed
opened 2026-10-08 15:14:20 +00:00 by spikerj · 1 comment
Owner

Summary

SpikerSoft.DockerMonitor deletes the SpikerSoft:docker:cluster:topology Redis key and re-writes it ~0.6 s later on every 30 s refresh. Since the [Hardware live] work landed (backend #1307, deployed 2026-10-08 ~01Z), the API polls that key every 5 s, so it lands in the hole roughly 11 times an hour. Each hit makes HardwareLiveSweepService call SetTopology(null) — the swarm overlay on /admin/hardware drops to "telemetry only" for one sweep — and logs two warnings to Seq.

Not on the board; found by swarm recon on 2026-10-08.

Evidence

Seq — Cluster topology snapshot not yet present in cache (key: docker:cluster:topology) (source DockerClusterTopologyService, emitted by SpikerSoft.Api on SERVER), paired 1:1 with Hardware live sweep: topology snapshot unavailable (...); nodes shown from telemetry only:

hour (UTC) count
10-06 .. 10-07 1–5 per day (on-demand /api/Cluster/nodes reads only)
10-08T01 17
10-08T02 39 (raft-storm window)
10-08T03..13 10–20 every hour, steady ~11/h
10-08T14 19

228 occurrences in the 72 h to 15:03Z; 190 warnings in the last 8 h.

The key is fine. redis-node-2 (master, slots 5461-10922) holds it with TTL 272/300 s, fields absexp/sldexp/data, 429 KB payload; replica redis-node-6 mirrors it. ACL user spikersoft has ~* +@all.

Redis MONITOR on redis-node-2 (15:10Z) — API (192.168.0.165:49108) reads every 5 s; docker-monitor (192.168.0.165:41270) refresh at :79.66 → :80.24:

1791472277.243 [49108] "HMGET"  "SpikerSoft:docker:cluster:topology" "absexp" "sldexp" "data"
1791472279.660 [41270] "UNLINK" "SpikerSoft:docker:cluster:topology"
1791472280.220 [41270] "HMSET"  "SpikerSoft:docker:cluster:topology" "absexp" "6392706938..." "sldexp" "-1" "data" "{\"Provider\":\"docker\",...
1791472280.241 [41270] "EXPIRE" "SpikerSoft:docker:cluster:topology" "300"
1791472282.242 [49108] "HMGET"  "SpikerSoft:docker:cluster:topology" "absexp" "sldexp" "data"

Hole = UNLINK→HMSET ≈ 560 ms (the daemon list-nodes/services/tasks round trip). 5 s poll × 0.56 s / 30 s ≈ 1.9 % of reads → ~13/h expected, ~11/h observed.

Where

  • SpikerSoft.EventHandlers.DockerMonitor/Services/DockerMonitorService.cs RefreshCacheAsync (≈ line 149): await _cacheService.ClearAllAsync() first, then GetAllServicesAsync, GetDaemonInfoAsync, RefreshClusterTopologyAsync.
  • SpikerSoft.EventHandlers.DockerMonitor/Services/DockerCacheService.cs ClearAllAsync (≈ line 212) removes docker:services:all, docker:daemon:info and docker:cluster:topology.
  • RefreshClusterTopologyAsync swallows errors ("next refresh tick will retry") — so when the daemon call fails (seen 04:52Z rpc error ... EOF, 14:32Z TaskCanceled during the Xavier re-provisioning) the key has already been unlinked and stays absent for a full 30 s, not just 0.6 s.
  • Reader: SpikerSoft.Business/Services/DockerClusterTopologyService.cs → throws ClusterTopologyUnavailableException on null; SpikerSoft.Business/Domain/Hardware/Live/HardwareLiveSweepService.cs ≈ line 102 → live.SetTopology(null, now) + warning (latched per streak).

Correlation

  • Pre-existing race in docker-monitor (code unchanged since before 10-04); it was invisible while the only reader was the on-demand /api/Cluster/nodes.
  • Became continuous when backend #1307 [Hardware live] ingest endpoint + HardwareHub push feed added the 5 s offline sweep that reads the topology every tick (HardwareLiveSweepService, Live.OfflineSweepSeconds).
  • Not related to the 2026-10-07 23:48Z burst of 125 × Error caching cluster topology — Command cannot be issued to a replica: HMSET SpikerSoft:docker:cluster:topology (a Redis cluster failover; StackExchange.Redis re-mapped within the hour, no recurrence).

Impact

  • /admin/hardware live feed: swarm node/workload overlay flickers out for one sweep (~5 s) ~11×/h; shows "telemetry only" nodes meanwhile.
  • 2 Seq warnings per hit (~260/day) burying real ones.
  • Worse under daemon hiccups: topology missing for 30 s instead of 0.6 s.
  • Side note: the API now pulls the 429 KB snapshot every 5 s ≈ 86 KB/s ≈ 7 GB/day of Redis traffic on the overlay — worth an in-process cache keyed on absexp or a pub/sub nudge.

Fix options

  1. Stop removing cluster:topology in ClearAllAsync — HMSET+EXPIRE already replaces it atomically enough; keep the 5-min TTL as the staleness guard (it is what StaleSnapshotThreshold in the reader already expects). Same for services:all / daemon:info if anything else polls them.
  2. Or build first, write, then clear only what was not rewritten.
  3. Reader side (defence): on null, keep the previous snapshot if it is younger than StaleSnapshotThreshold instead of SetTopology(null).

Reproduce / observe

Seq: @MessageTemplate like 'Cluster topology snapshot not yet present%' — ~11/h. Or redis-cli --tls --insecure -h redis-node-2 -p 6380 --user spikersoft MONITOR | grep topology from a container on the redis overlay and watch the UNLINK→HMSET gap every 30 s.

## Summary `SpikerSoft.DockerMonitor` deletes the `SpikerSoft:docker:cluster:topology` Redis key and re-writes it ~0.6 s later on **every 30 s refresh**. Since the `[Hardware live]` work landed (backend #1307, deployed 2026-10-08 ~01Z), the API polls that key **every 5 s**, so it lands in the hole roughly 11 times an hour. Each hit makes `HardwareLiveSweepService` call `SetTopology(null)` — the swarm overlay on `/admin/hardware` drops to "telemetry only" for one sweep — and logs two warnings to Seq. Not on the board; found by swarm recon on 2026-10-08. ## Evidence **Seq** — `Cluster topology snapshot not yet present in cache (key: docker:cluster:topology)` (source `DockerClusterTopologyService`, emitted by `SpikerSoft.Api` on SERVER), paired 1:1 with `Hardware live sweep: topology snapshot unavailable (...); nodes shown from telemetry only`: | hour (UTC) | count | |---|---| | 10-06 .. 10-07 | 1–5 per day (on-demand `/api/Cluster/nodes` reads only) | | 10-08T01 | 17 | | 10-08T02 | 39 (raft-storm window) | | 10-08T03..13 | 10–20 every hour, steady ~11/h | | 10-08T14 | 19 | 228 occurrences in the 72 h to 15:03Z; 190 warnings in the last 8 h. **The key is fine.** `redis-node-2` (master, slots 5461-10922) holds it with TTL 272/300 s, fields `absexp/sldexp/data`, 429 KB payload; replica `redis-node-6` mirrors it. ACL user `spikersoft` has `~* +@all`. **Redis MONITOR on redis-node-2 (15:10Z)** — API (`192.168.0.165:49108`) reads every 5 s; docker-monitor (`192.168.0.165:41270`) refresh at :79.66 → :80.24: ``` 1791472277.243 [49108] "HMGET" "SpikerSoft:docker:cluster:topology" "absexp" "sldexp" "data" 1791472279.660 [41270] "UNLINK" "SpikerSoft:docker:cluster:topology" 1791472280.220 [41270] "HMSET" "SpikerSoft:docker:cluster:topology" "absexp" "6392706938..." "sldexp" "-1" "data" "{\"Provider\":\"docker\",... 1791472280.241 [41270] "EXPIRE" "SpikerSoft:docker:cluster:topology" "300" 1791472282.242 [49108] "HMGET" "SpikerSoft:docker:cluster:topology" "absexp" "sldexp" "data" ``` Hole = UNLINK→HMSET ≈ 560 ms (the daemon list-nodes/services/tasks round trip). 5 s poll × 0.56 s / 30 s ≈ 1.9 % of reads → ~13/h expected, ~11/h observed. ## Where - `SpikerSoft.EventHandlers.DockerMonitor/Services/DockerMonitorService.cs` `RefreshCacheAsync` (≈ line 149): `await _cacheService.ClearAllAsync()` **first**, then `GetAllServicesAsync`, `GetDaemonInfoAsync`, `RefreshClusterTopologyAsync`. - `SpikerSoft.EventHandlers.DockerMonitor/Services/DockerCacheService.cs` `ClearAllAsync` (≈ line 212) removes `docker:services:all`, `docker:daemon:info` **and** `docker:cluster:topology`. - `RefreshClusterTopologyAsync` swallows errors ("next refresh tick will retry") — so when the daemon call fails (seen 04:52Z `rpc error ... EOF`, 14:32Z `TaskCanceled` during the Xavier re-provisioning) the key has already been unlinked and stays **absent for a full 30 s**, not just 0.6 s. - Reader: `SpikerSoft.Business/Services/DockerClusterTopologyService.cs` → throws `ClusterTopologyUnavailableException` on null; `SpikerSoft.Business/Domain/Hardware/Live/HardwareLiveSweepService.cs` ≈ line 102 → `live.SetTopology(null, now)` + warning (latched per streak). ## Correlation - Pre-existing race in docker-monitor (code unchanged since before 10-04); it was invisible while the only reader was the on-demand `/api/Cluster/nodes`. - Became continuous when backend #1307 `[Hardware live] ingest endpoint + HardwareHub push feed` added the 5 s offline sweep that reads the topology every tick (`HardwareLiveSweepService`, `Live.OfflineSweepSeconds`). - Not related to the 2026-10-07 23:48Z burst of 125 × `Error caching cluster topology — Command cannot be issued to a replica: HMSET SpikerSoft:docker:cluster:topology` (a Redis cluster failover; StackExchange.Redis re-mapped within the hour, no recurrence). ## Impact - `/admin/hardware` live feed: swarm node/workload overlay flickers out for one sweep (~5 s) ~11×/h; shows "telemetry only" nodes meanwhile. - 2 Seq warnings per hit (~260/day) burying real ones. - Worse under daemon hiccups: topology missing for 30 s instead of 0.6 s. - Side note: the API now pulls the 429 KB snapshot every 5 s ≈ 86 KB/s ≈ 7 GB/day of Redis traffic on the overlay — worth an in-process cache keyed on `absexp` or a pub/sub nudge. ## Fix options 1. Stop removing `cluster:topology` in `ClearAllAsync` — `HMSET`+`EXPIRE` already replaces it atomically enough; keep the 5-min TTL as the staleness guard (it is what `StaleSnapshotThreshold` in the reader already expects). Same for `services:all` / `daemon:info` if anything else polls them. 2. Or build first, write, then clear only what was not rewritten. 3. Reader side (defence): on `null`, keep the previous snapshot if it is younger than `StaleSnapshotThreshold` instead of `SetTopology(null)`. ## Reproduce / observe Seq: `@MessageTemplate like 'Cluster topology snapshot not yet present%'` — ~11/h. Or `redis-cli --tls --insecure -h redis-node-2 -p 6380 --user spikersoft MONITOR | grep topology` from a container on the `redis` overlay and watch the UNLINK→HMSET gap every 30 s.
Author
Owner

Moved to spikerj/spikersoft-backend#1663 (still live 2026-10-30, ~10/h); the fix PR closes it there.

Moved to spikerj/spikersoft-backend#1663 (still live 2026-10-30, ~10/h); the fix PR closes it there.
Sign in to join this conversation.