[Bug][Backend][DockerMonitor][Hardware live] RefreshCacheAsync UNLINKs docker:cluster:topology ~0.6 s before re-writing it — the API's 5 s hardware-live poll hits the hole ~11×/h and drops the swarm overlay #1190
Reference in new issue
Block a user
Summary
SpikerSoft.DockerMonitordeletes theSpikerSoft:docker:cluster:topologyRedis key and re-writes it ~0.6 s later on every 30 s refresh. Since the[Hardware live]work landed (backend #1307, deployed 2026-10-08 ~01Z), the API polls that key every 5 s, so it lands in the hole roughly 11 times an hour. Each hit makesHardwareLiveSweepServicecallSetTopology(null)— the swarm overlay on/admin/hardwaredrops to "telemetry only" for one sweep — and logs two warnings to Seq.Not on the board; found by swarm recon on 2026-10-08.
Evidence
Seq —
Cluster topology snapshot not yet present in cache (key: docker:cluster:topology)(sourceDockerClusterTopologyService, emitted bySpikerSoft.Apion SERVER), paired 1:1 withHardware live sweep: topology snapshot unavailable (...); nodes shown from telemetry only:/api/Cluster/nodesreads only)228 occurrences in the 72 h to 15:03Z; 190 warnings in the last 8 h.
The key is fine.
redis-node-2(master, slots 5461-10922) holds it with TTL 272/300 s, fieldsabsexp/sldexp/data, 429 KB payload; replicaredis-node-6mirrors it. ACL userspikersofthas~* +@all.Redis MONITOR on redis-node-2 (15:10Z) — API (
192.168.0.165:49108) reads every 5 s; docker-monitor (192.168.0.165:41270) refresh at :79.66 → :80.24:Hole = UNLINK→HMSET ≈ 560 ms (the daemon list-nodes/services/tasks round trip). 5 s poll × 0.56 s / 30 s ≈ 1.9 % of reads → ~13/h expected, ~11/h observed.
Where
SpikerSoft.EventHandlers.DockerMonitor/Services/DockerMonitorService.csRefreshCacheAsync(≈ line 149):await _cacheService.ClearAllAsync()first, thenGetAllServicesAsync,GetDaemonInfoAsync,RefreshClusterTopologyAsync.SpikerSoft.EventHandlers.DockerMonitor/Services/DockerCacheService.csClearAllAsync(≈ line 212) removesdocker:services:all,docker:daemon:infoanddocker:cluster:topology.RefreshClusterTopologyAsyncswallows errors ("next refresh tick will retry") — so when the daemon call fails (seen 04:52Zrpc error ... EOF, 14:32ZTaskCanceledduring the Xavier re-provisioning) the key has already been unlinked and stays absent for a full 30 s, not just 0.6 s.SpikerSoft.Business/Services/DockerClusterTopologyService.cs→ throwsClusterTopologyUnavailableExceptionon null;SpikerSoft.Business/Domain/Hardware/Live/HardwareLiveSweepService.cs≈ line 102 →live.SetTopology(null, now)+ warning (latched per streak).Correlation
/api/Cluster/nodes.[Hardware live] ingest endpoint + HardwareHub push feedadded the 5 s offline sweep that reads the topology every tick (HardwareLiveSweepService,Live.OfflineSweepSeconds).Error caching cluster topology — Command cannot be issued to a replica: HMSET SpikerSoft:docker:cluster:topology(a Redis cluster failover; StackExchange.Redis re-mapped within the hour, no recurrence).Impact
/admin/hardwarelive feed: swarm node/workload overlay flickers out for one sweep (~5 s) ~11×/h; shows "telemetry only" nodes meanwhile.absexpor a pub/sub nudge.Fix options
cluster:topologyinClearAllAsync—HMSET+EXPIREalready replaces it atomically enough; keep the 5-min TTL as the staleness guard (it is whatStaleSnapshotThresholdin the reader already expects). Same forservices:all/daemon:infoif anything else polls them.null, keep the previous snapshot if it is younger thanStaleSnapshotThresholdinstead ofSetTopology(null).Reproduce / observe
Seq:
@MessageTemplate like 'Cluster topology snapshot not yet present%'— ~11/h. Orredis-cli --tls --insecure -h redis-node-2 -p 6380 --user spikersoft MONITOR | grep topologyfrom a container on theredisoverlay and watch the UNLINK→HMSET gap every 30 s.Moved to spikerj/spikersoft-backend#1663 (still live 2026-10-30, ~10/h); the fix PR closes it there.