[Infra][Node] dreamstream6 (aarch64) down since ~16:30Z 2026-07-13 — heartbeat failure, host unreachable #539
Assignees
spikerj (Joseph Spiker)
Clear assignees
spikerj (Joseph Spiker)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: spikerj/spikersoft-issues#539
Reference in New Issue
Block a user
QA log watch, 2026-07-13 ~18:40Z.
Observed:
docker node lsshows dreamstream6 Down/Unreachable.docker node inspect: Status=down, Message='heartbeat failure', Addr=192.168.0.110, arch aarch64. ICMP to 192.168.0.110: 100% packet loss — the HOST is off the network, not just dockerd (power/kernel/NIC-level, unlike SERVER's #503 dockerd flaps).When: ~16:30–16:40Z — swarm rescheduled spikersoft-coderunner.2 to dreamstream1 at ~16:37Z, and node tasks stopped confirming state around then.
Blast radius (tasks stranded in stale 'Running' on the node):
redis-cluster_redis-node-6— redis cluster is running one node short; verify cluster health/slot coverage after the node rejoinsgitea-runners_arm_v8_2_act_runner— ARM CI capacity reducedjetson-influxdb_jetson-influx-6— metrics gap for this nodespikersoft-coderunner.2— already rescheduled (self-healed, briefly 5/4)spikersoft-node-agentinstance (9/8 until swarm ages out the stale task)Ask: physically check/power-cycle dreamstream6; after rejoin verify redis-cluster health and that the arm_v8 runner re-registers. No duplicate ticket exists (searched 'dreamstream'). Not SERVER-flap fallout — different node, different failure signature.
QA watch 2026-07-13 ~19:41Z — dreamstream6 REJOINED at ~19:38Z (outage ~16:30→19:38Z, ≈3h). Host pings again, node State=ready.
Workload recovery: redis-cluster_redis-node-6 Running (one 'swarm not active' Reject during rejoin, expected), jetson-influx-6 Running, node-agent/coderunner stale-task counts cleaned up (back to 8/8, 4/4). One straggler: gitea-runners_arm_v8_2_act_runner is crash-looping exit(1) ~every 20s since rejoin (4+ attempts). Same signature during the 07-13 laptop-server outage turned out to be 'gitea unreachable' and self-healed — gitea is healthy now, so giving it a few minutes; will confirm either way next pass.
Still unknown: root cause of the 3h host outage (power/kernel/NIC — nothing in swarm explains it; needs the node owner's eyes on the host's own logs, e.g. journalctl -b -1, when convenient). Recommend closing once the arm runner stabilizes + a root-cause note is added; also worth verifying redis cluster slot coverage (redis-cli cluster info) if anything looked odd during the window.
QA watch 2026-07-13 ~19:47Z — arm runner crash-loop root-caused: CONTAINER DNS, not gitea. Every restart logs: 'fail to invoke Declare … lookup git.spikersoft.com on 127.0.0.11:53 … i/o timeout' — Docker's embedded DNS inside the task's sandbox can't reach its upstream resolver since the host rejoined, so the runner daemon exits(1) on registration, ~every 20s, 10+ attempts. redis-node-6/jetson-influx-6 are fine (no name lookups needed), so this is likely stale DNS/netns state from the reboot rather than host-wide DNS death.
Fix options (host actions, not doable from the watch node): (a) restart dockerd on dreamstream6, or (b) docker service update --force gitea-runners_arm_v8_2_act_runner to get a fresh sandbox — (b) is cheapest if the host's own /etc/resolv.conf works. Note the #538 local-DNS-records work would also make git.spikersoft.com resolution on LAN more robust generally.
Side effect visible in CI: with the arm_v8_2 runner down, the post-#533-fix rebuild wave shows create_manifest run 10886 FAILED 19:43Z (multi-arch manifest needs the arm64 image half) while amd64 builds go green. Arm CI capacity is degraded until this runner re-registers.
QA watch 2026-07-13 ~19:52Z — concrete CI impact of the dead arm runner: backend's new code is NOT live despite green-looking runs. Sequence: urgent-fix build pushed spikersoft-backend:amd64 + :arm64v8 arch tags at 19:47Z, but create_manifest (run 10886, 19:43Z) FAILED and :latest is still the 00:27Z manifest — so the deploy step's service update was a silent no-op (spec still pins digest 1cb647f6ea7d from 00:27; service UpdatedAt 17:01Z; task 3h old). The #533/#530/#540 backend changes won't reach the API until arm64 builds + create_manifest complete, and arm capacity is degraded while arm_v8_2 crash-loops on the DNS issue above.
Also worth a look when triaging: the deploy job reported SUCCESS while create_manifest had failed and :latest was unchanged — a no-op deploy that looks green. If that's the job DAG (deploy not gated on create_manifest), it deserves a small CI fix so a stale-manifest deploy fails loudly instead. I'll confirm once the current arm builds (10902/10903) land whether manifest+deploy recover on their own.
Correction + resolution of the CI thread (2026-07-13 ~19:57Z): the API's 'silent no-op deploy' I described was me misreading interleaved task lists — spikersoft-api.yml has NO deploy job at all (chain is builds → create_manifest → notify=git-tag+Discord; only the worker services embeddings/decompile/artpipe-processor/image-description/trellis3d/quiz-generation have CI deploy jobs, gated with needs:). The 19:47Z 'deploy success' was a worker service.
Current true state: backend arm64 build completed once it got a working arm runner slot, create_manifest went GREEN at 19:53Z and spikersoft-backend:latest is now the new multi-arch image — so the #533/#530 API image is ready and just waits on the normal (out-of-band) API deploy process; service spec still 17:01Z/digest 1cb647f6 as expected until then. One create_manifest (10919, 19:53Z) did fail for another service — arm capacity is still degraded while arm_v8_2 crash-loops on the container-DNS issue, which remains the actionable item on this ticket.
QA watch 2026-07-13 ~20:57Z — redis cluster health after rejoin: VERIFIED OK. Cluster gossip shows no persistent failures; one replica briefly entered FAIL at 20:48:44Z and cleared at 20:49:40Z ('replica is reachable again') — 56s blip, self-healed, no cluster-state change, masters/slots unaffected. That closes the redis item from this ticket.
Remaining before close: (1) the arm_v8_2 runner is STILL crash-looping on the container-DNS issue (~85 min now — dockerd restart on ds6 or a force-update of the runner service still needed); (2) optional root-cause note for the 3h host outage from the node owner.
QA Team — watch 2026-07-14 ~02:20Z:
docker node lsshows Ready/Active.keycloak_postgres-keycloakonto ds6 and it was rejected 3x withNo such image: postgres:16@sha256:be01cf82...— the node couldn't pull a plain Docker Hub image by digest right after rejoining. So the container-DNS issue isn't limited to the arm_v8_2 runner; it affected fresh image pulls on the host too.QA Team — watch 2026-07-14 ~02:45Z:
gitea-runners_arm_v8_2_act_runneris still crash-looping on dreamstream6 (task: non-zero exit (1)every ~20s, currently 0/1) — roughly 6 hours since it was first flagged in this thread. The dockerd restart / force-update on ds6 called out earlier still hasn't happened. arm build capacity remains degraded.Recovered. The node is Ready / Active / Reachable and running tasks again; all ten nodes are reachable with a stable leader.
Closing on observed state, with the same caveat I left on #552: the root cause was not captured. The host came back on its own and nothing recorded why the heartbeat failed. Diagnosing a node flap after the node has returned is diagnosing an absence of evidence — which is precisely the gap #555 tracks (no node-down alerts; kernel alerts arrive after recovery and carry no kernel message). That is where the durable fix belongs.
— macbook-claude-session