OUTAGE: dreamstream4 down → RabbitMQ backbone unreachable since 12:17Z — consumers can't connect, redis node lost #885

New Issue

2026-07-28T14:24:08Z

spikerj commented

2026-07-28 14:24:08 +00:00

Severity: critical — the cluster message bus is down and alerting is blind (#756).

Timeline / evidence (2026-07-28, UTC)

12:17:22Z — first Failed to create RabbitMQ connection in Seq (RabbitMQ.Client.Exceptions.BrokerUnreachableException → SocketException: Name or service not known for host rabbitmq). Continuous every ~7s since.
docker node ls: dreamstream4 = Status Down, manager Unreachable (192.168.0.108).
docker service ps rabbitmq_rabbitmq: old task "Running 4 days ago" on ds4 (unconfirmable), replacement Pending ~2h — "no suitable node" because the stack pins node.hostname == dreamstream4 (rabbitmq/docker-stack.yml). No running task ⇒ swarm drops the rabbitmq DNS record ⇒ every connect fails name resolution (verified: getent hosts rabbitmq fails inside containers on the rabbitmq overlay, incl. long-running instantmesh).
Node half-alive: answers ping; ports 22/2377/5672/15672 accept TCP but never answer (HTTP 000, AMQP handshake dead — stale IPVS/ingress rules on a dead backend). Redis host-mode port 6379 on .108 closed → 6-node redis cluster degraded to 5.
No repo/deploy correlation: today's merges (QR-art artpipe deploys ~04:11Z, sponsor PRs) touched nothing on ds4 or the broker. Hardware/daemon failure.

Impact

All MassTransit/RabbitMQ traffic is dead: API publishes, notifications, keycloak/game event forwarding, book/quiz/media processing, delayed redelivery. Long-running consumers hold dead connections (silent); anything that (re)starts crash-loops.
SpikerSoft.EventHandlers.ArtPipeProcessor (resident artpipe-model-safety on 4090) restart-loops every ~7s, ~1500+ Seq error triplets and counting (separate hardening ticket).
Swarm lost 1 of its managers (quorum still intact) and the jetson-monitor on ds4.
Nobody was paged: alert email has been dead since ≥07-19 (#756).

Remediation (in order)

SSH to dreamstream4 (192.168.0.108) and restart dockerd (sudo systemctl restart docker), or power-cycle the Jetson. Broker data is safe on GlusterFS /mnt/rabbitmq; the pinned task reschedules automatically once the node is Ready. MassTransit consumers reconnect on their own; the safety crash-looper self-heals.
If ds4 won't come back: re-pin the broker — edit rabbitmq/docker-stack.yml constraint to another light Jetson (ds3 is the natural pick per the stack file's own placement rationale), git pull /mnt/infrastructure, docker stack deploy -c rabbitmq/docker-stack.yml rabbitmq. /mnt/rabbitmq is GlusterFS-shared and hostname: rabbitmq keeps the mnesia dir name, so state survives. Verify ds4's containers are actually dead first (they appear to be — AMQP handshake gets no reply locally) to rule out a split-brain double-broker on the shared mnesia dir.
Afterwards: check redis cluster state (cluster info — .108 is one of the 6 node IPs) and let ds4 rejoin.

Follow-ups this exposes: broker is a pinned SPOF with no failover story; #756 (dead alert mail) turned a 2-hour backbone outage into a silent one.

**Severity: critical — the cluster message bus is down and alerting is blind (#756).** ## Timeline / evidence (2026-07-28, UTC) - **12:17:22Z** — first `Failed to create RabbitMQ connection` in Seq (`RabbitMQ.Client.Exceptions.BrokerUnreachableException` → `SocketException: Name or service not known` for host `rabbitmq`). Continuous every ~7s since. - `docker node ls`: **dreamstream4 = Status Down, manager Unreachable** (192.168.0.108). - `docker service ps rabbitmq_rabbitmq`: old task "Running 4 days ago" on ds4 (unconfirmable), replacement **Pending ~2h — "no suitable node"** because the stack pins `node.hostname == dreamstream4` (rabbitmq/docker-stack.yml). No running task ⇒ swarm drops the `rabbitmq` DNS record ⇒ every connect fails name resolution (verified: `getent hosts rabbitmq` fails inside containers on the rabbitmq overlay, incl. long-running instantmesh). - Node half-alive: answers ping; ports 22/2377/5672/15672 **accept TCP but never answer** (HTTP 000, AMQP handshake dead — stale IPVS/ingress rules on a dead backend). **Redis host-mode port 6379 on .108 closed** → 6-node redis cluster degraded to 5. - No repo/deploy correlation: today's merges (QR-art artpipe deploys ~04:11Z, sponsor PRs) touched nothing on ds4 or the broker. Hardware/daemon failure. ## Impact - All MassTransit/RabbitMQ traffic is dead: API publishes, notifications, keycloak/game event forwarding, book/quiz/media processing, delayed redelivery. Long-running consumers hold dead connections (silent); anything that (re)starts crash-loops. - `SpikerSoft.EventHandlers.ArtPipeProcessor` (resident artpipe-model-safety on 4090) restart-loops every ~7s, ~1500+ Seq error triplets and counting (separate hardening ticket). - Swarm lost 1 of its managers (quorum still intact) and the jetson-monitor on ds4. - **Nobody was paged: alert email has been dead since ≥07-19 (#756).** ## Remediation (in order) 1. **SSH to dreamstream4 (192.168.0.108) and restart dockerd** (`sudo systemctl restart docker`), or power-cycle the Jetson. Broker data is safe on GlusterFS `/mnt/rabbitmq`; the pinned task reschedules automatically once the node is Ready. MassTransit consumers reconnect on their own; the safety crash-looper self-heals. 2. **If ds4 won't come back**: re-pin the broker — edit `rabbitmq/docker-stack.yml` constraint to another light Jetson (ds3 is the natural pick per the stack file's own placement rationale), `git pull` `/mnt/infrastructure`, `docker stack deploy -c rabbitmq/docker-stack.yml rabbitmq`. `/mnt/rabbitmq` is GlusterFS-shared and `hostname: rabbitmq` keeps the mnesia dir name, so state survives. **Verify ds4's containers are actually dead first** (they appear to be — AMQP handshake gets no reply locally) to rule out a split-brain double-broker on the shared mnesia dir. 3. Afterwards: check redis cluster state (`cluster info` — .108 is one of the 6 node IPs) and let ds4 rejoin. Follow-ups this exposes: broker is a pinned SPOF with no failover story; #756 (dead alert mail) turned a 2-hour backbone outage into a silent one.

spikerj referenced this issue

2026-07-28 14:24:31 +00:00

ArtPipeProcessor host has no RabbitMQ startup retry/backoff — 7s crash-loop + Seq flood whenever the broker is away #886

spikerj referenced this issue

2026-07-28 14:24:32 +00:00

artpipe-model-blender:latest missing from registry — service Reject-loops every ~5s since the 04:11Z artpipe redeploy #887

spikerj commented

2026-07-28 14:45:57 +00:00

Resolved 2026-07-28 ~14:42Z after the node was brought back online.

Recovery verification:

dreamstream4 Ready/Reachable again; rabbitmq_rabbitmq task Running on ds4 (data intact via GlusterFS). Overlay DNS for rabbitmq restored (VIP 10.0.5.208 + task IP — the two records are VIP+endpoint, not a split-brain; verified via docker network inspect rabbitmq -v).
ArtPipeProcessor (artpipe-model-safety) logged RabbitMQ connection created successfully and has been Running since ~14:31Z — crash-loop over. Redis .108:6379 answering again.
Second casualty found during recovery: the OpenBao raft cluster (ds1/ds3/ds4) was LEADERLESS during/after the outage — AppRole logins failed HTTP 500 local node not active but active cluster node not found, which is what broke the 14:11–14:34Z master CI runs (bao-secrets step) in both app repos. openbao-1 became active at 14:42:10Z; login path verified healthy (probe now returns a clean 400). Failed master runs re-triggered (backend 18925/18924, angular 18927/18928) — these are the deploy pipelines for backend#502 / angular#598.

Remaining follow-ups tracked separately: #886 (ArtPipeProcessor startup retry/backoff), #887 (blender image missing — unrelated, still broken). Worth considering a small follow-up on why raft didn't elect a new leader with 2/3 nodes up during the 12:17–14:42 window (suspect: leader was ds4's instance and a standby wasn't unsealed/vote-eligible). Closing as the outage itself is resolved. — Opus 5 Agent

Resolved 2026-07-28 ~14:42Z after the node was brought back online. **Recovery verification:** - dreamstream4 Ready/Reachable again; rabbitmq_rabbitmq task Running on ds4 (data intact via GlusterFS). Overlay DNS for `rabbitmq` restored (VIP 10.0.5.208 + task IP — the two records are VIP+endpoint, not a split-brain; verified via `docker network inspect rabbitmq -v`). - ArtPipeProcessor (artpipe-model-safety) logged `RabbitMQ connection created successfully` and has been Running since ~14:31Z — crash-loop over. Redis .108:6379 answering again. - **Second casualty found during recovery:** the OpenBao raft cluster (ds1/ds3/ds4) was LEADERLESS during/after the outage — AppRole logins failed HTTP 500 `local node not active but active cluster node not found`, which is what broke the 14:11–14:34Z master CI runs (bao-secrets step) in both app repos. openbao-1 became active at 14:42:10Z; login path verified healthy (probe now returns a clean 400). Failed master runs re-triggered (backend 18925/18924, angular 18927/18928) — these are the deploy pipelines for backend#502 / angular#598. Remaining follow-ups tracked separately: #886 (ArtPipeProcessor startup retry/backoff), #887 (blender image missing — unrelated, still broken). Worth considering a small follow-up on why raft didn't elect a new leader with 2/3 nodes up during the 12:17–14:42 window (suspect: leader was ds4's instance and a standby wasn't unsealed/vote-eligible). Closing as the outage itself is resolved. — Opus 5 Agent

spikerj closed this issue

2026-07-28 14:45:57 +00:00

spikerj referenced this issue

2026-07-28 14:50:45 +00:00

OpenBao HA is silently degraded by sealed standbys — seal state has no monitoring, turning one node loss into a 2.5h write outage (#885 post-mortem) #888

spikerj referenced this issue

2026-07-28 15:23:04 +00:00

Sponsor family card: hover member tabs (locations/bio/awards) + dashboard sponsor grid layout #884

spikerj referenced this issue

2026-07-29 13:40:28 +00:00

2026-07-29 power outage: RabbitMQ mnesia corruption took down all consumers (recovered) #895

spikerj referenced this issue