2026-07-29 power outage: RabbitMQ mnesia corruption took down all consumers (recovered) #895

New Issue

2026-07-29T13:40:28Z

spikerj commented

2026-07-29 13:40:28 +00:00

What happened

Power outage ~07:30-13:00Z (whole swarm). All 10 nodes rejoined cleanly, but every RabbitMQ consumer crash-looped afterward (spikersoft-backend exit 139, EventHandler hosts exiting on StartAndTuneAsync failure).

Root cause

The broker's mnesia DB on GlusterFS (/mnt/rabbitmq/mnesia, ds4) was corrupted by the power loss: broker booted and accepted TCP/AMQP handshakes, but its user table was empty (rabbitmqctl list_users → 0 rows, only ~1 queue left) and internal state was broken (add_user → Error: noproc). Every service got PLAIN login refused: user 'dockerUser' - invalid credentials. Because the node wasn't "fresh", the entrypoint never re-created the default user from RABBITMQ_DEFAULT_USER/PASS.

Note: overlay DNS + gluster mount were healthy — this was NOT a repeat of #885's DNS record loss, despite identical consumer symptoms.

Remediation (done 13:32-13:35Z)

docker service scale rabbitmq_rabbitmq=0
On ds4: mv /mnt/rabbitmq/mnesia /mnt/rabbitmq/mnesia.corrupt-20260729 (backup kept, can delete after a few days)
Scale back to 1 → fresh boot created dockerUser, consumers reconnected (49 successful auths in first 2 min, 0 refused).
Force-updated 12 services stuck in Complete (won't self-reschedule): gpu-coordinator, docker-monitor, upload-coordinator, gameserver, metadata-extractor, security-monitor, security-scanner, system-remediation, decompile, artpipe-photostack, clamav, node-agent.
book-management + influx-dashboard were Rejected: No such image (digest) — re-resolved digests via service update --image :latest --force.

Verified: learn.spikersoft.com 200, api.spikersoft.com/healthz 200, backend 1/1, all mongo/redis/keycloak/minio/bao stacks 1/1 (bao unsealed).

Lost with the broker DB

Queued-but-unprocessed messages from before the outage (delayed-exchange messages included). If anything looks unprocessed (notifications, media jobs), that's why.

Follow-ups

artpipe-model-blender still Reject-looping on missing image — pre-existing #887, unchanged.
Delete /mnt/rabbitmq/mnesia.corrupt-20260729 after a week.
Consider rabbitmqctl export_definitions cron → gluster, so a corrupt DB restore doesn't rely on env defaults (topology/users would reimport).
Alerting still blind (#756) — nobody was paged for this either.

## What happened Power outage ~07:30-13:00Z (whole swarm). All 10 nodes rejoined cleanly, but **every RabbitMQ consumer crash-looped** afterward (`spikersoft-backend` exit 139, EventHandler hosts exiting on `StartAndTuneAsync` failure). ## Root cause The broker's mnesia DB on GlusterFS (`/mnt/rabbitmq/mnesia`, ds4) was **corrupted by the power loss**: broker booted and accepted TCP/AMQP handshakes, but its user table was empty (`rabbitmqctl list_users` → 0 rows, only ~1 queue left) and internal state was broken (`add_user` → `Error: noproc`). Every service got `PLAIN login refused: user 'dockerUser' - invalid credentials`. Because the node wasn't "fresh", the entrypoint never re-created the default user from `RABBITMQ_DEFAULT_USER/PASS`. Note: overlay DNS + gluster mount were healthy — this was NOT a repeat of #885's DNS record loss, despite identical consumer symptoms. ## Remediation (done 13:32-13:35Z) 1. `docker service scale rabbitmq_rabbitmq=0` 2. On ds4: `mv /mnt/rabbitmq/mnesia /mnt/rabbitmq/mnesia.corrupt-20260729` (backup kept, can delete after a few days) 3. Scale back to 1 → fresh boot created `dockerUser`, consumers reconnected (49 successful auths in first 2 min, 0 refused). 4. Force-updated 12 services stuck in `Complete` (won't self-reschedule): gpu-coordinator, docker-monitor, upload-coordinator, gameserver, metadata-extractor, security-monitor, security-scanner, system-remediation, decompile, artpipe-photostack, clamav, node-agent. 5. book-management + influx-dashboard were `Rejected: No such image (digest)` — re-resolved digests via `service update --image :latest --force`. Verified: learn.spikersoft.com 200, api.spikersoft.com/healthz 200, backend 1/1, all mongo/redis/keycloak/minio/bao stacks 1/1 (bao unsealed). ## Lost with the broker DB Queued-but-unprocessed messages from before the outage (delayed-exchange messages included). If anything looks unprocessed (notifications, media jobs), that's why. ## Follow-ups - [ ] `artpipe-model-blender` still Reject-looping on missing image — pre-existing #887, unchanged. - [ ] Delete `/mnt/rabbitmq/mnesia.corrupt-20260729` after a week. - [ ] Consider `rabbitmqctl export_definitions` cron → gluster, so a corrupt DB restore doesn't rely on env defaults (topology/users would reimport). - [ ] Alerting still blind (#756) — nobody was paged for this either.

spikerj referenced this issue

2026-07-29 15:21:20 +00:00

ArtPipeProcessor host has no RabbitMQ startup retry/backoff — 7s crash-loop + Seq flood whenever the broker is away #886

spikerj referenced this issue

2026-07-29 15:43:40 +00:00

[Go-Live Checklist][Security] Rotate + Bao-migrate the dev-environment secrets before production #633

spikerj referenced this issue

2026-07-29 15:59:56 +00:00

[Bug][Prod][Data-loss] 10 messages are permanently dead in DLQs with ZERO consumers — 5 of them are real user quizzes killed by #553, now replayable #589

spikerj referenced this issue

2026-07-29 16:02:56 +00:00

[Critical][Ops][Notifications] Ops alerting is silently dead — every system-remediation-alert email fails SMTP AUTH 535; critical disk/OOM alerts never delivered #756