spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][Incident] Redis cluster amputation 2026-07-05 — nodes .106/.107/.108 restarted with fresh identities; add persistent volumes for nodes.conf/data #371

New Issue
Closed
opened 2026-07-05 02:59:34 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-05 02:59:34 +00:00
Owner
Copy Link
Copy Source

Incident (resolved): api.spikersoft.com/healthz reported Redis Cluster: Unhealthy, cluster_state: fail — slots 10923–16383 orphaned. Diagnosis: redis nodes 192.168.0.106/.107/.108 had restarted without persisted cluster state (nodes.conf/data), rejoining as brand-new empty single-node "clusters" (cluster_known_nodes: 1) while their old identities lingered as slot-owning :0@0 ghosts in the survivors' gossip.

Heal applied (Joey-approved, 2026-07-05): CLUSTER FORGET × 3 ghosts on each survivor → CLUSTER MEET .106/.107/.108 → CLUSTER ADDSLOTSRANGE 10923 16383 on .106 → CLUSTER REPLICATE .107→.110, .108→.106. Result: cluster_state ok on all 6 nodes, 16384/16384 slots, 3 masters + 3 replicas, healthz Redis → Healthy. Data in the orphaned range was already lost (cache/rate-limit class — acceptable).

Prevention (the actual ticket):

  • redis-cluster stack: persistent volume per node for /data (nodes.conf + RDB/AOF as desired) so restarts keep cluster identity — a container restart must never mint a new node ID
  • Verify restart_policy + placement pinning per node (identity ↔ host ↔ volume must stay together)
  • Runbook note in the infra README: the ghost-heal procedure above, for the next time
  • Consider cluster-require-full-coverage no evaluation: with it default-yes, losing one master fails ALL slots' availability for cluster-aware ops — decide policy deliberately

Collateral (separate action): RediSearch book_chunks_idx was on the wiped nodes → healthz Vector Search: Degraded. Recreate via POST api/vector-indexes (Admin) + re-embed, or the embeddings worker's normal flow if it self-heals. TileServerGL Unhealthy is unrelated/pre-existing (map-tile-server 0/1 since earlier today).

**Incident (resolved):** `api.spikersoft.com/healthz` reported `Redis Cluster: Unhealthy, cluster_state: fail` — slots 10923–16383 orphaned. Diagnosis: redis nodes 192.168.0.106/.107/.108 had restarted **without persisted cluster state** (`nodes.conf`/data), rejoining as brand-new empty single-node "clusters" (`cluster_known_nodes: 1`) while their old identities lingered as slot-owning `:0@0` ghosts in the survivors' gossip. **Heal applied (Joey-approved, 2026-07-05):** `CLUSTER FORGET` × 3 ghosts on each survivor → `CLUSTER MEET` .106/.107/.108 → `CLUSTER ADDSLOTSRANGE 10923 16383` on .106 → `CLUSTER REPLICATE` .107→.110, .108→.106. Result: **cluster_state ok on all 6 nodes, 16384/16384 slots, 3 masters + 3 replicas, healthz Redis → Healthy.** Data in the orphaned range was already lost (cache/rate-limit class — acceptable). **Prevention (the actual ticket):** - [ ] `redis-cluster` stack: persistent volume per node for `/data` (nodes.conf + RDB/AOF as desired) so restarts keep cluster identity — a container restart must never mint a new node ID - [ ] Verify restart_policy + placement pinning per node (identity ↔ host ↔ volume must stay together) - [ ] Runbook note in the infra README: the ghost-heal procedure above, for the next time - [ ] Consider `cluster-require-full-coverage no` evaluation: with it default-yes, losing one master fails ALL slots' availability for cluster-aware ops — decide policy deliberately **Collateral (separate action):** RediSearch `book_chunks_idx` was on the wiped nodes → healthz `Vector Search: Degraded`. Recreate via `POST api/vector-indexes` (Admin) + re-embed, or the embeddings worker's normal flow if it self-heals. TileServerGL Unhealthy is unrelated/pre-existing (`map-tile-server` 0/1 since earlier today).
spikerj added the bug label 2026-07-05 02:59:34 +00:00
spikerj commented 2026-07-05 03:14:13 +00:00
Author
Owner
Copy Link
Copy Source

Root cause corrected after code/image investigation — the volumes exist but are decoys.

redis-cluster/docker-stack.yml DOES bind-mount /docker-local/redis-node-N-data:/data:rw per node — but the stack's command: fully replaces the image entrypoint with a raw redis-server --port 6379 … that never sets --dir /data. The redis/redis-stack-server image has WorkingDir: None (= /) and the entrypoint is overridden, so redis writes nodes.conf and the AOF into the ephemeral container layer at /. Every container restart mints a fresh filesystem → fresh node ID → exactly this amputation. Verification: ls /docker-local/redis-node-N-data on the hosts — expect empty.

The permanent fix is six one-line changes (add --dir /data to each node's command) plus hygiene, now ticketed as #373:

  • explicit restart_policy on the node services (currently implicit any)
  • cluster_initiator hazard: it re-runs redis-cli --cluster create --cluster-yes on every redeploy and retries forever under default restart policy — against a simultaneously-wiped cluster it could mint a whole new topology. Make it one-shot (restart_policy: condition: none) or remove it and document manual init.
  • decide cluster-require-full-coverage (default yes = one lost master fails ALL slots; for a cache-class store no is worth evaluating)
  • delete/mark-legacy the broken dormant backup.yml + 7000–7005/ dirs (smart-quote typos; deploying it would be its own incident)
  • remember: /mnt/infrastructure must be pulled for any of this to deploy.

Applied heal runbook (2026-07-05, for the README): on each survivor CLUSTER FORGET <ghost-id> ×3 → from a survivor CLUSTER MEET <ip> 6379 for each rejoining node → CLUSTER ADDSLOTSRANGE <orphaned-range> on a fresh master → CLUSTER REPLICATE <master-id> to re-pair replicas → verify cluster_state:ok + cluster_slots_ok:16384 on all nodes.

Auto-remediation architecture (Joey's follow-up question) is documented and ticketed separately: detection + signature-gated healing belongs in SystemRemediation (already a singleton, posture-gated runbook engine on the redis overlay — the heal is pure Redis protocol, no host privilege needed), NOT NodeAgent (per-node scope, host-command actuator, and currently the least reliable layer — see #374/#375). School-operator visibility gets its own thin capability-status surface (#377) separate from the #297 admin console. Full layering table in #376.

**Root cause corrected after code/image investigation — the volumes exist but are decoys.** `redis-cluster/docker-stack.yml` DOES bind-mount `/docker-local/redis-node-N-data:/data:rw` per node — but the stack's `command:` fully replaces the image entrypoint with a raw `redis-server --port 6379 …` that **never sets `--dir /data`**. The `redis/redis-stack-server` image has `WorkingDir: None` (= `/`) and the entrypoint is overridden, so redis writes `nodes.conf` and the AOF **into the ephemeral container layer at `/`**. Every container restart mints a fresh filesystem → fresh node ID → exactly this amputation. Verification: `ls /docker-local/redis-node-N-data` on the hosts — expect empty. **The permanent fix is six one-line changes** (add `--dir /data` to each node's command) **plus hygiene**, now ticketed as #373: - explicit `restart_policy` on the node services (currently implicit `any`) - **`cluster_initiator` hazard:** it re-runs `redis-cli --cluster create --cluster-yes` on every redeploy and retries forever under default restart policy — against a simultaneously-wiped cluster it could mint a whole new topology. Make it one-shot (`restart_policy: condition: none`) or remove it and document manual init. - decide `cluster-require-full-coverage` (default yes = one lost master fails ALL slots; for a cache-class store `no` is worth evaluating) - delete/mark-legacy the broken dormant `backup.yml` + `7000–7005/` dirs (smart-quote typos; deploying it would be its own incident) - remember: `/mnt/infrastructure` must be pulled for any of this to deploy. **Applied heal runbook (2026-07-05, for the README):** on each survivor `CLUSTER FORGET <ghost-id>` ×3 → from a survivor `CLUSTER MEET <ip> 6379` for each rejoining node → `CLUSTER ADDSLOTSRANGE <orphaned-range>` on a fresh master → `CLUSTER REPLICATE <master-id>` to re-pair replicas → verify `cluster_state:ok` + `cluster_slots_ok:16384` on all nodes. **Auto-remediation architecture** (Joey's follow-up question) is documented and ticketed separately: detection + signature-gated healing belongs in **SystemRemediation** (already a singleton, posture-gated runbook engine on the `redis` overlay — the heal is pure Redis protocol, no host privilege needed), NOT NodeAgent (per-node scope, host-command actuator, and currently the least reliable layer — see #374/#375). School-operator visibility gets its own thin capability-status surface (#377) separate from the #297 admin console. Full layering table in #376.
spikerj commented 2026-07-11 23:47:46 +00:00
Author
Owner
Copy Link
Copy Source

All four prevention-checklist items are now shipped, closing. Verified in redis-cluster/docker-stack.yml: per-node /docker-local/redis-node-N-data:/data volumes (infra PR #10) so restarts keep nodes.conf/cluster identity; restart_policy condition: any + per-node hostname pins keeping identity↔host↔volume together; --cluster-require-full-coverage no set deliberately on all six nodes so surviving slots keep serving during a partial outage. The final item — the ghost-heal runbook — landed in infra PR #24 (merged): README Common operations now records the FORGET → MEET → ADDSLOTSRANGE → REPLICATE procedure plus the RediSearch-index collateral check. Closing.

All four prevention-checklist items are now shipped, closing. Verified in redis-cluster/docker-stack.yml: per-node /docker-local/redis-node-N-data:/data volumes (infra PR #10) so restarts keep nodes.conf/cluster identity; restart_policy condition: any + per-node hostname pins keeping identity↔host↔volume together; --cluster-require-full-coverage no set deliberately on all six nodes so surviving slots keep serving during a partial outage. The final item — the ghost-heal runbook — landed in infra PR #24 (merged): README Common operations now records the FORGET → MEET → ADDSLOTSRANGE → REPLICATE procedure plus the RediSearch-index collateral check. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#371