[Infra] redis-cluster: make persistence real — add --dir /data (volumes are currently decoys) + initiator/coverage hygiene #373

New Issue

2026-07-05T03:14:56Z

spikerj commented

2026-07-05 03:14:56 +00:00

Permanent fix for #371. The per-node /docker-local/redis-node-N-data:/data:rw mounts exist but redis never writes to them: the stack command: overrides the image entrypoint without --dir /data, and the image's WorkingDir is / — so nodes.conf/AOF live in the ephemeral container layer and every restart mints a new node identity.

Add --dir /data to all six node commands (verify beforehand on hosts: ls /docker-local/redis-node-N-data — expect empty, confirming diagnosis)
Explicit restart_policy on node services
cluster_initiator: one-shot (restart_policy: condition: none) or remove + document manual init — today it re-runs --cluster create --cluster-yes on every redeploy and could mint a fresh topology against a wiped cluster
Decide cluster-require-full-coverage (cache-class store: no keeps surviving slots serving during a partial outage)
Delete or mark-legacy the broken dormant backup.yml + 7000–7005/ dirs (smart-quote typos)
Rolling apply (one node at a time, verify cluster_state:ok between) + /mnt/infrastructure pull; add the #371 ghost-heal runbook to the repo README

Permanent fix for #371. The per-node `/docker-local/redis-node-N-data:/data:rw` mounts exist but redis never writes to them: the stack `command:` overrides the image entrypoint without `--dir /data`, and the image's WorkingDir is `/` — so `nodes.conf`/AOF live in the ephemeral container layer and every restart mints a new node identity. - [ ] Add `--dir /data` to all six node commands (verify beforehand on hosts: `ls /docker-local/redis-node-N-data` — expect empty, confirming diagnosis) - [ ] Explicit `restart_policy` on node services - [ ] `cluster_initiator`: one-shot (`restart_policy: condition: none`) or remove + document manual init — today it re-runs `--cluster create --cluster-yes` on every redeploy and could mint a fresh topology against a wiped cluster - [ ] Decide `cluster-require-full-coverage` (cache-class store: `no` keeps surviving slots serving during a partial outage) - [ ] Delete or mark-legacy the broken dormant `backup.yml` + `7000–7005/` dirs (smart-quote typos) - [ ] Rolling apply (one node at a time, verify `cluster_state:ok` between) + `/mnt/infrastructure` pull; add the #371 ghost-heal runbook to the repo README

spikerj added the bug label 2026-07-05 03:14:56 +00:00

spikerj commented

2026-07-05 06:33:30 +00:00

Code fix resolved in spikersoft-infrastructure PR #10 (merged to master): --dir /data on all six nodes, one-shot cluster_initiator, --cluster-require-full-coverage no, explicit node restart policies, and a README with the persistence rationale + ghost-heal runbook + first-deploy procedure. docker compose config validates.

Not yet deployed — takes effect only on the deliberate first-deploy cut-over (a naive rolling restart would re-trigger #371 one node at a time; use the re-init or pre-seed procedure in the PR/README). The currently-running hand-healed cluster is unaffected until then. Closing the code fix; the ops cut-over is tracked under the incident #371. Automated detection/heal is #376 (in progress).

Code fix resolved in spikersoft-infrastructure PR #10 (merged to `master`): `--dir /data` on all six nodes, one-shot `cluster_initiator`, `--cluster-require-full-coverage no`, explicit node restart policies, and a README with the persistence rationale + ghost-heal runbook + first-deploy procedure. `docker compose config` validates. **Not yet deployed** — takes effect only on the deliberate first-deploy cut-over (a naive rolling restart would re-trigger #371 one node at a time; use the re-init or pre-seed procedure in the PR/README). The currently-running hand-healed cluster is unaffected until then. Closing the code fix; the ops cut-over is tracked under the incident #371. Automated detection/heal is #376 (in progress).

spikerj closed this issue

2026-07-05 06:33:31 +00:00

spikerj referenced this issue

2026-07-17 18:40:09 +00:00

Security: blog body sanitization uses bypassable regex (stored XSS) — replace with HtmlSanitizer #674

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#373