spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[SystemRemediation] Redis cluster auto-remediation — InfrastructureProbe detection + signature-gated ghost-heal runbook (dry-run first) #376

New Issue
Open
opened 2026-07-05 03:16:00 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-05 03:16:00 +00:00
Owner
Copy Link
Copy Source

The auto-remediation home decision from the #371 investigation. SystemRemediation, not NodeAgent: (1) cluster healing is a singleton decision — SystemRemediation is already the single-replica rule engine; (2) the heal is pure Redis protocol over TCP (FORGET/MEET/ADDSLOTSRANGE/REPLICATE) — SystemRemediation sits on the redis overlay and needs zero host privilege, while NodeAgent's actuator is nsenter+systemd whitelist; (3) NodeAgent is currently the least reliable layer (#374/#375); (4) blast-radius: keep cluster surgery out of the most-privileged component. NodeAgent remains the actuator for genuinely host-scoped actions only.

Phase D1 — detection: new InfrastructureProbe hosted service polling CLUSTER INFO/CLUSTER NODES (~30 s): typed RedisClusterState findings (state, ghost ids, orphaned slot ranges, per-endpoint known_nodes) into the existing incident pipeline (Mongo system_incidents, alerting w/ cooldowns). Do NOT route via API /healthz — that signal stays human-facing (nothing machine-consumes it today; if the API stack ever gets a swarm healthcheck, keep liveness separate from dependency readiness so a Redis outage can't restart-loop the API).
Phase D2 — dry-run runbook: the #371 ghost-heal as a typed runbook with per-step verification; posture recommend logs/stores the exact command plan without executing (matches the existing Mode/DispatchCommands discipline).
Phase D3 — gated execution: exact-signature match ONLY (ghosts :0@0-disconnected AND same endpoints re-present as empty cluster_known_nodes:1 nodes AND surviving majority consistent — anything else stays alert-only), data-loss policy flag per store class (redis = cache-class → slot loss acceptable; hard-refuse otherwise), cooldown + max-attempts per incident, abort-on-divergence, full audit through the existing incident/alert path.

Visibility: incidents + dry-run plans surface in the #297 dev-ops console; school operators get the separate capability-status view (#377).

The auto-remediation home decision from the #371 investigation. **SystemRemediation, not NodeAgent**: (1) cluster healing is a singleton decision — SystemRemediation is already the single-replica rule engine; (2) the heal is pure Redis protocol over TCP (`FORGET/MEET/ADDSLOTSRANGE/REPLICATE`) — SystemRemediation sits on the `redis` overlay and needs zero host privilege, while NodeAgent's actuator is nsenter+systemd whitelist; (3) NodeAgent is currently the least reliable layer (#374/#375); (4) blast-radius: keep cluster surgery out of the most-privileged component. NodeAgent remains the actuator for genuinely host-scoped actions only. **Phase D1 — detection:** new `InfrastructureProbe` hosted service polling `CLUSTER INFO`/`CLUSTER NODES` (~30 s): typed `RedisClusterState` findings (state, ghost ids, orphaned slot ranges, per-endpoint known_nodes) into the existing incident pipeline (Mongo `system_incidents`, alerting w/ cooldowns). Do NOT route via API `/healthz` — that signal stays human-facing (nothing machine-consumes it today; if the API stack ever gets a swarm healthcheck, keep liveness separate from dependency readiness so a Redis outage can't restart-loop the API). **Phase D2 — dry-run runbook:** the #371 ghost-heal as a typed runbook with per-step verification; posture `recommend` logs/stores the exact command plan without executing (matches the existing Mode/DispatchCommands discipline). **Phase D3 — gated execution:** exact-signature match ONLY (ghosts `:0@0`-disconnected AND same endpoints re-present as empty `cluster_known_nodes:1` nodes AND surviving majority consistent — anything else stays alert-only), **data-loss policy flag per store class** (redis = cache-class → slot loss acceptable; hard-refuse otherwise), cooldown + max-attempts per incident, abort-on-divergence, full audit through the existing incident/alert path. Visibility: incidents + dry-run plans surface in the #297 dev-ops console; school operators get the separate capability-status view (#377).
spikerj added the enhancement label 2026-07-05 03:16:00 +00:00
spikerj commented 2026-07-05 06:54:13 +00:00
Author
Owner
Copy Link
Copy Source

D1+D2 merged in spikersoft-backend PR #85 (to master). Observe-only Redis cluster auto-remediation now lives in SystemRemediation:

  • D1 detection — RedisClusterProbe polls CLUSTER INFO/CLUSTER NODES (30s) via read-only IRedisClusterInspector; pure RedisClusterParser → typed state with :0@0 ghost detection + orphaned-range computation; degraded → dedup'd RedisClusterDegraded incident + alert (existing cooldown), healthy → resolves.
  • D2 dry-run runbook — RedisHealPlanBuilder produces the ordered #371 heal plan (FORGET→MEET→ADDSLOTSRANGE→REPLICATE) stored on incident.ProposedPlan; RedisHealSignature.Evaluate is a pure would-auto-heal predicate (exact-signature + cache-class policy). No execution path wired — confirmed by test (dispatch never called).

61/61 SystemRemediation tests. Keeping #376 open for D3 — gated live CLUSTER surgery behind Mode=auto + policy + cooldown + abort-on-divergence. Runtime follow-up flagged: if a live cluster rejects CLUSTER commands, add allowAdmin=true to the SystemRemediation Redis:ConnectionString.

**D1+D2 merged** in spikersoft-backend PR #85 (to `master`). Observe-only Redis cluster auto-remediation now lives in SystemRemediation: - **D1 detection** — `RedisClusterProbe` polls `CLUSTER INFO`/`CLUSTER NODES` (30s) via read-only `IRedisClusterInspector`; pure `RedisClusterParser` → typed state with `:0@0` ghost detection + orphaned-range computation; degraded → dedup'd `RedisClusterDegraded` incident + alert (existing cooldown), healthy → resolves. - **D2 dry-run runbook** — `RedisHealPlanBuilder` produces the ordered #371 heal plan (FORGET→MEET→ADDSLOTSRANGE→REPLICATE) stored on `incident.ProposedPlan`; `RedisHealSignature.Evaluate` is a pure would-auto-heal predicate (exact-signature + cache-class policy). **No execution path wired** — confirmed by test (dispatch never called). 61/61 SystemRemediation tests. **Keeping #376 open for D3** — gated live CLUSTER surgery behind `Mode=auto` + policy + cooldown + abort-on-divergence. Runtime follow-up flagged: if a live cluster rejects `CLUSTER` commands, add `allowAdmin=true` to the SystemRemediation `Redis:ConnectionString`.
spikerj commented 2026-07-22 02:55:32 +00:00
Author
Owner
Copy Link
Copy Source

Board-sweep status (2026-07-22): D1 detection + D2 dry-run merged (PR #85 — RedisClusterProbe + RedisHealPlanBuilder verified in tree). REMAINING: signature-gated heal EXECUTION (observe-only today).

Board-sweep status (2026-07-22): D1 detection + D2 dry-run merged (PR #85 — RedisClusterProbe + RedisHealPlanBuilder verified in tree). REMAINING: signature-gated heal EXECUTION (observe-only today).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#376