spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

Host-level remediation + security monitoring across the swarm #287

New Issue
Closed
opened 2026-06-30 13:18:37 +00:00 by spikerj · 1 comment
spikerj commented 2026-06-30 13:18:37 +00:00
Owner
Copy Link
Copy Source

Problem

The 9-node swarm (7 arm64 Jetson dreamstream1-dreamstream7, amd64 SERVER + laptop-server) has no host/OS-level self-healing or intrusion detection. System-level issues (disk/inode pressure, failed systemd units, stuck mounts, time-sync drift, OOM, thermal throttling) and host-level attacks (SSH brute force, sudo abuse, new listeners/users, authorized_keys changes, kernel firewall drops) are invisible until something breaks.

Proposal

A two-layer, host/OS-level capability built entirely in the existing EventHandlerHostBuilder + Serilog/Seq + RabbitMQ + Redis + Mongo + Jaeger idiom, reusing spikersoft-notifications for email/SMS. No Grafana/Loki/Prometheus introduced - Seq is the console.

  • SpikerSoft.EventHandlers.NodeAgent (global, privileged, every node incl. arm64): tails host journald and probes host health via nsenter, ships HostJournalEvent / HostHealthSnapshot to the system.events exchange, and executes a curated, rate-limited, whitelisted remediation command set.
  • SpikerSoft.EventHandlers.SystemRemediation (1 replica, SERVER): rules engine over journal/health signals, SystemIncident read-model in Mongo (+ Redis), operator notifications, and SAFE auto-remediation dispatch.
  • SpikerSoft.EventHandlers.SecurityMonitor (1 replica, SERVER): stateful Redis detectors (SSH brute force, root/new-source login, sudo abuse, firewall drops, kernel anomalies, host inventory changes) emitting structured SecurityAlerts to Seq + notifications; opt-in IP auto-block (off by default).

Curated SAFE auto-remediation set

vacuum journald / free disk caches when a FS or inode is near full; reset-failed + restart a crash-looping/failed unit (capped); restart a failed mount or time-sync unit. Everything else (reboots, service/node restarts, thermal/OOM root-cause, IP block) is alert/recommend only.

Rollout (safe by default, toggled via appsettings.Production.json)

  1. Observe-only - NodeAgent ships journal/health; no command path.
  2. Recommend-only - SystemRemediation detects + alerts; dispatches nothing.
  3. Enable SAFE auto-remediation behind per-action toggles.
  4. SecurityMonitor alert-first; IP auto-block opt-in.

Scope (one issue, two PRs)

  • spikersoft-backend: 3 new event-handler projects + shared contracts + system.events/system.commands topology + multi-arch CI workflows + per-project & root README docs.
  • spikersoft-infrastructure: 3 swarm stacks (node-agent global/privileged, remediation + security pinned to SERVER) + persistent-journald prerequisite script.

Prerequisites / call-outs

  • Privileged global agent is the main tradeoff; mitigated by strict whitelist + rate limits, full Seq/Mongo audit, no inbound route, SAFE-set actions only.
  • Persistent journald required on every node.
  • Swarm caveat: docker stack deploy ignores privileged/pid:host; cap_add is honoured (stack lists the caps nsenter needs). Agent degrades gracefully if host access is unavailable.
## Problem The 9-node swarm (7 arm64 Jetson `dreamstream1`-`dreamstream7`, amd64 `SERVER` + `laptop-server`) has no host/OS-level self-healing or intrusion detection. System-level issues (disk/inode pressure, failed systemd units, stuck mounts, time-sync drift, OOM, thermal throttling) and host-level attacks (SSH brute force, sudo abuse, new listeners/users, `authorized_keys` changes, kernel firewall drops) are invisible until something breaks. ## Proposal A two-layer, host/OS-level capability built entirely in the existing `EventHandlerHostBuilder` + Serilog/Seq + RabbitMQ + Redis + Mongo + Jaeger idiom, reusing `spikersoft-notifications` for email/SMS. **No Grafana/Loki/Prometheus introduced** - Seq is the console. - **`SpikerSoft.EventHandlers.NodeAgent`** (global, privileged, every node incl. arm64): tails host `journald` and probes host health via `nsenter`, ships `HostJournalEvent` / `HostHealthSnapshot` to the `system.events` exchange, and executes a curated, rate-limited, whitelisted remediation command set. - **`SpikerSoft.EventHandlers.SystemRemediation`** (1 replica, SERVER): rules engine over journal/health signals, `SystemIncident` read-model in Mongo (+ Redis), operator notifications, and SAFE auto-remediation dispatch. - **`SpikerSoft.EventHandlers.SecurityMonitor`** (1 replica, SERVER): stateful Redis detectors (SSH brute force, root/new-source login, sudo abuse, firewall drops, kernel anomalies, host inventory changes) emitting structured `SecurityAlert`s to Seq + notifications; opt-in IP auto-block (off by default). ## Curated SAFE auto-remediation set vacuum journald / free disk caches when a FS or inode is near full; `reset-failed` + restart a crash-looping/failed unit (capped); restart a failed mount or time-sync unit. Everything else (reboots, service/node restarts, thermal/OOM root-cause, IP block) is alert/recommend only. ## Rollout (safe by default, toggled via `appsettings.Production.json`) 1. Observe-only - NodeAgent ships journal/health; no command path. 2. Recommend-only - SystemRemediation detects + alerts; dispatches nothing. 3. Enable SAFE auto-remediation behind per-action toggles. 4. SecurityMonitor alert-first; IP auto-block opt-in. ## Scope (one issue, two PRs) - `spikersoft-backend`: 3 new event-handler projects + shared contracts + `system.events`/`system.commands` topology + multi-arch CI workflows + per-project & root README docs. - `spikersoft-infrastructure`: 3 swarm stacks (node-agent global/privileged, remediation + security pinned to SERVER) + persistent-journald prerequisite script. ## Prerequisites / call-outs - Privileged global agent is the main tradeoff; mitigated by strict whitelist + rate limits, full Seq/Mongo audit, no inbound route, SAFE-set actions only. - Persistent journald required on every node. - Swarm caveat: `docker stack deploy` ignores `privileged`/`pid:host`; `cap_add` is honoured (stack lists the caps `nsenter` needs). Agent degrades gracefully if host access is unavailable.
spikerj commented 2026-06-30 13:46:44 +00:00
Author
Owner
Copy Link
Copy Source

Follow-up: production alert recipients are now set before merge.

  • SystemRemediation → Alerts:Email:To = team@spikersoft.com
  • SecurityMonitor → SecurityAlerts:Email:To = team@spikersoft.com

Pushed to the backend branch as commit cf0eb83 (part of PR spikerj/spikersoft-backend#35). SMS escalation remains disabled (Sms:Enabled=false) for both. Notifications still only fire once the rollout posture is advanced (NodeAgent Commands:Enabled, SystemRemediation Mode=auto).

Follow-up: production alert recipients are now set before merge. - `SystemRemediation` → `Alerts:Email:To` = `team@spikersoft.com` - `SecurityMonitor` → `SecurityAlerts:Email:To` = `team@spikersoft.com` Pushed to the backend branch as commit `cf0eb83` (part of PR spikerj/spikersoft-backend#35). SMS escalation remains disabled (`Sms:Enabled=false`) for both. Notifications still only fire once the rollout posture is advanced (NodeAgent `Commands:Enabled`, SystemRemediation `Mode=auto`).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#287