spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[NodeAgent] Restore real host access — journalctl in-image reading the mounted journal (de-privileged) vs per-node compose deployment #375

New Issue
Closed
opened 2026-07-05 03:15:39 +00:00 by spikerj · 1 comment
spikerj commented 2026-07-05 03:15:39 +00:00
Owner
Copy Link
Copy Source

Decision ticket from the #371 investigation. Under swarm deploy NodeAgent has NO real host access (see #374), so the entire host-signal pipeline — journal events for SecurityMonitor's six detectors and SystemRemediation's crash-loop/disk rules, plus honest HostHealthProbe data — has never actually flowed in prod. The Commands actuation path (whitelisted remediations) is also dark (Commands:Enabled=false AND no host reach).

Option A (recommended for the journal path): de-privilege. Install systemd (journalctl binary) in the image and read the ALREADY-MOUNTED host journal directly: journalctl -D /var/log/journal -o json -f. No nsenter, no pid:host — works under plain stack deploy, keeps swarm global-mode management. Prereq: persistent journald on hosts (scripts/enable-persistent-journald.sh exists).
Option B: re-privilege. Per-node docker compose/systemd-unit deployment (the stack file's own suggestion) so pid: host is honored — full capability (systemctl, ss, remediation commands) at the cost of 9-node operational drag outside swarm.

Suggested: A now for journal + health file-based signals; B (or a future engine that honors privileges) only when/if host actuation is actually enabled — until then HostHealthProbe's systemctl/ss/timedatectl sections should report "unavailable" honestly (#374). Downstream consumers to re-verify once signal flows: SecurityMonitor detectors, SystemRemediation rules (both currently starved).

Decision ticket from the #371 investigation. Under swarm deploy NodeAgent has NO real host access (see #374), so the entire host-signal pipeline — journal events for SecurityMonitor's six detectors and SystemRemediation's crash-loop/disk rules, plus honest HostHealthProbe data — has never actually flowed in prod. The `Commands` actuation path (whitelisted remediations) is also dark (`Commands:Enabled=false` AND no host reach). **Option A (recommended for the journal path): de-privilege.** Install `systemd` (journalctl binary) in the image and read the ALREADY-MOUNTED host journal directly: `journalctl -D /var/log/journal -o json -f`. No nsenter, no pid:host — works under plain `stack deploy`, keeps swarm global-mode management. Prereq: persistent journald on hosts (`scripts/enable-persistent-journald.sh` exists). **Option B: re-privilege.** Per-node `docker compose`/systemd-unit deployment (the stack file's own suggestion) so `pid: host` is honored — full capability (systemctl, ss, remediation commands) at the cost of 9-node operational drag outside swarm. Suggested: A now for journal + health file-based signals; B (or a future engine that honors privileges) only when/if host *actuation* is actually enabled — until then `HostHealthProbe`'s systemctl/ss/timedatectl sections should report "unavailable" honestly (#374). Downstream consumers to re-verify once signal flows: SecurityMonitor detectors, SystemRemediation rules (both currently starved).
spikerj added the enhancement label 2026-07-05 03:15:39 +00:00
spikerj commented 2026-07-05 07:10:25 +00:00
Author
Owner
Copy Link
Copy Source

Resolved (code) in spikersoft-backend PR #86 (merged to master) — Option A, de-privilege. JournalAccessMode=MountedDirectory (default) reads the already-mounted host journal via journalctl -D /var/log/journal (no nsenter/pid:host), so the journal signal flows under swarm stack deploy; probe verifies the mount is reachable; HostHealthProbe now degrades loudly (host-command signals reported Unavailable with the new HostCommandsAvailable/UnavailableSignals fields, not silent "all clear" — the #374 residual). Dockerfile installs journalctl; no infra stack change needed (dirs already mounted). 114/114 NodeAgent tests; SecurityMonitor/SystemRemediation 144 tests still pass.

Deploy validation (yours): rebuild+push the NodeAgent image and redeploy, then confirm in Seq that HostJournalEvents flow (SecurityMonitor detectors + SystemRemediation rules leave their starved state) and the 5s warning flood is gone. Host prereq: persistent journald on each node via scripts/enable-persistent-journald.sh (else the mount is empty and the agent idles honestly). Host-command signals (systemctl/ss/timedatectl) stay intentionally unavailable until an Option-B privileged deployment — a separate future decision, not this ticket. Closing the build; deploy-verify at your convenience.

Resolved (code) in spikersoft-backend PR #86 (merged to `master`) — Option A, de-privilege. `JournalAccessMode=MountedDirectory` (default) reads the already-mounted host journal via `journalctl -D /var/log/journal` (no nsenter/pid:host), so the journal signal flows under swarm `stack deploy`; probe verifies the mount is reachable; `HostHealthProbe` now degrades **loudly** (host-command signals reported `Unavailable` with the new `HostCommandsAvailable`/`UnavailableSignals` fields, not silent "all clear" — the #374 residual). Dockerfile installs `journalctl`; no infra stack change needed (dirs already mounted). 114/114 NodeAgent tests; SecurityMonitor/SystemRemediation 144 tests still pass. **Deploy validation (yours):** rebuild+push the NodeAgent image and redeploy, then confirm in Seq that `HostJournalEvent`s flow (SecurityMonitor detectors + SystemRemediation rules leave their starved state) and the 5s warning flood is gone. **Host prereq:** persistent journald on each node via `scripts/enable-persistent-journald.sh` (else the mount is empty and the agent idles honestly). Host-command signals (systemctl/ss/timedatectl) stay intentionally unavailable until an Option-B privileged deployment — a separate future decision, not this ticket. Closing the build; deploy-verify at your convenience.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#375