[NodeAgent] Journal-tail Seq flood: no backoff, swallowed stderr — and the host-access probe is tautological #374

New Issue

2026-07-05T03:15:19Z

spikerj commented

2026-07-05 03:15:19 +00:00

From the #371 follow-up investigation. Prod Seq gets "journalctl stream ended; resuming in 5s." at Warning level every 5 s from every node (~17k/day/node × 9).

Root cause chain: swarm stack deploy ignores privileged:/pid: host (documented in the stack file itself) → nsenter -t 1 enters the container's own namespaces → the availability probe (RunShellAsync("true")) passes tautologically (proves sh runs, not host reach) → the tail loop runs journalctl inside the container where the binary doesn't exist (aspnet slim image) → exit 127, stdout EOF → fixed 5 s retry, Warning each cycle; the smoking-gun stderr is redirected but never read (HostCommandRunner.StreamShellLinesAsync ignores stderr and exit code).

Backoff: 5 s → exponential, cap ~5 min; Warning once then Debug with attempt counter
Capture stderr + exit code in the streaming path and include in the log line ("exit 127: journalctl: not found")
Fix the probe: verify real host namespace (e.g. cat /proc/1/comm == systemd/init, or journalctl --version) so the agent idles honestly with a single warning when privileges are stripped
Same-class audit: HostHealthProbe silently samples the container (empty systemctl --failed, no ss, no timedatectl) — degrade loudly, not silently

Restoring actual host access is #375 (separate decision ticket). Related: #374's fixes make the degradation visible; they don't restore the signal.

From the #371 follow-up investigation. Prod Seq gets "journalctl stream ended; resuming in 5s." at Warning level every 5 s from every node (~17k/day/node × 9). **Root cause chain:** swarm `stack deploy` ignores `privileged:`/`pid: host` (documented in the stack file itself) → `nsenter -t 1` enters the container's own namespaces → the availability probe (`RunShellAsync("true")`) passes tautologically (proves `sh` runs, not host reach) → the tail loop runs `journalctl` inside the container where the binary doesn't exist (aspnet slim image) → exit 127, stdout EOF → fixed 5 s retry, Warning each cycle; the smoking-gun stderr is redirected but never read (`HostCommandRunner.StreamShellLinesAsync` ignores stderr and exit code). - [ ] Backoff: 5 s → exponential, cap ~5 min; Warning once then Debug with attempt counter - [ ] Capture stderr + exit code in the streaming path and include in the log line ("exit 127: journalctl: not found") - [ ] Fix the probe: verify real host namespace (e.g. `cat /proc/1/comm` == systemd/init, or `journalctl --version`) so the agent idles honestly with a single warning when privileges are stripped - [ ] Same-class audit: `HostHealthProbe` silently samples the container (empty `systemctl --failed`, no `ss`, no `timedatectl`) — degrade loudly, not silently Restoring *actual* host access is #375 (separate decision ticket). Related: #374's fixes make the degradation visible; they don't restore the signal.

spikerj added the bug label 2026-07-05 03:15:19 +00:00

spikerj referenced this issue

2026-07-05 03:15:39 +00:00

[NodeAgent] Restore real host access — journalctl in-image reading the mounted journal (de-privileged) vs per-node compose deployment #375

spikerj commented

2026-07-05 06:29:36 +00:00

Resolved in spikersoft-backend PR #82 (merged to master). Exponential backoff (5s→cap 300s, resets on a delivered line), log-level demotion (Warning once → Debug thereafter → Information on recovery), stderr+exit-code surfaced into the failure log ("exit 127: journalctl: not found"), and a real host-access probe (/proc/1/comm must be systemd/init) so a privilege-stripped swarm deploy idles honestly instead of hot-looping. 28 new unit tests; NodeAgent suite 101/101.

Takes effect on the next NodeAgent image deploy (the flood continues until then). The HostHealthProbe "empty sample = all clear" residual and true host-access restoration are tracked under #375. Closing.

Resolved in spikersoft-backend PR #82 (merged to `master`). Exponential backoff (5s→cap 300s, resets on a delivered line), log-level demotion (Warning once → Debug thereafter → Information on recovery), stderr+exit-code surfaced into the failure log ("exit 127: journalctl: not found"), and a real host-access probe (`/proc/1/comm` must be systemd/init) so a privilege-stripped swarm deploy idles honestly instead of hot-looping. 28 new unit tests; NodeAgent suite 101/101. Takes effect on the next NodeAgent image deploy (the flood continues until then). The `HostHealthProbe` "empty sample = all clear" residual and true host-access restoration are tracked under #375. Closing.

spikerj closed this issue

2026-07-05 06:29:37 +00:00

spikerj referenced this issue

2026-07-05 07:10:25 +00:00

[NodeAgent] Restore real host access — journalctl in-image reading the mounted journal (de-privileged) vs per-node compose deployment #375

spikerj referenced this issue

2026-07-05 11:24:34 +00:00

[NodeAgent/SystemRemediation] Host disk-space probe — detect/alert on low disk before it stalls services #381

spikerj referenced this issue

2026-07-17 18:42:37 +00:00

PROD: sponsor/profiles + geography/travel-stats 500 — UserProfile.Social required-owned-nav throws on pre-#631 documents (schema drift) #672

spikerj referenced this issue