spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[NodeAgent/SystemRemediation] Host disk-space probe — detect/alert on low disk before it stalls services #381

New Issue
Closed
opened 2026-07-05 11:24:34 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-05 11:24:34 +00:00
Owner
Copy Link
Copy Source

Motivation (real incident, 2026-07-05)

git.spikersoft.com ran out of disk (Postgres No space left on device), which stalled the Gitea act-runners → ~300 CI runs stuck queued, blocked issue creation, and put pushes/registry at risk. Nothing in our monitoring caught the low-disk condition before it became an outage — it was noticed only when a write failed. This is exactly the class of silent host degradation the NodeAgent/HostHealthProbe + SystemRemediation layer (#374/#375/#376) exists to surface.

Gap

HostHealthProbe (NodeAgent) reports host command availability / journal reachability but (verify) does not probe filesystem free space. So a node approaching disk-full is invisible until something breaks.

Proposal (observe-only first, mirroring the Redis probe #376 D1/D2)

  1. NodeAgent disk-space probe — report per-mount free/total/percent for the host root + key data mounts (/, /var/lib/docker, the registry/DB volumes if reachable). De-privileged read (df / statvfs), matching the #375 host-access modes; degrade-loudly if a mount isn't readable.
  2. SystemRemediation rule — a disk-space rule in RemediationRuleEngine that raises a SystemIncident (now browsable via the #297 api/ops/incidents) + AlertNotifier at configurable thresholds (e.g. warn ≥85%, critical ≥95%), with cooldowns. Observe/recommend posture only (no auto-cleanup in v1 — recommend registry GC / log prune in the incident evidence, for a later gated auto-remediation sweep).
  3. Fits the existing incident model so it flows to email + the ops console automatically.

Acceptance

  • NodeAgent reports per-mount disk usage (pure parser unit-tested; missing/unreadable mount degrades loudly, doesn't crash).
  • SystemRemediation raises an incident + alert at warn/critical thresholds (configurable, cooldown'd), observe-only.
  • Incident evidence includes the offending mount(s) + a recommended remediation (registry GC / log prune) so a future auto-sweep can act.
  • Tests for threshold classification + de-privileged read fallback.

Severity: High (this exact failure just caused a CI outage). Related: #374, #375, #376, #297.

## Motivation (real incident, 2026-07-05) git.spikersoft.com ran out of disk (Postgres `No space left on device`), which stalled the Gitea act-runners → ~300 CI runs stuck `queued`, blocked issue creation, and put pushes/registry at risk. **Nothing in our monitoring caught the low-disk condition before it became an outage** — it was noticed only when a write failed. This is exactly the class of silent host degradation the NodeAgent/`HostHealthProbe` + SystemRemediation layer (#374/#375/#376) exists to surface. ## Gap `HostHealthProbe` (NodeAgent) reports host command availability / journal reachability but (verify) does **not** probe filesystem free space. So a node approaching disk-full is invisible until something breaks. ## Proposal (observe-only first, mirroring the Redis probe #376 D1/D2) 1. **NodeAgent disk-space probe** — report per-mount free/total/percent for the host root + key data mounts (`/`, `/var/lib/docker`, the registry/DB volumes if reachable). De-privileged read (`df` / `statvfs`), matching the #375 host-access modes; degrade-loudly if a mount isn't readable. 2. **SystemRemediation rule** — a disk-space rule in `RemediationRuleEngine` that raises a `SystemIncident` (now browsable via the #297 `api/ops/incidents`) + `AlertNotifier` at configurable thresholds (e.g. warn ≥85%, critical ≥95%), with cooldowns. Observe/recommend posture only (no auto-cleanup in v1 — recommend registry GC / log prune in the incident evidence, for a later gated auto-remediation sweep). 3. Fits the existing incident model so it flows to email + the ops console automatically. ## Acceptance - [ ] NodeAgent reports per-mount disk usage (pure parser unit-tested; missing/unreadable mount degrades loudly, doesn't crash). - [ ] SystemRemediation raises an incident + alert at warn/critical thresholds (configurable, cooldown'd), observe-only. - [ ] Incident evidence includes the offending mount(s) + a recommended remediation (registry GC / log prune) so a future auto-sweep can act. - [ ] Tests for threshold classification + de-privileged read fallback. Severity: High (this exact failure just caused a CI outage). Related: #374, #375, #376, #297.
spikerj commented 2026-07-05 11:46:34 +00:00
Author
Owner
Copy Link
Copy Source

Merged in spikersoft-backend PR #100. Root cause was subtler than the ticket assumed: disk was already probed (HostHealthProbe df -PB1 + a disk-pressure rule), but HostHealthProbe's filesystem collectors are marked UNAVAILABLE under the default swarm MountedDirectory mode (they need nsenter, which stack deploy strips) — so disk was invisible on the actual swarm nodes exactly when it mattered. New de-privileged DiskSpaceProbe (NodeAgent) + disk-space incident rule (SystemRemediation) close that gap: reads via the bind-mounted host path (no nsenter), raises a cooldown'd incident + alert at warn 85/crit 95, browsable via api/ops/incidents?kind=disk-space. Observe/recommend-only. 37 tests.

Enable (yours, infra): bind-mount host root read-only into NodeAgent + set Disk:HostMountPrefix (e.g. /host) in the nodeagent stack file. Deferred: gated auto-sweep + incident auto-resolve (#381 D3). Leaving ticket open for the deploy + D3.

Merged in spikersoft-backend PR #100. **Root cause was subtler than the ticket assumed:** disk *was* already probed (`HostHealthProbe` `df -PB1` + a `disk-pressure` rule), but `HostHealthProbe`'s filesystem collectors are marked UNAVAILABLE under the default swarm `MountedDirectory` mode (they need nsenter, which `stack deploy` strips) — so disk was invisible on the actual swarm nodes exactly when it mattered. New de-privileged `DiskSpaceProbe` (NodeAgent) + `disk-space` incident rule (SystemRemediation) close that gap: reads via the bind-mounted host path (no nsenter), raises a cooldown'd incident + alert at warn 85/crit 95, browsable via `api/ops/incidents?kind=disk-space`. Observe/recommend-only. 37 tests. **Enable (yours, infra):** bind-mount host root read-only into NodeAgent + set `Disk:HostMountPrefix` (e.g. `/host`) in the nodeagent stack file. Deferred: gated auto-sweep + incident auto-resolve (#381 D3). Leaving ticket open for the deploy + D3.
spikerj commented 2026-07-15 18:23:13 +00:00
Author
Owner
Copy Link
Copy Source

Implemented and merged — closing.

Landed in spikersoft-backend PR #100 (feature/host-disk-space-probe-381, commit 3878c55), verified present on current origin/master. All four acceptance criteria are met:

  • NodeAgent reports per-mount disk usage; pure parser; degrade-loudly — NodeAgent/Services/DiskSpaceProbe.cs (observe-only BackgroundService) reads each watched mount with its own de-privileged df -PB1 call; an unreadable mount lands in DiskUsageReport.UnreadableMounts and the others still publish (never crashes), and a fully-unreadable cycle degrades loudly once rather than flooding Seq. DiskUsageParser is side-effect-free (skips header/short/garbage rows). Deliberately separate from HostHealthProbe because df against a bind-mounted host path needs no nsenter, so disk stays visible under swarm MountedDirectory mode (#375).
  • SystemRemediation raises incident + alert at warn/critical, configurable, cooldown'd, observe-only — RemediationRuleEngine.EvaluateDisk classifies each mount via DiskSpaceClassifier (Remediation:DiskSpace:WarnPercent=85, CriticalPercent=95, inclusive bounds) and emits a disk-space finding with ProposedAction = null (no auto-cleanup). Warn maps to high severity so it clears the Alerts:MinSeverity=high gate; AlertNotifier throttles via a Redis cooldown key (Alerts:CooldownSeconds=3600).
  • Incident evidence carries offending mount(s) + recommended remediation — the finding's Evidence includes mount, usedPercent, availableBytes/totalBytes, level, and recommended_remediation (registry GC → prune Gitea Actions logs → Postgres vacuum → docker system prune), built by the pure DiskSpaceSignature.ForMount, with a would_auto_sweep=no verdict recorded for a future gated D3 executor.
  • Tests — threshold classification at the 84/85/94/95/100 boundaries, de-privileged/unreadable-mount fallback, and the pure df parser. Ran green locally on origin/master (macOS/arm64):
    • SpikerSoft.EventHandlers.NodeAgent.Tests (~Disk): 16 passed / 0 failed
    • SpikerSoft.EventHandlers.SystemRemediation.Tests (DiskSpace + RemediationRuleEngine): 39 passed / 0 failed

Auto-cleanup remains deliberately deferred (observe-only v1, TODO #381-D3) exactly as the ticket specified — a future gated auto-sweep is out of scope for this issue.

Implemented and merged — closing. Landed in **spikersoft-backend PR #100** (`feature/host-disk-space-probe-381`, commit `3878c55`), verified present on current `origin/master`. All four acceptance criteria are met: - **NodeAgent reports per-mount disk usage; pure parser; degrade-loudly** — `NodeAgent/Services/DiskSpaceProbe.cs` (observe-only `BackgroundService`) reads each watched mount with its own de-privileged `df -PB1` call; an unreadable mount lands in `DiskUsageReport.UnreadableMounts` and the others still publish (never crashes), and a fully-unreadable cycle degrades loudly *once* rather than flooding Seq. `DiskUsageParser` is side-effect-free (skips header/short/garbage rows). Deliberately separate from `HostHealthProbe` because `df` against a bind-mounted host path needs no nsenter, so disk stays visible under swarm `MountedDirectory` mode (#375). - **SystemRemediation raises incident + alert at warn/critical, configurable, cooldown'd, observe-only** — `RemediationRuleEngine.EvaluateDisk` classifies each mount via `DiskSpaceClassifier` (`Remediation:DiskSpace:WarnPercent`=85, `CriticalPercent`=95, inclusive bounds) and emits a `disk-space` finding with `ProposedAction = null` (no auto-cleanup). Warn maps to `high` severity so it clears the `Alerts:MinSeverity=high` gate; `AlertNotifier` throttles via a Redis cooldown key (`Alerts:CooldownSeconds`=3600). - **Incident evidence carries offending mount(s) + recommended remediation** — the finding's `Evidence` includes `mount`, `usedPercent`, `availableBytes`/`totalBytes`, `level`, and `recommended_remediation` (registry GC → prune Gitea Actions logs → Postgres vacuum → `docker system prune`), built by the pure `DiskSpaceSignature.ForMount`, with a `would_auto_sweep=no` verdict recorded for a future gated D3 executor. - **Tests** — threshold classification at the 84/85/94/95/100 boundaries, de-privileged/unreadable-mount fallback, and the pure df parser. Ran green locally on `origin/master` (macOS/arm64): - `SpikerSoft.EventHandlers.NodeAgent.Tests` (`~Disk`): **16 passed / 0 failed** - `SpikerSoft.EventHandlers.SystemRemediation.Tests` (`DiskSpace` + `RemediationRuleEngine`): **39 passed / 0 failed** Auto-cleanup remains deliberately deferred (observe-only v1, `TODO #381-D3`) exactly as the ticket specified — a future gated auto-sweep is out of scope for this issue.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#381