[Infra][SERVER] Root disk hit 99% from accumulated stale deploy images — pruned 222.7GB (now 49%); add scheduled prune on service nodes + investigate OOM kills #757

New Issue

2026-07-21T02:54:57Z

spikerj commented

2026-07-21 02:54:57 +00:00

Found during 2026-07-21 swarm recon; the CRITICAL disk alerts were never delivered because ops email is dead (#756).

What happened

SERVER / climbed 93% (07-20 14:21Z) → 99% (07-21 02:22Z), ~1%/hour, with 5.1G free of 436.5G at 02:5xZ.
docker system df on SERVER: 680 images totalling 276.5GB, only 91 in use — 216GB (78%) reclaimable. overlay2 was 262.3GB. Cause: every CI deploy pulls a new :latest digest and the superseded images accumulate as dangling; SERVER hosts the most services so it accumulates fastest.

Remediation done (2026-07-21 ~03:0xZ)

Ran the standard conservative cleanup (one-shot swarm job pinned to SERVER):

docker container prune -f → 116.3MB
docker image prune -f (dangling only — cannot touch in-use or tagged images) → 222.7GB reclaimed
After: / at 49% (214.6G free).

Follow-ups needed

Scheduled prune on service nodes (SERVER at minimum, ideally all nodes): a cron/swarm-cronjob running docker image prune -f (dangling) weekly, or docker image prune -af --filter until=168h for a deeper clean. Same class of problem as #514 (build-node GITEA-ACTION volumes) but on the service nodes' image cache — #514's fix does not cover this.
OOM kills on SERVER — HIGH: OOM kill on SERVER alerts (undelivered) at 20:32, 22:58, 00:27, 02:24Z on 07-20/21. Which process is being killed and whether it correlates with the fleet redeploy memory footprint needs a look (journalctl -k | grep -i oom on SERVER).
Kernel errors on dreamstream1 at 16:42, 20:50, 22:08, 01:31Z — also unseen due to #756; needs journalctl -k triage.

Found during 2026-07-21 swarm recon; the CRITICAL disk alerts were never delivered because ops email is dead (#756). ## What happened - SERVER `/` climbed 93% (07-20 14:21Z) → 99% (07-21 02:22Z), ~1%/hour, with 5.1G free of 436.5G at 02:5xZ. - `docker system df` on SERVER: **680 images totalling 276.5GB, only 91 in use — 216GB (78%) reclaimable**. overlay2 was 262.3GB. Cause: every CI deploy pulls a new `:latest` digest and the superseded images accumulate as dangling; SERVER hosts the most services so it accumulates fastest. ## Remediation done (2026-07-21 ~03:0xZ) Ran the standard conservative cleanup (one-shot swarm job pinned to SERVER): - `docker container prune -f` → 116.3MB - `docker image prune -f` (dangling only — cannot touch in-use or tagged images) → **222.7GB reclaimed** - After: `/` at **49%** (214.6G free). ## Follow-ups needed 1. **Scheduled prune on service nodes** (SERVER at minimum, ideally all nodes): a cron/swarm-cronjob running `docker image prune -f` (dangling) weekly, or `docker image prune -af --filter until=168h` for a deeper clean. Same class of problem as #514 (build-node GITEA-ACTION volumes) but on the service nodes' image cache — #514's fix does not cover this. 2. **OOM kills on SERVER** — `HIGH: OOM kill on SERVER` alerts (undelivered) at 20:32, 22:58, 00:27, 02:24Z on 07-20/21. Which process is being killed and whether it correlates with the fleet redeploy memory footprint needs a look (`journalctl -k | grep -i oom` on SERVER). 3. **Kernel errors on dreamstream1** at 16:42, 20:50, 22:08, 01:31Z — also unseen due to #756; needs `journalctl -k` triage.

spikerj referenced this issue

2026-07-29 15:52:33 +00:00

[Infra][CI] Gitea Actions leaves GITEA-ACTION* volumes behind → build nodes recurrently fill up (need runner auto-cleanup / scheduled prune) #514

spikerj commented

2026-07-29 16:06:52 +00:00

Audited against origin/master — item 1 NOT DONE; items 2–3 unverifiable from git. Staying open.

First, confirming these are genuinely two different tickets, because the overlap with #514 is easy to assume away. The only prune script in the repo is gitea-act-runner/cleanup/gitea-actions-prune.sh (image prune at :74) — that lives under gitea-act-runner/ and is the build/runner-node cleanup #514 covers. This ticket is about service nodes, and git grep -rn "image prune\|system prune\|container prune" origin/master returns only those lines in that one runner script. No cron unit, no swarm-cronjob, no until=168h deep clean anywhere for service nodes.

So the 222.7 GB that was pruned by hand will accumulate again with nothing scheduled to stop it.

Items 2 and 3 — OOM kills on SERVER, kernel errors on dreamstream1 — are pure live-host diagnostics. Git can say nothing; they're settled by journalctl -k on those two nodes.

Worth connecting to what the other audits turned up, because item 2 may not need separate investigation:

#503 established confirmed bad RAM on SERVER, mitigated with a GRUB_BADRAM hole but the DIMM was never replaced and memtest was never run to map the full extent. OOM kills on a machine with known-bad memory pages are worth checking against that before treating them as a capacity problem.
#756 means nobody was ever paged for any of this. The disk alert that climbed 93% → 99% over twelve hours was found only by log sweep, and #756 is still open, so the same silence applies to the next occurrence.

Remaining: the scheduled service-node prune (item 1) in full; OOM and kernel-error triage, ideally checked against #503's bad-RAM finding first.

Audited against `origin/master` — **item 1 NOT DONE; items 2–3 unverifiable from git.** Staying open. **First, confirming these are genuinely two different tickets**, because the overlap with #514 is easy to assume away. The only prune script in the repo is `gitea-act-runner/cleanup/gitea-actions-prune.sh` (`image prune` at `:74`) — that lives under `gitea-act-runner/` and is the **build/runner-node** cleanup #514 covers. This ticket is about **service nodes**, and `git grep -rn "image prune\|system prune\|container prune" origin/master` returns **only** those lines in that one runner script. No cron unit, no swarm-cronjob, no `until=168h` deep clean anywhere for service nodes. So the 222.7 GB that was pruned by hand will accumulate again with nothing scheduled to stop it. **Items 2 and 3** — OOM kills on SERVER, kernel errors on dreamstream1 — are pure live-host diagnostics. Git can say nothing; they're settled by `journalctl -k` on those two nodes. **Worth connecting to what the other audits turned up**, because item 2 may not need separate investigation: - **#503** established **confirmed bad RAM on SERVER**, mitigated with a `GRUB_BADRAM` hole but the DIMM was never replaced and memtest was never run to map the full extent. OOM kills on a machine with known-bad memory pages are worth checking against that before treating them as a capacity problem. - **#756** means nobody was ever paged for any of this. The disk alert that climbed 93% → 99% over twelve hours was found only by log sweep, and #756 is still open, so the same silence applies to the next occurrence. **Remaining:** the scheduled service-node prune (item 1) in full; OOM and kernel-error triage, ideally checked against #503's bad-RAM finding first.

spikerj commented

2026-08-07 13:51:19 +00:00

Migrated to spikerj/spikersoft-infrastructure#181 as part of the umbrella-tracker breakup.

Verified 2026-08-07 — Code: spikersoft-infrastructure@86d03ff — no scheduled prune for service nodes exists. The only prune artifact is gitea-act-runner/cleanup/gitea-actions-prune.sh, which is #514's build-node GITEA-ACTION volume cleanup and explicitly does not cover the service nodes' image cache — the thing that filled SERVER's root disk. No cron, no swarm-cronjob, no docker image prune anywhere else in the repo. Live: docker service ls — SERVER is Ready and hosting its full pinned tier (minio, backend, mongo-router, gpu-coordinator) with tasks Running; curl https://api.spikersoft.com/healthz → Healthy. The 2026-07-21 manual prune (222.7 GB, / back to 49%) held, but nothing prevents the same 1%/hour climb recurring — every CI deploy still pulls a new :latest digest and leaves the superseded image dangling.
Status: not started — the one-off prune was done by hand on 2026-07-21; no scheduled prune, no OOM triage

Closing here. Work now lives in the repo that holds the fix, so fixes #<N> in a PR will
auto-close it on merge. The umbrella tracker keeps cross-repo epics only.

— Opus 5 Agent

Migrated to **spikerj/spikersoft-infrastructure#181** as part of the umbrella-tracker breakup. Verified 2026-08-07 — **Code:** `spikersoft-infrastructure@86d03ff` — no scheduled prune for service nodes exists. The only prune artifact is `gitea-act-runner/cleanup/gitea-actions-prune.sh`, which is #514's **build-node GITEA-ACTION volume** cleanup and explicitly does not cover the service nodes' image cache — the thing that filled SERVER's root disk. No cron, no swarm-cronjob, no `docker image prune` anywhere else in the repo. **Live:** `docker service ls` — SERVER is Ready and hosting its full pinned tier (minio, backend, mongo-router, gpu-coordinator) with tasks Running; `curl https://api.spikersoft.com/healthz` → Healthy. The 2026-07-21 manual prune (222.7 GB, `/` back to 49%) held, but nothing prevents the same 1%/hour climb recurring — every CI deploy still pulls a new `:latest` digest and leaves the superseded image dangling. Status: not started — the one-off prune was done by hand on 2026-07-21; no scheduled prune, no OOM triage Closing here. Work now lives in the repo that holds the fix, so `fixes #<N>` in a PR will auto-close it on merge. The umbrella tracker keeps cross-repo epics only. — Opus 5 Agent

spikerj closed this issue

2026-08-07 13:51:19 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#757