[Infra][CI] Gitea Actions leaves GITEA-ACTION* volumes behind → build nodes recurrently fill up (need runner auto-cleanup / scheduled prune) #514

New Issue

2026-07-12T19:58:40Z

spikerj commented

2026-07-12 19:58:40 +00:00

Summary

Build/CI nodes (e.g. laptop-server) recurrently run out of disk, and the confirmed culprit is leftover GITEA-ACTION* Docker volumes that the Gitea Actions runner creates per job and never cleans up. They accumulate indefinitely until the node fills and manual docker volume rm is needed. This has recurred multiple times (also intertwined with the separate GITEA-ACTIONS-* leftover-network / IPv4 address-pool exhaustion issue).

Immediate remediation (manual, per node)

docker volume ls -q --filter name=GITEA-ACTION | wc -l                       # scale
docker volume ls -q --filter name=GITEA-ACTION | xargs -r docker volume rm   # in-use skipped safely
docker system df                                                             # verify reclaim

Durable fixes (pick one or combine)

Runner-level auto-cleanup — configure the Gitea Actions runner so job volumes are removed on completion. Depending on runner/version: enable container/volume auto-remove for the job executor, or set the runner's cleanup/GC option so per-task volumes don't persist after the job.
Scheduled prune (belt-and-suspenders) — a systemd timer / cron on each build node:
```
# daily, off-hours
docker volume ls -q --filter name=GITEA-ACTION | xargs -r docker volume rm 2>/dev/null
docker builder prune -f --filter until=48h
docker image prune -f
```
(Name-filtered volume rm avoids touching real named data volumes; rm skips any volume still attached to a running job.)
Disk monitoring/alerting — alert at e.g. 80% on build nodes so this is caught before it wedges CI, rather than discovered when a job fails.

Impact

When a build node fills: CI jobs fail, and it can compound with the network-address-pool exhaustion (leftover GITEA-ACTIONS-* networks) to block new job containers entirely. Recurring manual toil to reclaim space.

Separate but same-family: GITEA-ACTIONS-* network leftovers → could not find an available, non-overlapping IPv4 address pool (fix: docker network prune -f + daemon.json default-address-pools). Consider bundling both cleanups into the same scheduled job.
Disk-cleanup recipe kept in the on-call notes (build cache / images / volumes ordering).

Verification

After enabling runner auto-cleanup: run a few CI jobs, confirm docker volume ls --filter name=GITEA-ACTION does not grow (count returns to baseline after jobs finish).
After scheduled-prune: confirm the timer fires and disk stays under threshold across a day of CI activity.

## Summary Build/CI nodes (e.g. `laptop-server`) **recurrently run out of disk**, and the confirmed culprit is **leftover `GITEA-ACTION*` Docker volumes** that the Gitea Actions runner creates per job and never cleans up. They accumulate indefinitely until the node fills and manual `docker volume rm` is needed. This has recurred multiple times (also intertwined with the separate `GITEA-ACTIONS-*` leftover-network / IPv4 address-pool exhaustion issue). ## Immediate remediation (manual, per node) ```bash docker volume ls -q --filter name=GITEA-ACTION | wc -l # scale docker volume ls -q --filter name=GITEA-ACTION | xargs -r docker volume rm # in-use skipped safely docker system df # verify reclaim ``` ## Durable fixes (pick one or combine) 1. **Runner-level auto-cleanup** — configure the Gitea Actions runner so job volumes are removed on completion. Depending on runner/version: enable container/volume auto-remove for the job executor, or set the runner's cleanup/GC option so per-task volumes don't persist after the job. 2. **Scheduled prune (belt-and-suspenders)** — a systemd timer / cron on each build node: ``` # daily, off-hours docker volume ls -q --filter name=GITEA-ACTION | xargs -r docker volume rm 2>/dev/null docker builder prune -f --filter until=48h docker image prune -f ``` (Name-filtered volume rm avoids touching real named data volumes; `rm` skips any volume still attached to a running job.) 3. **Disk monitoring/alerting** — alert at e.g. 80% on build nodes so this is caught before it wedges CI, rather than discovered when a job fails. ## Impact When a build node fills: CI jobs fail, and it can compound with the network-address-pool exhaustion (leftover `GITEA-ACTIONS-*` networks) to block new job containers entirely. Recurring manual toil to reclaim space. ## Related - Separate but same-family: `GITEA-ACTIONS-*` **network** leftovers → `could not find an available, non-overlapping IPv4 address pool` (fix: `docker network prune -f` + `daemon.json` `default-address-pools`). Consider bundling both cleanups into the same scheduled job. - Disk-cleanup recipe kept in the on-call notes (build cache / images / volumes ordering). ## Verification - After enabling runner auto-cleanup: run a few CI jobs, confirm `docker volume ls --filter name=GITEA-ACTION` does not grow (count returns to baseline after jobs finish). - After scheduled-prune: confirm the timer fires and disk stays under threshold across a day of CI activity.

spikerj commented

2026-07-12 21:23:35 +00:00

Durable fix up: infra PR #48 — gitea-act-runner/cleanup/ with a name-filtered prune script (GITEA-ACTION* volumes + GITEA-ACTIONS-* networks + 48h build-cache + dangling images; real named volumes structurally untouchable; in-use resources skipped), systemd daily 04:30 timer, and a per-node install.sh that reclaims immediately on install.

Root-cause note: the runner configs already set container.rm: true, but the current act_runner nightly removes only job containers — the per-job workspace volumes leak regardless. The timer is the version-proof backstop; pinning act_runner past nightly when a stable tag ships volume cleanup is the complementary follow-up.

After merge, one command per runner node: cd /mnt/infrastructure/gitea-act-runner/cleanup && sudo bash install.sh — will close on your confirmation it ran (the journald lines show reclaimed counts).

Durable fix up: **infra PR #48** — `gitea-act-runner/cleanup/` with a name-filtered prune script (GITEA-ACTION* volumes + GITEA-ACTIONS-* networks + 48h build-cache + dangling images; real named volumes structurally untouchable; in-use resources skipped), systemd daily 04:30 timer, and a per-node `install.sh` that reclaims immediately on install. Root-cause note: the runner configs already set `container.rm: true`, but the current act_runner **nightly** removes only job containers — the per-job workspace volumes leak regardless. The timer is the version-proof backstop; pinning act_runner past nightly when a stable tag ships volume cleanup is the complementary follow-up. After merge, one command per runner node: `cd /mnt/infrastructure/gitea-act-runner/cleanup && sudo bash install.sh` — will close on your confirmation it ran (the journald lines show reclaimed counts).