[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path #558

New Issue

2026-07-14T04:59:48Z

spikerj commented

2026-07-14 04:59:48 +00:00

QA Team — found during QA log watch 2026-07-14 ~04:55Z.

Symptom: system-remediation is firing disk-space incidents for SERVER:/ at 94% used — occurrences climbed 39→46 in a single 15-minute window (checked every ~2 min), severity=high, alerted-only (no auto-remediation exists for this class). The counter implies this crossed threshold ~1.5h ago and is still filling.

Why it's urgent-ish: SERVER hosts the API, mongo router, gpu-coordinator, embeddings, and (until 04:58Z) quiz-generation — plus tonight's 182-image book run wrote uploads/artifacts through services on this node. A full root filesystem on this node takes the core of the platform down, and tonight has already demonstrated (#552, #554) how ungracefully things fail when a node's storage misbehaves.

Likely suspects to check (in order): docker system df on SERVER (build cache/dangling images — the standing cleanup recipe for build nodes applies), container json-log growth (several services log very verbosely — e.g. every MQL query at INF; the docker-monitor junk stream #541 also inflates logs fleet-wide), /var/lib/docker volumes, and whether tonight's book-processing wrote to / instead of /mnt/fusionio.

Also ties to #555: this is another alert that (a) may be lost in the cooldown pattern and (b) reports percentages without the top-consumers detail that would make it actionable from the alert alone.

**QA Team** — found during QA log watch 2026-07-14 ~04:55Z. **Symptom:** system-remediation is firing `disk-space` incidents for **SERVER:/ at 94% used** — occurrences climbed 39→46 in a single 15-minute window (checked every ~2 min), severity=high, `alerted-only` (no auto-remediation exists for this class). The counter implies this crossed threshold ~1.5h ago and is still filling. **Why it's urgent-ish:** SERVER hosts the API, mongo router, gpu-coordinator, embeddings, and (until 04:58Z) quiz-generation — plus tonight's 182-image book run wrote uploads/artifacts through services on this node. A full root filesystem on this node takes the core of the platform down, and tonight has already demonstrated (#552, #554) how ungracefully things fail when a node's storage misbehaves. **Likely suspects to check (in order):** docker system df on SERVER (build cache/dangling images — the standing cleanup recipe for build nodes applies), container json-log growth (several services log very verbosely — e.g. every MQL query at INF; the docker-monitor junk stream #541 also inflates logs fleet-wide), /var/lib/docker volumes, and whether tonight's book-processing wrote to / instead of /mnt/fusionio. **Also ties to #555:** this is another alert that (a) may be lost in the cooldown pattern and (b) reports percentages without the top-consumers detail that would make it actionable from the alert alone.

spikerj commented

2026-07-14 05:45:29 +00:00

QA Team — likely root cause identified 2026-07-14 ~05:50Z: MinIO's data dir had been temporarily relocated onto SERVER's root filesystem while the fusion card was suspect (bad RAM on SERVER since isolated via GRUB boot-time memmap — see #505). At 05:43:41Z ops swapped the MinIO bind back to /mnt/fusionio/minio, and SERVER:/ is already down 94% → 92% in the first post-swap readings. Expect this to resolve fully once the old temporary copy on / is deleted (worth confirming that cleanup actually happens — a forgotten 100+ GB stale copy on / would leave the timebomb armed at 92%). Keep open until / is back under threshold and remediation stops firing.

**QA Team** — likely root cause identified 2026-07-14 ~05:50Z: MinIO's data dir had been **temporarily relocated onto SERVER's root filesystem** while the fusion card was suspect (bad RAM on SERVER since isolated via GRUB boot-time memmap — see #505). At 05:43:41Z ops swapped the MinIO bind back to `/mnt/fusionio/minio`, and SERVER:/ is **already down 94% → 92%** in the first post-swap readings. Expect this to resolve fully once the old temporary copy on / is deleted (worth confirming that cleanup actually happens — a forgotten 100+ GB stale copy on / would leave the timebomb armed at 92%). Keep open until / is back under threshold and remediation stops firing.

spikerj referenced this issue

2026-07-14 05:45:33 +00:00

[Bug][Infra] MinIO single drive faulted under first sustained concurrent write load — writers aborted mid-stream, task left LB #505

spikerj commented

2026-07-14 06:07:57 +00:00

QA Team — symptom resolved 2026-07-14 ~06:10Z: remediation's disk-space incidents for SERVER:/ stopped firing (last occurrence #98; the detector fires every ~2 min while above threshold, and it's been silent since). Sequence held up exactly as diagnosed: MinIO bind restored to the fusion card at 05:43:41Z → 94% → 92% → below threshold once the temporary copy on / was cleaned. Fine to close from QA's side once ops confirms the cleanup was the deliberate deletion of the old MinIO copy (and not something else making room). The alert-quality asks (top-consumers detail in the alert body, #555 cooldown interaction) remain noted there.

**QA Team** — symptom resolved 2026-07-14 ~06:10Z: remediation's disk-space incidents for SERVER:/ **stopped firing** (last occurrence #98; the detector fires every ~2 min while above threshold, and it's been silent since). Sequence held up exactly as diagnosed: MinIO bind restored to the fusion card at 05:43:41Z → 94% → 92% → below threshold once the temporary copy on / was cleaned. Fine to close from QA's side once ops confirms the cleanup was the deliberate deletion of the old MinIO copy (and not something else making room). The alert-quality asks (top-consumers detail in the alert body, #555 cooldown interaction) remain noted there.

spikerj closed this issue

2026-07-14 06:13:01 +00:00

spikerj commented

2026-07-14 06:18:42 +00:00

Acute condition resolved: SERVER root is now 32% used, 286 GB free (was 94% and climbing, alerting every ~2 min).

Reclaimed by moving the MinIO data root onto the FusionIO card and deleting what that made redundant:

the duplicate gitea-packages copy left on the NVMe after the migration (the container registry's blobs — the single largest consumer)
an abandoned multipart-upload tree MinIO never reaped

The move itself was done with a checksummed, verified copy (object counts and byte totals identical on both sides before the swap), then the old copy was removed on the owner's explicit instruction.

Keeping this open for the second half of the ticket, which is the part that actually prevents recurrence: there is still no scheduled cleanup path, so this refills on its own. The registry is the growth driver — nothing prunes superseded image versions, and the baked model images are 10-40 GB each with no shared layers. A retention script now exists (infrastructure PR #73, dry-run by default, protects latest/arch tags/digest manifests/the newest N pinned tags), but it is not scheduled yet — it has only been run as a dry run.

Remaining criterion: the prune runs on a schedule, unattended. Related: #514 (Gitea Actions leaves GITEA-ACTION* volumes behind on build nodes) is the other recurring filler.

— macbook-claude-session

Acute condition resolved: **SERVER root is now 32% used, 286 GB free** (was 94% and climbing, alerting every ~2 min). Reclaimed by moving the MinIO data root onto the FusionIO card and deleting what that made redundant: - the duplicate `gitea-packages` copy left on the NVMe after the migration (the container registry's blobs — the single largest consumer) - an abandoned multipart-upload tree MinIO never reaped The move itself was done with a checksummed, verified copy (object counts and byte totals identical on both sides before the swap), then the old copy was removed on the owner's explicit instruction. **Keeping this open for the second half of the ticket, which is the part that actually prevents recurrence:** there is still no scheduled cleanup path, so this refills on its own. The registry is the growth driver — nothing prunes superseded image versions, and the baked model images are 10-40 GB each with no shared layers. A retention script now exists (infrastructure PR #73, dry-run by default, protects `latest`/arch tags/digest manifests/the newest N pinned tags), but **it is not scheduled yet** — it has only been run as a dry run. Remaining criterion: the prune runs on a schedule, unattended. Related: #514 (Gitea Actions leaves `GITEA-ACTION*` volumes behind on build nodes) is the other recurring filler. — macbook-claude-session

spikerj referenced this issue

2026-07-14 17:42:49 +00:00

[Bug][Infra][Data] Keycloak postgres pg_statistic toast corruption persists after #552 closed — ClearExpiredEvents/session purge failing every 15-min cycle since 02:29Z, repair never run #561

spikerj referenced this issue

2026-07-14 21:12:24 +00:00

[Bug][Backend][Observability] Serilog MinimumLevel is hardcoded to Debug in EventHandlerHostBuilder — EVERY appsettings log-level setting is dead config, and ~2.9M Debug events/day are shipped to Seq #594

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#558