spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra] MinIO single drive faulted under first sustained concurrent write load — writers aborted mid-stream, task left LB #505

New Issue
Closed
opened 2026-07-12 12:33:39 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-12 12:33:39 +00:00
Owner
Copy Link
Copy Source

Incident (2026-07-12 ~07:45–08:30 UTC)

During the first real concurrent write load on MinIO (three writers at once: 16.3 GiB qwen model mirror from a LAN client, mc mirror of the ai tree from SERVER incl. ~1,000 small mp4s, plus normal registry traffic), MinIO degraded:

  • mc admin info: Drives: 0/1 OK, drive /data state faulty, 0 GiB total/avail
  • Writers aborted mid-stream: You did not provide the number of bytes specified by the Content-Length HTTP header (killed the Mistral GGUF upload)
  • Listings/du returned listPathRaw: 0 drives provided; admin API then timed out; health endpoints went 404 via Traefik (task dropped from LB)
  • Recovered by a forced service restart (fresh task/bind); Drives: 1/1 OK since, data intact

Ruled out (evidence captured live)

  • Thermal: fio-status internal temp 66.9°C, max 70.9°C (ioMemory throttle ~78°C+); reserves 100% Healthy
  • Space: /dev/fioa 2.9T, 3% used, 2.7T free
  • Card/link: 8 lanes @ 5.0GT/s, healthy

Suspects (unconfirmed — container logs were not captured before restart)

  1. MinIO server fault under concurrent multipart load on the -cpuv1 build (RELEASE.2025-09-07)
  2. A transient host-side flap of the /mnt/fusionio bind leaving the container with a dead view of /data

Damage & cleanup

  • qwen3-vl-8b-instruct seed lost 3/4 safetensors shards despite a 'successful' mc summary (re-mirror in progress)
  • Mistral GGUF never landed (re-run mc mirror /mnt/fusionio/spikersoft/ai spiker/ai-models on SERVER — idempotent)
  • ai-models/unprocessed/ — 1,011 mp4s (2.1 GiB) rode along with the ai-tree mirror; not model data, decide keep/relocate
  • Verify-after-write lesson: mc mirror's exit summary can claim success while the server faults — bucket-side mc du/file-count checks are now part of the seeding runbook

Follow-ups

  • If it recurs under load: capture docker service logs minio_minio BEFORE restarting; consider newer MinIO release (cpuv1 variant) and/or single-writer seeding discipline
  • Monitoring: extend node-agent probe (#381) — MinIO Drives: N/N OK check + alert (an unhealthy drive was invisible until a human noticed); the registry now depends on this service (#498 blast radius)

Related: #498, #497, #493, #381.

## Incident (2026-07-12 ~07:45–08:30 UTC) During the first real concurrent write load on MinIO (three writers at once: 16.3 GiB qwen model mirror from a LAN client, `mc mirror` of the ai tree from SERVER incl. ~1,000 small mp4s, plus normal registry traffic), MinIO degraded: - `mc admin info`: **Drives: 0/1 OK**, drive `/data` state **faulty**, 0 GiB total/avail - Writers aborted mid-stream: `You did not provide the number of bytes specified by the Content-Length HTTP header` (killed the Mistral GGUF upload) - Listings/du returned `listPathRaw: 0 drives provided`; admin API then timed out; health endpoints went 404 via Traefik (task dropped from LB) - Recovered by a forced service restart (fresh task/bind); **Drives: 1/1 OK** since, data intact ## Ruled out (evidence captured live) - **Thermal**: fio-status internal temp 66.9°C, max 70.9°C (ioMemory throttle ~78°C+); reserves 100% Healthy - **Space**: `/dev/fioa` 2.9T, **3% used**, 2.7T free - **Card/link**: 8 lanes @ 5.0GT/s, healthy ## Suspects (unconfirmed — container logs were not captured before restart) 1. MinIO server fault under concurrent multipart load on the `-cpuv1` build (RELEASE.2025-09-07) 2. A transient host-side flap of the /mnt/fusionio bind leaving the container with a dead view of /data ## Damage & cleanup - qwen3-vl-8b-instruct seed lost 3/4 safetensors shards despite a 'successful' mc summary (re-mirror in progress) - Mistral GGUF never landed (re-run `mc mirror /mnt/fusionio/spikersoft/ai spiker/ai-models` on SERVER — idempotent) - `ai-models/unprocessed/` — 1,011 mp4s (2.1 GiB) rode along with the ai-tree mirror; not model data, decide keep/relocate - **Verify-after-write lesson**: mc mirror's exit summary can claim success while the server faults — bucket-side `mc du`/file-count checks are now part of the seeding runbook ## Follow-ups - If it recurs under load: capture `docker service logs minio_minio` BEFORE restarting; consider newer MinIO release (cpuv1 variant) and/or single-writer seeding discipline - Monitoring: extend node-agent probe (#381) — MinIO `Drives: N/N OK` check + alert (an unhealthy drive was invisible until a human noticed); the registry now depends on this service (#498 blast radius) Related: #498, #497, #493, #381.
spikerj commented 2026-07-12 19:15:24 +00:00
Author
Owner
Copy Link
Copy Source

Follow-up delivered: backend PR #241 — MinIO cluster-health alarm in docker-monitor (drive-quorum endpoint, once-per-episode LogError naming the blast radius + recovery notice + dashboard cluster events; 4 state-machine tests). Combined with infra #42's drive-aware healthcheck (deployed): the wedge now self-heals AND pages exactly once. Suggest closing this ticket on #241's merge — it stands as the incident record; remaining wishlist items (drive temp, docker network-pool headroom) fold into #381/#480's probe backlog.

Follow-up delivered: **backend PR #241** — MinIO cluster-health alarm in docker-monitor (drive-quorum endpoint, once-per-episode LogError naming the blast radius + recovery notice + dashboard cluster events; 4 state-machine tests). Combined with infra #42's drive-aware healthcheck (deployed): the wedge now self-heals AND pages exactly once. Suggest closing this ticket on #241's merge — it stands as the incident record; remaining wishlist items (drive temp, docker network-pool headroom) fold into #381/#480's probe backlog.
spikerj commented 2026-07-12 19:23:07 +00:00
Author
Owner
Copy Link
Copy Source

Closing: backend PR #241 merged (docker-monitor MinIO cluster-health watcher — polls the drive-quorum endpoint every 60s, raises exactly one LogError alarm after 3 consecutive failures + publishes cluster events, clears once on recovery; 4 xUnit state-machine tests). Together with infra #42 (drive-aware mc ls healthcheck, deployed) the original failure mode now self-heals via task replacement AND pages once instead of silently 404ing through Traefik. Data damage from the incident was already repaired (qwen re-mirror verified 16/16 objects, Mistral GGUF re-uploaded).

Closing: **backend PR #241 merged** (docker-monitor MinIO cluster-health watcher — polls the drive-quorum endpoint every 60s, raises exactly one LogError alarm after 3 consecutive failures + publishes cluster events, clears once on recovery; 4 xUnit state-machine tests). Together with infra #42 (drive-aware `mc ls` healthcheck, deployed) the original failure mode now self-heals via task replacement AND pages once instead of silently 404ing through Traefik. Data damage from the incident was already repaired (qwen re-mirror verified 16/16 objects, Mistral GGUF re-uploaded).
spikerj commented 2026-07-14 05:45:33 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — post-close root-cause record 2026-07-14 ~05:50Z, so the story is findable later: the drive faults this ticket documented were traced to bad RAM on SERVER, now isolated via GRUB boot-time memory mapping. Interim mitigation had MinIO's data on SERVER:/ (which quietly filled the root filesystem — #558); at 05:43:41Z the bind was restored to /mnt/fusionio/minio alongside the hardened data-path healthcheck from #504. QA watch items going forward: MinIO under the next sustained write load on the fusion card (the original failure mode), and whether the new healthcheck catches any recurrence unattended. Related fleet context: tonight's #552 diagnosis found PCIe/NVMe-level corrected errors on the dreamstream nodes — different hardware, same class of storage-path flakiness; worth keeping both in view if faults recur.

**QA Team** — post-close root-cause record 2026-07-14 ~05:50Z, so the story is findable later: the drive faults this ticket documented were traced to **bad RAM on SERVER**, now isolated via GRUB boot-time memory mapping. Interim mitigation had MinIO's data on SERVER:/ (which quietly filled the root filesystem — #558); at 05:43:41Z the bind was restored to `/mnt/fusionio/minio` alongside the hardened data-path healthcheck from #504. QA watch items going forward: MinIO under the next sustained write load on the fusion card (the original failure mode), and whether the new healthcheck catches any recurrence unattended. Related fleet context: tonight's #552 diagnosis found PCIe/NVMe-level corrected errors on the dreamstream nodes — different hardware, same class of storage-path flakiness; worth keeping both in view if faults recur.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#505