spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra] MinIO doesn't recover after a transient /mnt/fusionio drive blip — stuck '0 drives provided', 'mc ready local' healthcheck misses it, needs manual restart #504

New Issue
Closed
opened 2026-07-12 08:09:32 +00:00 by spikerj · 5 comments
spikerj commented 2026-07-12 08:09:32 +00:00
Owner
Copy Link
Copy Source

Summary

After a transient /mnt/fusionio drive blip on SERVER (see #503 — heavy-load-induced, the device recovered on its own), MinIO did not recover. It came up while the drive was absent and stays stuck serving:

Error: listPathRaw: 0 drives provided
  cmd/metacache-server-pool.go:231 cmd.(*erasureServerPools).listPath.func2()
API: ListObjectsV2(bucket=gitea-packages)   → fails

The drive is back, but MinIO won't re-attach it without a manual restart. Meanwhile it keeps answering S3 requests with errors (gitea package registry pulls failing live).

Why swarm doesn't self-heal it

The stack's healthcheck is mc ready local, which reports the server process ready — it returns healthy even when the erasure pool has 0 drives. So swarm never marks the task unhealthy and never restarts it. Net: any /mnt/fusionio blip → MinIO wedged in a "0 drives" serving-errors state indefinitely until a human forces a restart.

Immediate remediation

docker service update --force minio_minio   # rolling restart → re-detects /mnt/fusionio/minio

Fix options (durable)

  1. Drive-aware healthcheck — probe erasure/drive health, not just process readiness, so swarm auto-restarts on the 0-drives state. e.g. curl -f http://localhost:9000/minio/health/cluster (returns 503 when the pool can't serve) instead of / in addition to mc ready local.
  2. Mount stability — ensure the /mnt/fusionio/minio bind survives a device re-enumeration (bind propagation / mount unit ordering) so a transient device drop doesn't leave the container with an empty path. If fusionio can vanish under load, MinIO needs to fail hard (exit) rather than run driveless.
  3. Consider restart_policy + a real liveness probe so a driveless MinIO exits and reschedules cleanly.

Impact

While wedged: security-scanner S3 reads (#493), metadata-extractor dual-write (#493), gitea package/registry (#498), file-movement retention (#499) all fail. Data itself is intact on the (recovered) drive — this is a recovery/liveness gap, not data loss.

Related

  • #503 — the SERVER heavy-load event / fusionio blip that triggered this.
  • #493 / #498 / #499 — MinIO-dependent work that fails while MinIO is wedged.
## Summary After a **transient `/mnt/fusionio` drive blip** on SERVER (see #503 — heavy-load-induced, the device recovered on its own), **MinIO did not recover**. It came up while the drive was absent and stays stuck serving: ``` Error: listPathRaw: 0 drives provided cmd/metacache-server-pool.go:231 cmd.(*erasureServerPools).listPath.func2() API: ListObjectsV2(bucket=gitea-packages) → fails ``` The drive is back, but MinIO won't re-attach it without a **manual restart**. Meanwhile it keeps answering S3 requests with errors (gitea package registry pulls failing live). ## Why swarm doesn't self-heal it The stack's healthcheck is `mc ready local`, which reports the **server process** ready — it returns healthy even when the erasure pool has **0 drives**. So swarm never marks the task unhealthy and never restarts it. Net: any `/mnt/fusionio` blip → MinIO wedged in a "0 drives" serving-errors state indefinitely until a human forces a restart. ## Immediate remediation ```bash docker service update --force minio_minio # rolling restart → re-detects /mnt/fusionio/minio ``` ## Fix options (durable) 1. **Drive-aware healthcheck** — probe erasure/drive health, not just process readiness, so swarm auto-restarts on the 0-drives state. e.g. `curl -f http://localhost:9000/minio/health/cluster` (returns 503 when the pool can't serve) instead of / in addition to `mc ready local`. 2. **Mount stability** — ensure the `/mnt/fusionio/minio` bind survives a device re-enumeration (bind propagation / mount unit ordering) so a transient device drop doesn't leave the container with an empty path. If fusionio can vanish under load, MinIO needs to fail hard (exit) rather than run driveless. 3. Consider `restart_policy` + a real liveness probe so a driveless MinIO exits and reschedules cleanly. ## Impact While wedged: security-scanner S3 reads (#493), metadata-extractor dual-write (#493), gitea package/registry (#498), file-movement retention (#499) all fail. Data itself is intact on the (recovered) drive — this is a recovery/liveness gap, not data loss. ## Related - #503 — the SERVER heavy-load event / fusionio blip that triggered this. - #493 / #498 / #499 — MinIO-dependent work that fails while MinIO is wedged.
spikerj commented 2026-07-12 17:59:23 +00:00
Author
Owner
Copy Link
Copy Source

Fix up in spikersoft-infrastructure PR #42: drive-aware healthcheck. One correction to this ticket's option 1: curl was REMOVED from MinIO images in 2023, so the probe is mc-based — mc ready local (process) plus an authenticated mc ls (data path — the exact listPathRaw operation that wedged), creds container-expanded ($$-escaped, never in the service spec). 3 failures → swarm restarts the task → drive re-attaches: the manual service update --force recovery, automated.

Options 2 (bind propagation across device re-enumeration) and 3 (fail-hard on driveless boot) remain open as hardening follow-ups — but with the probe in place, both failure shapes now converge to an automatic restart within ~2 minutes instead of an indefinite wedge, which was the operational pain. Suggest closing on deploy + keeping 2/3 as a note here or a slim follow-up ticket if a future blip shows the restart loop isn't sufficient.

Fix up in **spikersoft-infrastructure PR #42**: drive-aware healthcheck. One correction to this ticket's option 1: curl was REMOVED from MinIO images in 2023, so the probe is mc-based — `mc ready local` (process) **plus an authenticated `mc ls`** (data path — the exact `listPathRaw` operation that wedged), creds container-expanded ($$-escaped, never in the service spec). 3 failures → swarm restarts the task → drive re-attaches: the manual `service update --force` recovery, automated. Options 2 (bind propagation across device re-enumeration) and 3 (fail-hard on driveless boot) remain open as hardening follow-ups — but with the probe in place, both failure shapes now converge to an automatic restart within ~2 minutes instead of an indefinite wedge, which was the operational pain. Suggest closing on deploy + keeping 2/3 as a note here or a slim follow-up ticket if a future blip shows the restart loop isn't sufficient.
spikerj commented 2026-07-12 18:48:12 +00:00
Author
Owner
Copy Link
Copy Source

Fixed, deployed, and verified as far as possible without a real blip — closing. Infra #42 (merged, deployed live during the session): the healthcheck now runs an authenticated mc ls data-path probe (the exact listPathRaw operation that wedged) alongside mc ready; 3 failures → swarm restarts the task → drive re-attaches. Creds are $$-escaped (probe-time expansion — never in the service spec). Final proof arrives with the next real /mnt/fusionio blip: expect self-recovery within ~2 minutes; if it ever wedges again, reopen with the service logs (also see #505's follow-ups for the node-agent MinIO drives alert).

Fixed, deployed, and verified as far as possible without a real blip — closing. Infra #42 (merged, deployed live during the session): the healthcheck now runs an authenticated `mc ls` data-path probe (the exact listPathRaw operation that wedged) alongside `mc ready`; 3 failures → swarm restarts the task → drive re-attaches. Creds are $$-escaped (probe-time expansion — never in the service spec). Final proof arrives with the next real /mnt/fusionio blip: expect self-recovery within ~2 minutes; if it ever wedges again, reopen with the service logs (also see #505's follow-ups for the node-agent MinIO drives alert).
spikerj commented 2026-07-13 02:37:18 +00:00
Author
Owner
Copy Link
Copy Source

MinIO down again, new failure mode (2026-07-13 ~02:31Z) — and it takes the container registry with it.

  • minio_minio task exited cleanly (state Complete, exit 0) at ~02:31Z and swarm will NOT replace it: restart policy is on-failure, and an exit-0 task never triggers it → stuck 0/1 indefinitely. This is a different hole than the 0-drives wedge this ticket started with (same class as the #513 exit-0 issue).
  • Deployed spec is unchanged since 07-12 18:02 UTC, so this was not a deploy; something made minio exit 0 (mc admin service stop? SIGTERM on SERVER? — can't read its logs, SERVER log RPC still dead per #503).
  • Blast radius is the entire Gitea container registry: it's S3-backed by MinIO, so every docker manifest inspect git.spikersoft.com/... now returns 'manifest unknown' (verified on artpipe-processor/artpipe-base/model-env images that resolved fine at 02:25Z). Any service reschedule needing a pull will fail; ModelEnvImages runs 10775/10777 (mid-build) will fail at push; all Storage__UseS3 services are down a backend.
  • Remedy: docker service update --force minio_minio (fresh task, fresh restart budget). My attempt was permission-blocked — needs a human (or the #510-B auto-reconciler, but it treats Complete tasks as one-shot jobs and correctly skips them, so it won't catch this).
  • Durable fix for this ticket's scope: restart policy condition: any for minio (like jaeger got in 8a8889e/#481), so even clean exits respawn.
**MinIO down again, new failure mode (2026-07-13 ~02:31Z) — and it takes the container registry with it.** - `minio_minio` task exited **cleanly** (state Complete, exit 0) at ~02:31Z and swarm will NOT replace it: restart policy is `on-failure`, and an exit-0 task never triggers it → stuck 0/1 indefinitely. This is a different hole than the 0-drives wedge this ticket started with (same class as the #513 exit-0 issue). - Deployed spec is unchanged since 07-12 18:02 UTC, so this was not a deploy; something made minio exit 0 (mc admin service stop? SIGTERM on SERVER? — can't read its logs, SERVER log RPC still dead per #503). - **Blast radius is the entire Gitea container registry**: it's S3-backed by MinIO, so every `docker manifest inspect git.spikersoft.com/...` now returns 'manifest unknown' (verified on artpipe-processor/artpipe-base/model-env images that resolved fine at 02:25Z). Any service reschedule needing a pull will fail; ModelEnvImages runs 10775/10777 (mid-build) will fail at push; all Storage__UseS3 services are down a backend. - Remedy: `docker service update --force minio_minio` (fresh task, fresh restart budget). My attempt was permission-blocked — needs a human (or the #510-B auto-reconciler, but it treats Complete tasks as one-shot jobs and correctly skips them, so it won't catch this). - Durable fix for this ticket's scope: restart policy `condition: any` for minio (like jaeger got in 8a8889e/#481), so even clean exits respawn.
spikerj commented 2026-07-13 16:19:08 +00:00
Author
Owner
Copy Link
Copy Source

Drive-drop recurrence (2026-07-13 ~17:0xZ): minio exited clean, restart:any respawned it, and the new task loops 'Read/Write quorum could not be established... drives-online: 0' — /mnt/fusionio/minio is GONE at the host level while SERVER stays Ready. The drive-aware healthcheck is correctly holding the service unhealthy (working as designed — no phantom-healthy wedge), but recovery needs the HOST: remount/restart the fusionio device on SERVER (or it may self-recover as on 07-12 08:08). Registry + all Storage__UseS3 services are down meanwhile. Cross-posting to #503: per the 07-13 02:31→02:50 sequence, this device drop is the ~20-min early-warning signal for a node crash.

Drive-drop recurrence (2026-07-13 ~17:0xZ): minio exited clean, restart:any respawned it, and the new task loops 'Read/Write quorum could not be established... drives-online: 0' — /mnt/fusionio/minio is GONE at the host level while SERVER stays Ready. The drive-aware healthcheck is correctly holding the service unhealthy (working as designed — no phantom-healthy wedge), but recovery needs the HOST: remount/restart the fusionio device on SERVER (or it may self-recover as on 07-12 08:08). Registry + all Storage__UseS3 services are down meanwhile. Cross-posting to #503: per the 07-13 02:31→02:50 sequence, this device drop is the ~20-min early-warning signal for a node crash.
spikerj commented 2026-07-14 05:44:41 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — post-close deploy verification 2026-07-14 ~05:45Z: the hardened healthcheck from this ticket went live at 05:43:41Z — the service healthcheck is now mc ready local && mc alias set health ... && mc ls health/ (authenticated data-path probe) instead of the bare mc ready local that missed the stuck '0 drives provided' state. Redeploy converged cleanly (≈60s blip while the new task passed its start period; back to 1/1). The real test is the next drive blip — with tonight's #552 PCIe/NVMe findings on the fleet, it may come sooner than anyone likes; QA will watch whether the healthcheck now catches it and swarm restarts MinIO unattended.

**QA Team** — post-close deploy verification 2026-07-14 ~05:45Z: the hardened healthcheck from this ticket went live at 05:43:41Z — the service healthcheck is now `mc ready local && mc alias set health ... && mc ls health/` (authenticated data-path probe) instead of the bare `mc ready local` that missed the stuck '0 drives provided' state. Redeploy converged cleanly (≈60s blip while the new task passed its start period; back to 1/1). The real test is the next drive blip — with tonight's #552 PCIe/NVMe findings on the fleet, it may come sooner than anyone likes; QA will watch whether the healthcheck now catches it and swarm restarts MinIO unattended.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#504