spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services #503

New Issue
Open
opened 2026-07-12 07:45:34 +00:00 by spikerj · 24 comments
spikerj commented 2026-07-12 07:45:34 +00:00
Owner
Copy Link
Copy Source

Summary

The SERVER node's Docker daemon is crash-looping the whole node, taking down every SERVER-pinned service repeatedly. Root cause is a kernel memory-corruption fault, not application code:

BUG: Bad page state in process dockerd  pfn:fdfde8

Bad page state means the kernel found a physical page in an inconsistent state. dockerd is the process that happened to touch it — it is the victim, not the cause. This almost always indicates faulty RAM / a bad DIMM / memory-controller fault (occasionally a driver corrupting memory). This host-level fault explains the entire flapping cascade below and is unrelated to any recently-pushed work.

Impact / blast radius

SERVER is a critical single point (minio on /mnt/fusionio, mongo-config, gpu-coordinator, backend are all hostname-pinned there). Each daemon crash bounces them and they get rejected until swarm re-forms:

Observed flaps (2026-07-12, from swarm task history, times approx): ~05:3x, ~07:09, ~07:38 — recurring roughly every ~30 min and clustering. On each flap:

gpu-coordinator / backend / minio / mongo-config-01 / jaeger:
  Rejected: "cannot create a swarm scoped network when swarm is not active"
  backend also: Failed "error while removing network: unknown network seq-attachable ..."

Then they recover ~1–2 min later. jaeger_jaeger and jaeger_elasticsearch (#481) currently down as fallout. Book/GPU pipelines blip on every flap.

Confirm the hardware fault (run on SERVER)

# Recurrence + any accompanying machine-check / ECC / EDAC errors
sudo dmesg -T | grep -c 'Bad page state'
sudo dmesg -T | grep -iE 'bad page|BUG:|call trace|mce|machine check|hardware error|edac|dimm|corrupt' | tail -60
# ECC / memory-controller error counts (labels the failing DIMM if ECC)
sudo ras-mc-ctl --summary 2>/dev/null; sudo ras-mc-ctl --errors 2>/dev/null | tail -20
# Authoritative: schedule memtest86+ on next reboot (or boot the memtest ISO)
sudo apt-get install -y memtest86+ 2>/dev/null; echo "reboot -> GRUB -> Memory test"

Recommended actions

  1. Confirm & locate the bad DIMM — memtest86+ (multiple passes) and/or EDAC (ras-mc-ctl). If ECC, EDAC names the channel/slot.
  2. Reseat / replace the failing module; re-run memtest to verify clean.
  3. Interim mitigation — because minio/mongo-config/coordinator/backend are hard-pinned to SERVER, a plain drain isn't clean. Options: (a) accept brief downtime and take SERVER out for the RAM swap; (b) temporarily relocate the pins/labels (e.g. move artpipe-gpu and GPU lanes off SERVER — already partly on 4090) to reduce blast radius during the window.
  4. Until fixed, expect intermittent full-stack blips on SERVER-pinned services on every flap; don't chase the swarm not active / DNS / poison-message fallout as separate bugs — they are downstream of this.

Related

  • #481 — jaeger_elasticsearch down on SERVER (likely a symptom / worsened by these flaps).
  • #414 — production deployment & secrets model for real hardware (single-point-of-failure topology context).
  • #500 / #502 — image-description on SERVER/4090 (blips on each flap).
## Summary The **SERVER** node's Docker daemon is crash-looping the whole node, taking down every SERVER-pinned service repeatedly. Root cause is a **kernel memory-corruption fault**, not application code: ``` BUG: Bad page state in process dockerd pfn:fdfde8 ``` `Bad page state` means the kernel found a physical page in an inconsistent state. `dockerd` is the process that happened to touch it — it is the **victim, not the cause**. This almost always indicates **faulty RAM / a bad DIMM / memory-controller fault** (occasionally a driver corrupting memory). This host-level fault explains the entire flapping cascade below and is unrelated to any recently-pushed work. ## Impact / blast radius SERVER is a critical single point (minio on `/mnt/fusionio`, mongo-config, gpu-coordinator, backend are all hostname-pinned there). Each daemon crash bounces them and they get rejected until swarm re-forms: Observed flaps (2026-07-12, from swarm task history, times approx): **~05:3x, ~07:09, ~07:38** — recurring roughly every ~30 min and clustering. On each flap: ``` gpu-coordinator / backend / minio / mongo-config-01 / jaeger: Rejected: "cannot create a swarm scoped network when swarm is not active" backend also: Failed "error while removing network: unknown network seq-attachable ..." ``` Then they recover ~1–2 min later. `jaeger_jaeger` and `jaeger_elasticsearch` (#481) currently down as fallout. Book/GPU pipelines blip on every flap. ## Confirm the hardware fault (run on SERVER) ```bash # Recurrence + any accompanying machine-check / ECC / EDAC errors sudo dmesg -T | grep -c 'Bad page state' sudo dmesg -T | grep -iE 'bad page|BUG:|call trace|mce|machine check|hardware error|edac|dimm|corrupt' | tail -60 # ECC / memory-controller error counts (labels the failing DIMM if ECC) sudo ras-mc-ctl --summary 2>/dev/null; sudo ras-mc-ctl --errors 2>/dev/null | tail -20 # Authoritative: schedule memtest86+ on next reboot (or boot the memtest ISO) sudo apt-get install -y memtest86+ 2>/dev/null; echo "reboot -> GRUB -> Memory test" ``` ## Recommended actions 1. **Confirm & locate the bad DIMM** — memtest86+ (multiple passes) and/or EDAC (`ras-mc-ctl`). If ECC, EDAC names the channel/slot. 2. **Reseat / replace** the failing module; re-run memtest to verify clean. 3. **Interim mitigation** — because minio/mongo-config/coordinator/backend are hard-pinned to SERVER, a plain `drain` isn't clean. Options: (a) accept brief downtime and take SERVER out for the RAM swap; (b) temporarily relocate the pins/labels (e.g. move `artpipe-gpu` and GPU lanes off SERVER — already partly on 4090) to reduce blast radius during the window. 4. Until fixed, expect intermittent full-stack blips on SERVER-pinned services on every flap; **don't chase the `swarm not active` / DNS / poison-message fallout as separate bugs** — they are downstream of this. ## Related - #481 — jaeger_elasticsearch down on SERVER (likely a symptom / worsened by these flaps). - #414 — production deployment & secrets model for real hardware (single-point-of-failure topology context). - #500 / #502 — image-description on SERVER/4090 (blips on each flap).
spikerj commented 2026-07-12 08:06:53 +00:00
Author
Owner
Copy Link
Copy Source

Update (2026-07-12 ~08:05): storage side of the same fault — MinIO lost its drive

The SERVER node is Ready and the daemon has been stable ~26 min (no flap since ~07:39), but minio_minio just went 0/1 and is spinning on:

Error: listPathRaw: 0 drives provided
  cmd/metacache-server-pool.go:231 cmd.(*erasureServerPools).listPath.func2()

MinIO's erasure backend sees zero drives — i.e. its /mnt/fusionio/minio bind is gone/empty. Node is up but the fusionio-backed store vanished.

Hypothesis (strengthened): the root cause is the Fusion-io / ioMemory device (or its driver) on SERVER, not generic RAM — that single subsystem explains BOTH the kernel Bad page state memory corruption AND a mid-run storage drop. Legacy ioMemory drivers on modern kernels are a known source of exactly this pair.

Impact: MinIO down takes out all the recent S3-migration work — security-scanner S3 reads (#493), metadata-extractor dual-write (#493), gitea package registry (#498), file-movement retention (#499). AI models (#497) and art_pipe (#501) also live on /mnt/fusionio and would be gone if the whole device dropped. Possible data-at-risk if the erasure drive is physically failing.

Confirm on SERVER:

mount | grep -i fusionio          # is /mnt/fusionio still mounted?
ls -la /mnt/fusionio/ /mnt/fusionio/minio/ 2>&1 | head
sudo dmesg -T | grep -iE 'fio|iomemory|fusion|nvme|EXT4-fs error|I/O error|remount|read-only' | tail -40
sudo fio-status -a 2>/dev/null || sudo modinfo iomemory-vsl 2>/dev/null | head   # ioMemory device health

If the device dropped: remounting/reloading the ioMemory driver (or a reboot) may bring it back, but the underlying card health needs checking before trusting it with the MinIO/erasure data again.

## Update (2026-07-12 ~08:05): storage side of the same fault — MinIO lost its drive The SERVER **node** is Ready and the daemon has been stable ~26 min (no flap since ~07:39), but **`minio_minio` just went 0/1** and is spinning on: ``` Error: listPathRaw: 0 drives provided cmd/metacache-server-pool.go:231 cmd.(*erasureServerPools).listPath.func2() ``` MinIO's erasure backend sees **zero drives** — i.e. its `/mnt/fusionio/minio` bind is gone/empty. Node is up but the **fusionio-backed store vanished**. **Hypothesis (strengthened):** the root cause is the **Fusion-io / ioMemory device (or its driver)** on SERVER, not generic RAM — that single subsystem explains BOTH the kernel `Bad page state` memory corruption AND a mid-run storage drop. Legacy ioMemory drivers on modern kernels are a known source of exactly this pair. **Impact:** MinIO down takes out all the recent S3-migration work — security-scanner S3 reads (#493), metadata-extractor dual-write (#493), gitea package registry (#498), file-movement retention (#499). AI models (#497) and art_pipe (#501) also live on `/mnt/fusionio` and would be gone if the whole device dropped. **Possible data-at-risk** if the erasure drive is physically failing. **Confirm on SERVER:** ```bash mount | grep -i fusionio # is /mnt/fusionio still mounted? ls -la /mnt/fusionio/ /mnt/fusionio/minio/ 2>&1 | head sudo dmesg -T | grep -iE 'fio|iomemory|fusion|nvme|EXT4-fs error|I/O error|remount|read-only' | tail -40 sudo fio-status -a 2>/dev/null || sudo modinfo iomemory-vsl 2>/dev/null | head # ioMemory device health ``` If the device dropped: remounting/reloading the ioMemory driver (or a reboot) may bring it back, but the underlying card health needs checking before trusting it with the MinIO/erasure data again.
spikerj commented 2026-07-12 08:09:32 +00:00
Author
Owner
Copy Link
Copy Source

Correction / update (2026-07-12 ~08:08)

Per node owner: SERVER was under super heavy load; the fusionio drive recovered on its own and the node is healthy again — so this reads as a load/stress-induced instability, not a confirmed dead DIMM/card. Downgrading the 'likely bad RAM → RMA' framing accordingly.

That said, a load-triggered kernel Bad page state is not nothing — marginal RAM or a driver that only faults under thermal/IO stress presents exactly this way. Recommend a memtest86+ pass during the next maintenance window and keeping an eye on dmesg for repeat Bad page state/MCE lines; if it only ever appears under peak load and never again, it can be closed as a one-off stress event.

Split out the MinIO-specific fallout (it did not self-recover after the drive returned) into its own reliability ticket: #504.

## Correction / update (2026-07-12 ~08:08) Per node owner: SERVER was under **super heavy load**; the **fusionio drive recovered on its own** and the node is healthy again — so this reads as a **load/stress-induced** instability, not a confirmed dead DIMM/card. Downgrading the 'likely bad RAM → RMA' framing accordingly. That said, a load-triggered kernel `Bad page state` is not nothing — marginal RAM or a driver that only faults under thermal/IO stress presents exactly this way. Recommend a **memtest86+ pass during the next maintenance window** and keeping an eye on `dmesg` for repeat `Bad page state`/MCE lines; if it only ever appears under peak load and never again, it can be closed as a one-off stress event. Split out the MinIO-specific fallout (it did **not** self-recover after the drive returned) into its own reliability ticket: #504.
spikerj commented 2026-07-12 08:30:32 +00:00
Author
Owner
Copy Link
Copy Source

ESCALATION (2026-07-12 ~08:29): SERVER is HARD DOWN again — recurrence, live outage

After ~45 min of stability, SERVER went Down / Unreachable again at ~08:26 (docker node ls → SERVER Down Unreachable). This time it's not a quick dockerd flap (node stayed Ready in the earlier ones) — the whole node is gone (kernel panic / hang / power), which is more severe and suggests the Bad page state fault is recurring, not a one-off load spike.

Full timeline today (approx): ~05:3x, ~07:09, ~07:38 (dockerd flaps, node recovered) → stable ~07:39–08:26 → 08:26 hard down, still unreachable at 08:29.

Live blast radius — every SERVER-hostname-pinned service is stuck Pending (no suitable node) and cannot reschedule:

  • spikersoft-backend — API down (api/learn.spikersoft.com)
  • mongo-cluster_mongo-router + mongo-config — MongoDB routing down → every worker throws MongoConnectionException / server selection timeout (e.g. keycloak-events on 4090)
  • minio (was already wedged #504)
  • gpu-coordinator

Actions:

  1. Get SERVER back — it's unreachable, so needs console/IPMI: power-cycle if hung, capture the panic on the console/serial (the kernel Bad page state may have escalated to a full panic this time).
  2. Confirm the fault — memtest86+ before returning it to production; this recurrence makes marginal RAM / a faulty driver the leading hypothesis, not load.
  3. Architecture — backend, mongo-router, minio single-pinned to SERVER makes it a hard SPOF; when it dies the app dies (see #414). Worth relocating/replicating the critical stateful pins so one bad node can't take the whole stack down.

Reclassifying this as an active production incident, not a resolved load event.

## ESCALATION (2026-07-12 ~08:29): SERVER is HARD DOWN again — recurrence, live outage After ~45 min of stability, SERVER went **Down / Unreachable** again at ~08:26 (`docker node ls` → `SERVER Down Unreachable`). This time it's not a quick dockerd flap (node stayed Ready in the earlier ones) — the **whole node is gone** (kernel panic / hang / power), which is more severe and suggests the `Bad page state` fault is **recurring, not a one-off load spike**. **Full timeline today (approx):** ~05:3x, ~07:09, ~07:38 (dockerd flaps, node recovered) → stable ~07:39–08:26 → **08:26 hard down, still unreachable at 08:29**. **Live blast radius** — every SERVER-hostname-pinned service is stuck `Pending (no suitable node)` and cannot reschedule: - `spikersoft-backend` — API down (api/learn.spikersoft.com) - `mongo-cluster_mongo-router` + mongo-config — MongoDB routing down → every worker throws `MongoConnectionException` / `server selection timeout` (e.g. keycloak-events on 4090) - `minio` (was already wedged #504) - `gpu-coordinator` **Actions:** 1. **Get SERVER back** — it's unreachable, so needs console/IPMI: power-cycle if hung, capture the panic on the console/serial (the kernel `Bad page state` may have escalated to a full panic this time). 2. **Confirm the fault** — memtest86+ before returning it to production; this recurrence makes marginal RAM / a faulty driver the leading hypothesis, not load. 3. **Architecture** — backend, mongo-router, minio single-pinned to SERVER makes it a hard SPOF; when it dies the app dies (see #414). Worth relocating/replicating the critical stateful pins so one bad node can't take the whole stack down. Reclassifying this as an active production incident, not a resolved load event.
spikerj commented 2026-07-12 08:58:54 +00:00
Author
Owner
Copy Link
Copy Source

Recovery note (2026-07-12 ~08:57): SERVER back at ~08:51 (~25 min outage), but 6 services wedged post-recovery

SERVER rejoined ~08:51; minio/backend/mongo-router self-recovered (minio re-attached its drive — good). But 6 services did NOT self-heal and sit 0/1, all with the same failure from the swarm re-form window:

Rejected: cannot create a swarm scoped network when swarm is not active
Failed:   error while removing network: unknown network jaeger / seq-attachable <stale-id>

Affected: upload-coordinator, security-scanner, metadata-extractor (→ book pipeline down), plus docker-monitor, system-remediation, security-monitor. Common factor: they attach to the jaeger/seq-attachable overlays, which were recreated with new IDs when the swarm re-formed, so stale-task cleanup fails and they can't reschedule.

Remediation: docker service update --force <svc> per stuck service clears the stale ref and reschedules them.

Reliability gap (same theme as #504): node-crash recovery isn't clean — overlay-attached services wedge on stale network refs and need a manual force-restart. Worth a self-healing pass (e.g. a post-recovery reconcile, or not tying so many services to the observability overlays) so a node bounce doesn't leave the book pipeline down until someone intervenes.

## Recovery note (2026-07-12 ~08:57): SERVER back at ~08:51 (~25 min outage), but 6 services wedged post-recovery SERVER rejoined ~08:51; minio/backend/mongo-router self-recovered (minio re-attached its drive — good). But **6 services did NOT self-heal** and sit 0/1, all with the same failure from the swarm re-form window: ``` Rejected: cannot create a swarm scoped network when swarm is not active Failed: error while removing network: unknown network jaeger / seq-attachable <stale-id> ``` Affected: `upload-coordinator`, `security-scanner`, `metadata-extractor` (→ **book pipeline down**), plus `docker-monitor`, `system-remediation`, `security-monitor`. Common factor: they attach to the `jaeger`/`seq-attachable` overlays, which were recreated with new IDs when the swarm re-formed, so stale-task cleanup fails and they can't reschedule. **Remediation:** `docker service update --force <svc>` per stuck service clears the stale ref and reschedules them. **Reliability gap (same theme as #504):** node-crash recovery isn't clean — overlay-attached services wedge on stale network refs and need a manual force-restart. Worth a self-healing pass (e.g. a post-recovery reconcile, or not tying so many services to the observability overlays) so a node bounce doesn't leave the book pipeline down until someone intervenes.
spikerj commented 2026-07-12 17:35:55 +00:00
Author
Owner
Copy Link
Copy Source

Follow-up #506 traced to this incident's fallout path: the image-description worker's GPU-lease client can be None when the GpuCoordinator (SERVER-pinned) is transiently unreachable during a flap → AttributeError → misclassified permanent → caption dead-lettered without retry. Not the same as the restart-budget wedge; it's a code-level retry-classification bug that a coordinator blip triggers.

Follow-up #506 traced to this incident's fallout path: the image-description worker's GPU-lease client can be None when the GpuCoordinator (SERVER-pinned) is transiently unreachable during a flap → AttributeError → misclassified permanent → caption dead-lettered without retry. Not the same as the restart-budget wedge; it's a code-level retry-classification bug that a coordinator blip triggers.
spikerj commented 2026-07-12 18:26:17 +00:00
Author
Owner
Copy Link
Copy Source

Recurrence 2026-07-12 ~18:21 UTC (node owner reports another SERVER crash). Much milder than the 08:26 hard-down: SERVER came back Ready/Active/Reachable quickly (daemon flap, not a hard-down). Critical stateful services all HELD at 1/1 — minio, all mongo shards/config/router, backend, keycloak/rabbitmq, keycloak-events. Only casualty: gpu-coordinator (single replica, SERVER-pinned) went 0/1 via the familiar wedge — task Failed on error while removing network: unknown network jaeger… during the flap, then swarm stopped reconciling (UpdateStatus=completed, on-failure/maxAttempts=3). Recovered it with docker service update --force spikersoft-gpu-coordinator_... → 1/1 Running; embeddings + image-description GPU lane confirmed healthy after. Log-stream to SERVER (node jq6qtygwrlyk9od36supw0v4i) was briefly unavailable during this = same flap. Still recommend the memtest/EDAC at maintenance — these flaps keep recurring. (Restart-budget wedge angle also tracked via #481.)

Recurrence 2026-07-12 ~18:21 UTC (node owner reports another SERVER crash). **Much milder than the 08:26 hard-down:** SERVER came back Ready/Active/Reachable quickly (daemon flap, not a hard-down). Critical stateful services all HELD at 1/1 — minio, all mongo shards/config/router, backend, keycloak/rabbitmq, keycloak-events. Only casualty: `gpu-coordinator` (single replica, SERVER-pinned) went 0/1 via the familiar wedge — task Failed on `error while removing network: unknown network jaeger…` during the flap, then swarm stopped reconciling (UpdateStatus=completed, on-failure/maxAttempts=3). Recovered it with `docker service update --force spikersoft-gpu-coordinator_...` → 1/1 Running; embeddings + image-description GPU lane confirmed healthy after. Log-stream to SERVER (node jq6qtygwrlyk9od36supw0v4i) was briefly unavailable during this = same flap. Still recommend the memtest/EDAC at maintenance — these flaps keep recurring. (Restart-budget wedge angle also tracked via #481.)
spikerj commented 2026-07-12 18:29:51 +00:00
Author
Owner
Copy Link
Copy Source

Filed exploration #510 — resilience to these SERVER flaps. Scopes the fixes cheapest-first: (A) restart-policy window on the 11 on-failure/maxAttempts=3/no-window services that keep wedging, (B) auto-reconcile sweep to replace the manual force-updates, (C) registry/deploy digest verification, (D) degraded-state health checks (#504), (E) de-SPOF the ~40 SERVER-pinned services incl. the entire mongo cluster, (F) node root-cause = this ticket, (G) overlay-network stability. #503 (this) is the trigger/cure; #510 is the mitigation + HA design.

Filed exploration #510 — resilience to these SERVER flaps. Scopes the fixes cheapest-first: (A) restart-policy `window` on the 11 on-failure/maxAttempts=3/no-window services that keep wedging, (B) auto-reconcile sweep to replace the manual force-updates, (C) registry/deploy digest verification, (D) degraded-state health checks (#504), (E) de-SPOF the ~40 SERVER-pinned services incl. the entire mongo cluster, (F) node root-cause = this ticket, (G) overlay-network stability. #503 (this) is the trigger/cure; #510 is the mitigation + HA design.
spikerj commented 2026-07-13 00:52:48 +00:00
Author
Owner
Copy Link
Copy Source

New fallout data point (2026-07-13, ~00:50 local): SERVER's log RPC endpoint is dead while the node reports Ready and its tasks run. docker service logs for ANY SERVER-hosted task (minio_minio, backend, security-scanner, metadata-extractor, jaeger_elasticsearch...) returns 'incomplete log stream: node jq6qtygwrlyk9od36supw0v4i is not available'. All SERVER tasks show 'Running 6 hours ago', i.e. a mass restart around ~19:00 on 07-12 — consistent with another flap, after which the daemon came back in this degraded half-alive state. Impact: we are blind to logs for the whole MinIO-migration tier (it's all SERVER-pinned) — blocks diagnosis of #481 among others. MinIO itself is confirmed healthy via HTTP (live+ready 200). A dockerd restart on SERVER at the next safe window would likely restore the log endpoint.

New fallout data point (2026-07-13, ~00:50 local): SERVER's log RPC endpoint is dead while the node reports Ready and its tasks run. `docker service logs` for ANY SERVER-hosted task (minio_minio, backend, security-scanner, metadata-extractor, jaeger_elasticsearch...) returns 'incomplete log stream: node jq6qtygwrlyk9od36supw0v4i is not available'. All SERVER tasks show 'Running 6 hours ago', i.e. a mass restart around ~19:00 on 07-12 — consistent with another flap, after which the daemon came back in this degraded half-alive state. Impact: we are blind to logs for the whole MinIO-migration tier (it's all SERVER-pinned) — blocks diagnosis of #481 among others. MinIO itself is confirmed healthy via HTTP (live+ready 200). A dockerd restart on SERVER at the next safe window would likely restore the log endpoint.
spikerj commented 2026-07-13 02:52:00 +00:00
Author
Owner
Copy Link
Copy Source

Recurrence 2026-07-13 ~02:50Z (user-confirmed crash; node rejoined by ~02:53Z, recovery wave in progress — minio/backend/mongo-router/docker-monitor all restarting through the usual 'swarm not active' rejections).

Diagnostic detail worth keeping: minio exited CLEANLY (exit 0) at 02:31Z, ~20 minutes BEFORE the node crash — while it was down, the Gitea container registry (S3-backed) was blind and two ModelEnvImages CI runs (10775/10777) died at push. Same device-first ordering as the 07-12 08:05 incident (fusionio drive dropped, then the node went). If the Fusion-io card is the root cause, minio's behavior is an early-warning signal ~20 min ahead of the crash — a MinIO health alarm (#505's MinioHealthWatcher) firing should be treated as a SERVER pre-failure warning, not just a storage blip.

Post-recovery checklist I'm watching: minio ready 200 + registry manifests resolve again, restart-budget wedges (auto-reconciler #510-B should sweep them now), ModelEnvImages re-dispatch needed for Phase 2 (#518).

**Recurrence 2026-07-13 ~02:50Z** (user-confirmed crash; node rejoined by ~02:53Z, recovery wave in progress — minio/backend/mongo-router/docker-monitor all restarting through the usual 'swarm not active' rejections). Diagnostic detail worth keeping: **minio exited CLEANLY (exit 0) at 02:31Z, ~20 minutes BEFORE the node crash** — while it was down, the Gitea container registry (S3-backed) was blind and two ModelEnvImages CI runs (10775/10777) died at push. Same device-first ordering as the 07-12 08:05 incident (fusionio drive dropped, then the node went). If the Fusion-io card is the root cause, minio's behavior is an early-warning signal ~20 min ahead of the crash — a MinIO health alarm (#505's MinioHealthWatcher) firing should be treated as a SERVER pre-failure warning, not just a storage blip. Post-recovery checklist I'm watching: minio ready 200 + registry manifests resolve again, restart-budget wedges (auto-reconciler #510-B should sweep them now), ModelEnvImages re-dispatch needed for Phase 2 (#518).
spikerj commented 2026-07-13 02:53:02 +00:00
Author
Owner
Copy Link
Copy Source

New crash data point: SERVER went down again ~2026-07-13 03:0x UTC — second crash in ~9h (uptime was 8h21m at 02:43 UTC, so the previous boot was ~18:22 UTC 2026-07-12). Correlates with tonight's heaviest sustained memory/IO load in a while (baked-image builds + multi-GB registry pushes, #516) — consistent with bad RAM expressing under pressure rather than a software trigger.

Related collateral hardening landed tonight: the MinIO stack could previously stay DOWN after a healthcheck kill (exits 0 on SIGTERM + condition:on-failure = never rescheduled; it also needs ~240s to become ready, far beyond the old 30s start_period). Emergency-fixed live via docker service update (condition:any + 30s/5/300s healthcheck) and codified in spikersoft-infrastructure PR #54 — that combination also means MinIO now self-recovers after every one of THIS ticket's crashes instead of needing a human.

New crash data point: SERVER went down again ~2026-07-13 03:0x UTC — second crash in ~9h (uptime was 8h21m at 02:43 UTC, so the previous boot was ~18:22 UTC 2026-07-12). Correlates with tonight's heaviest sustained memory/IO load in a while (baked-image builds + multi-GB registry pushes, #516) — consistent with bad RAM expressing under pressure rather than a software trigger. Related collateral hardening landed tonight: the MinIO stack could previously stay DOWN after a healthcheck kill (exits 0 on SIGTERM + condition:on-failure = never rescheduled; it also needs ~240s to become ready, far beyond the old 30s start_period). Emergency-fixed live via docker service update (condition:any + 30s/5/300s healthcheck) and codified in spikersoft-infrastructure PR #54 — that combination also means MinIO now self-recovers after every one of THIS ticket's crashes instead of needing a human.
spikerj commented 2026-07-13 04:28:41 +00:00
Author
Owner
Copy Link
Copy Source

Recurrence #4 (2026-07-13 ~04:2xZ): HARD DOWN again — docker node ls → SERVER Down/Unreachable (user-confirmed). Same full-outage mode as 07-12 08:29: everything hostname-pinned to SERVER (backend API, mongo router+config+all shards, minio→container registry, gpu-coordinator) unschedulable until the node is power-cycled.

Cadence is accelerating: 07-12 ~07:40 flaps → 08:29 hard down → 07-13 ~02:50 crash → 04:2x hard down. ~90 min between tonight's two events. The 02:31Z minio clean-exit early-warning preceded the 02:50 crash; tonight's 04:2x event I have no minio warning for (it was healthy at 04:19Z checks) — worth pulling SERVER's kernel log on reboot to see if it's the same 'Bad page state' signature.

Collateral this time: ModelEnvImages run 10778 (~85 min into the Phase 2 batch, laptop runner) loses its registry mid-run — third build casualty of the SERVER instability if it pushes before the node returns (#518 chain). Positives when it's back: minio now has restart:any deployed (tonight's 1a6eec2) and the #510-B reconciler handled all wedges at 02:53 — recovery should be hands-off. The memtest/EDAC check from this ticket's original diagnosis is now overdue by any reasonable on-call standard.

**Recurrence #4 (2026-07-13 ~04:2xZ): HARD DOWN again** — `docker node ls` → SERVER Down/Unreachable (user-confirmed). Same full-outage mode as 07-12 08:29: everything hostname-pinned to SERVER (backend API, mongo router+config+all shards, minio→container registry, gpu-coordinator) unschedulable until the node is power-cycled. Cadence is accelerating: 07-12 ~07:40 flaps → 08:29 hard down → 07-13 ~02:50 crash → 04:2x hard down. ~90 min between tonight's two events. The 02:31Z minio clean-exit early-warning preceded the 02:50 crash; tonight's 04:2x event I have no minio warning for (it was healthy at 04:19Z checks) — worth pulling SERVER's kernel log on reboot to see if it's the same 'Bad page state' signature. Collateral this time: ModelEnvImages run 10778 (~85 min into the Phase 2 batch, laptop runner) loses its registry mid-run — third build casualty of the SERVER instability if it pushes before the node returns (#518 chain). Positives when it's back: minio now has restart:any deployed (tonight's 1a6eec2) and the #510-B reconciler handled all wedges at 02:53 — recovery should be hands-off. The memtest/EDAC check from this ticket's original diagnosis is now overdue by any reasonable on-call standard.
spikerj commented 2026-07-13 04:29:59 +00:00
Author
Owner
Copy Link
Copy Source

Crash #3: ~2026-07-13 03:5x UTC — roughly ONE HOUR after the reboot from crash #2 (03:51 UTC boot). Intervals are shrinking: ~8h → ~1h, again under memory/IO load (a ~20 GB registry push was in flight). This is textbook degrading-DIMM behavior; suggest bumping this to the top of the hardware queue — memtest86+ run or preemptive DIMM swap — before it starts corrupting writes instead of just crashing. Note the storage tier now self-recovers per infra PR #54 (MinIO condition:any verified across crashes #2 and #3), so blast radius per crash is bounded, but Mongo/Redis primaries on SERVER take an unclean stop every time.

Crash #3: ~2026-07-13 03:5x UTC — roughly ONE HOUR after the reboot from crash #2 (03:51 UTC boot). Intervals are shrinking: ~8h → ~1h, again under memory/IO load (a ~20 GB registry push was in flight). This is textbook degrading-DIMM behavior; suggest bumping this to the top of the hardware queue — memtest86+ run or preemptive DIMM swap — before it starts corrupting writes instead of just crashing. Note the storage tier now self-recovers per infra PR #54 (MinIO condition:any verified across crashes #2 and #3), so blast radius per crash is bounded, but Mongo/Redis primaries on SERVER take an unclean stop every time.
spikerj commented 2026-07-13 06:28:42 +00:00
Author
Owner
Copy Link
Copy Source

Recurrence #5 (2026-07-13 ~06:27Z): SERVER Down/Unreachable again. Cadence is now firmly periodic: 02:50 → ~04:25 → ~06:27, roughly every 90–100 min tonight. That regularity plus load-correlation (the hung 98 GB push #537 was idle-ish, so tonight's crashes aren't all load-triggered) keeps pointing at failing hardware. Standard blast radius (minio/registry/backend/mongo pending until power-cycle). Respectfully re-escalating the memtest/EDAC check — at this cadence the node is down ~15% of the night and every SERVER-pinned stateful service rides the same dice roll (#510-E).

**Recurrence #5 (2026-07-13 ~06:27Z): SERVER Down/Unreachable again.** Cadence is now firmly periodic: 02:50 → ~04:25 → ~06:27, roughly every 90–100 min tonight. That regularity plus load-correlation (the hung 98 GB push #537 was idle-ish, so tonight's crashes aren't all load-triggered) keeps pointing at failing hardware. Standard blast radius (minio/registry/backend/mongo pending until power-cycle). Respectfully re-escalating the memtest/EDAC check — at this cadence the node is down ~15% of the night and every SERVER-pinned stateful service rides the same dice roll (#510-E).
spikerj commented 2026-07-13 14:19:49 +00:00
Author
Owner
Copy Link
Copy Source

Recovery timeline for last night's #5 (closing the loop): SERVER went down ~06:27Z and stayed down ~8h until the morning power-cycle (~14:35Z). Once the node rejoined, recovery was fully hands-off: minio self-restarted (restart:any, 1a6eec2), registry restored, and every wedged worker — including the exit-0 class from #510's new gap — came back without a single manual force-update. The resilience work shipped during the incident demonstrably paid for itself. Still outstanding from the night: the memtest (5 crashes, 90-100min cadence while up), and note the SEPARATE laptop-server disk-full took gitea down twice in parallel (#514/#537) — resolved by manual prunes ~14:50Z.

**Recovery timeline for last night's #5 (closing the loop):** SERVER went down ~06:27Z and stayed down ~8h until the morning power-cycle (~14:35Z). Once the node rejoined, recovery was **fully hands-off**: minio self-restarted (restart:any, 1a6eec2), registry restored, and every wedged worker — including the exit-0 class from #510's new gap — came back without a single manual force-update. The resilience work shipped during the incident demonstrably paid for itself. Still outstanding from the night: the memtest (5 crashes, 90-100min cadence while up), and note the SEPARATE laptop-server disk-full took gitea down twice in parallel (#514/#537) — resolved by manual prunes ~14:50Z.
spikerj commented 2026-07-13 14:26:33 +00:00
Author
Owner
Copy Link
Copy Source

Recurrence #6 (2026-07-13 ~15:1xZ, user-reported): SERVER crashed and this time recovered fully automatically within ~3 min of rejoining — all pinned services (minio/backend/mongo-router/mailserver/jaeger) restarted hands-off, no wedges of either class, baseline intact. Swarm leader re-elected dreamstream7→dreamstream1 during the disruption (quorum held). The mitigation stack is now proven across two consecutive crashes; the remaining risk is purely the hardware itself — memtest/EDAC still the open action.

Recurrence #6 (2026-07-13 ~15:1xZ, user-reported): SERVER crashed and this time recovered **fully automatically within ~3 min of rejoining** — all pinned services (minio/backend/mongo-router/mailserver/jaeger) restarted hands-off, no wedges of either class, baseline intact. Swarm leader re-elected dreamstream7→dreamstream1 during the disruption (quorum held). The mitigation stack is now proven across two consecutive crashes; the remaining risk is purely the hardware itself — memtest/EDAC still the open action.
spikerj commented 2026-07-13 14:27:22 +00:00
Author
Owner
Copy Link
Copy Source

Crash #4: ~2026-07-13 06:2x UTC — during the sdxllightning image push (multi-GB blob commit into MinIO on /mnt/fusionio). That makes 3 of tonight's 4 crashes coincide with sustained MinIO ingest. The trigger profile is now specific: heavy write + page-cache pressure reliably kills the box within minutes; idle/light load survives hours. Escalating from 'schedule memtest' to: treat SERVER as unable to accept bulk writes until the RAM is tested/replaced. Practical holds until then: pause #518 remaining image pushes (sdxllightning/triposr), pause the #521 first mirror sync (15-25 GB of ingest), keep the #517 pilot running (already pulled to the 4090, unaffected). Every crash is also an unclean stop for the Mongo/Redis primaries colocated there.

Crash #4: ~2026-07-13 06:2x UTC — during the sdxllightning image push (multi-GB blob commit into MinIO on /mnt/fusionio). That makes 3 of tonight's 4 crashes coincide with sustained MinIO ingest. The trigger profile is now specific: heavy write + page-cache pressure reliably kills the box within minutes; idle/light load survives hours. Escalating from 'schedule memtest' to: **treat SERVER as unable to accept bulk writes until the RAM is tested/replaced.** Practical holds until then: pause #518 remaining image pushes (sdxllightning/triposr), pause the #521 first mirror sync (15-25 GB of ingest), keep the #517 pilot running (already pulled to the 4090, unaffected). Every crash is also an unclean stop for the Mongo/Redis primaries colocated there.
spikerj commented 2026-07-13 16:19:09 +00:00
Author
Owner
Copy Link
Copy Source

⚠️ Early-warning fired (2026-07-13 ~17:0xZ): the Fusion-io device just dropped again (minio drives-online: 0, see #504) while SERVER reports Ready. Last time this exact sequence preceded the node crash by ~20 min. If SERVER is going to be power-cycled anyway, now is the low-cost moment — and the strongest argument yet for scheduling the memtest/device diagnosis.

⚠️ Early-warning fired (2026-07-13 ~17:0xZ): the Fusion-io device just dropped again (minio drives-online: 0, see #504) while SERVER reports Ready. Last time this exact sequence preceded the node crash by ~20 min. If SERVER is going to be power-cycled anyway, now is the low-cost moment — and the strongest argument yet for scheduling the memtest/device diagnosis.
spikerj commented 2026-07-13 16:23:11 +00:00
Author
Owner
Copy Link
Copy Source

Recurrence #7 (2026-07-13 ~17:2xZ) — and the early-warning model is now 2-for-2: fusionio drive dropped ~17:00Z (#504), node crashed ~20 min later, same as the 02:31→02:50 sequence. This is strong evidence the Fusion-io card (or its slot/PSU rail) is the primary failing component, with the node crash as the secondary effect — worth prioritizing the device swap/reseat over general memtest when the maintenance window happens. Holding pattern until power-cycle; recovery expected hands-off as per the last two.

Recurrence #7 (2026-07-13 ~17:2xZ) — and the early-warning model is now **2-for-2**: fusionio drive dropped ~17:00Z (#504), node crashed ~20 min later, same as the 02:31→02:50 sequence. This is strong evidence the Fusion-io card (or its slot/PSU rail) is the primary failing component, with the node crash as the secondary effect — worth prioritizing the device swap/reseat over general memtest when the maintenance window happens. Holding pattern until power-cycle; recovery expected hands-off as per the last two.
spikerj commented 2026-07-13 16:25:00 +00:00
Author
Owner
Copy Link
Copy Source

Crash #5 (~2026-07-13 16:20 UTC): SERVER dropped SSH mid-command and went unreachable ~10 minutes after a MinIO 'cluster not ready' wedge that itself followed a canceled registry push. Notable sequencing this time: the box SURVIVED an 11 GB sustained MinIO ingest (~15:40-15:55 UTC, first clean bulk write since the crashes began), then wedged and crashed while near-idle — consistent with progressive memory corruption rather than a pure load trigger. Five crashes in ~24h. Recommendation stands and hardens: no further bulk writes targeted at this box, memtest86+/DIMM swap as the immediate next unit of work. All #515/#518/#521 pushes and Workstream A backfills remain frozen until then.

Crash #5 (~2026-07-13 16:20 UTC): SERVER dropped SSH mid-command and went unreachable ~10 minutes after a MinIO 'cluster not ready' wedge that itself followed a canceled registry push. Notable sequencing this time: the box SURVIVED an 11 GB sustained MinIO ingest (~15:40-15:55 UTC, first clean bulk write since the crashes began), then wedged and crashed while near-idle — consistent with progressive memory corruption rather than a pure load trigger. Five crashes in ~24h. Recommendation stands and hardens: no further bulk writes targeted at this box, memtest86+/DIMM swap as the immediate next unit of work. All #515/#518/#521 pushes and Workstream A backfills remain frozen until then.
spikerj commented 2026-07-13 16:30:16 +00:00
Author
Owner
Copy Link
Copy Source

Crash-#5 forensics change this ticket's thesis: prime suspect is now the iomemory-vsl4 driver, not RAM.

Evidence (journalctl, boot -1):

  • kernel BUG at mm/rmap.c:1102 / folio_mkclean at 12:08:23 EDT, process minio, call stack = ext4 writeback/fsync ONTO the fusionio filesystem. Prior boots: 'Bad page state' (GC Thread, runc), same rmap BUG at 00:24 — all page-state corruption in/around fusionio writeback windows.
  • Kernel tainted OE by iomemory-vsl4 — the community (snuf) port, dkms commit 965240b, on 6.8.0-134 (installed 2026-07-03). Kernel 6.8 reworked exactly the folio/rmap machinery it's dying in.
  • Timing: crashes began 2026-07-12 — the same window MinIO started the first-ever sustained bulk writes through this card (gitea-packages migration #498 + #516 image pushes). Not 'RAM got worse'; 'a write workload this card never had before'.
  • Power rail CLEARED: clean attach, zero fioerr/AER/detach all boot.
  • RAM can't be confirmed or cleared from logs: MEG Z390 consumer board, no ECC (EDAC: 'No ECC support'); the earlier MCE 'matches' were substring noise.

Decisive experiment (pending owner go-ahead): A/B write stress — 20 GB dd to the NVMe root, then 20 GB to /mnt/fusionio. NVMe survives + fusionio crashes ⇒ driver, RAM exonerated, no memtest needed. Caveat: the B half may deliberately crash the box once.

Remediation options if driver:

  1. Move MinIO's data root off fusionio → NVMe (root LV has 316G free, MinIO uses ~100G) — removes the trigger immediately, everything else unchanged. Fastest path to unfreezing #518/#521.
  2. Check snuf/iomemory-vsl4 upstream for folio-writeback fixes newer than 965240b; rebuild.
  3. Longer term: retire the SX350 — on this board a modern NVMe outperforms it, with an in-tree driver.
**Crash-#5 forensics change this ticket's thesis: prime suspect is now the iomemory-vsl4 driver, not RAM.** Evidence (journalctl, boot -1): - `kernel BUG at mm/rmap.c:1102` / `folio_mkclean` at 12:08:23 EDT, **process minio**, call stack = ext4 writeback/fsync ONTO the fusionio filesystem. Prior boots: 'Bad page state' (GC Thread, runc), same rmap BUG at 00:24 — all page-state corruption in/around fusionio writeback windows. - Kernel tainted **OE** by `iomemory-vsl4` — the community (snuf) port, dkms commit 965240b, on 6.8.0-134 (installed 2026-07-03). Kernel 6.8 reworked exactly the folio/rmap machinery it's dying in. - **Timing**: crashes began 2026-07-12 — the same window MinIO started the first-ever sustained bulk writes through this card (gitea-packages migration #498 + #516 image pushes). Not 'RAM got worse'; 'a write workload this card never had before'. - Power rail CLEARED: clean attach, zero fioerr/AER/detach all boot. - RAM can't be confirmed or cleared from logs: MEG Z390 consumer board, **no ECC** (EDAC: 'No ECC support'); the earlier MCE 'matches' were substring noise. **Decisive experiment (pending owner go-ahead):** A/B write stress — 20 GB dd to the NVMe root, then 20 GB to /mnt/fusionio. NVMe survives + fusionio crashes ⇒ driver, RAM exonerated, no memtest needed. Caveat: the B half may deliberately crash the box once. **Remediation options if driver:** 1. **Move MinIO's data root off fusionio → NVMe** (root LV has 316G free, MinIO uses ~100G) — removes the trigger immediately, everything else unchanged. Fastest path to unfreezing #518/#521. 2. Check snuf/iomemory-vsl4 upstream for folio-writeback fixes newer than 965240b; rebuild. 3. Longer term: retire the SX350 — on this board a modern NVMe outperforms it, with an in-tree driver.
spikerj commented 2026-07-13 17:01:28 +00:00
Author
Owner
Copy Link
Copy Source

Recurrence #8 (2026-07-13 ~18:0xZ): another flap-and-rejoin, full SERVER tier restarting (recovery in progress, expecting hands-off). The interval is shrinking: ~45 min since #7, vs the 90-100 min overnight cadence. Degrading-hardware curves look exactly like this — reprioritizing the fusionio reseat/swap from 'maintenance window' to 'soon' seems warranted; at the current trend the node will be flapping faster than services can finish recovering.

Recurrence #8 (2026-07-13 ~18:0xZ): another flap-and-rejoin, full SERVER tier restarting (recovery in progress, expecting hands-off). **The interval is shrinking: ~45 min since #7, vs the 90-100 min overnight cadence.** Degrading-hardware curves look exactly like this — reprioritizing the fusionio reseat/swap from 'maintenance window' to 'soon' seems warranted; at the current trend the node will be flapping faster than services can finish recovering.
spikerj commented 2026-07-13 17:04:38 +00:00
Author
Owner
Copy Link
Copy Source

ROOT CAUSE FOUND + MITIGATION APPLIED (2026-07-13 ~18:1xZ, by joey): a bad memory sector was identified on SERVER and a GRUB memory exclusion applied for the affected 16 MB region (badram/memmap-style boot exclusion). This matches the original kernel signature from 07-12 (BUG: Bad page state in process dockerd — memory corruption) and plausibly explains the whole incident chain: corrupted pages → dockerd/kernel crashes; the fusionio drive drops were likely a secondary symptom of the same corruption (driver structures in the bad region) rather than a failing card.

Validation plan (watching via the 5-min loop): the crash cadence was ~45-100 min, so a few hours of clean uptime is meaningful evidence; 24h ≈ confirmation. If crashes persist post-exclusion, next suspects revert to the fusionio card/slot or additional bad regions (full memtest pass would map them all). Leaving this ticket OPEN pending the soak; will report the soak result here.

**ROOT CAUSE FOUND + MITIGATION APPLIED (2026-07-13 ~18:1xZ, by joey):** a bad memory sector was identified on SERVER and a GRUB memory exclusion applied for the affected 16 MB region (badram/memmap-style boot exclusion). This matches the original kernel signature from 07-12 (`BUG: Bad page state in process dockerd` — memory corruption) and plausibly explains the whole incident chain: corrupted pages → dockerd/kernel crashes; the fusionio drive drops were likely a secondary symptom of the same corruption (driver structures in the bad region) rather than a failing card. Validation plan (watching via the 5-min loop): the crash cadence was ~45-100 min, so **a few hours of clean uptime is meaningful evidence; 24h ≈ confirmation**. If crashes persist post-exclusion, next suspects revert to the fusionio card/slot or additional bad regions (full memtest pass would map them all). Leaving this ticket OPEN pending the soak; will report the soak result here.
spikerj commented 2026-07-13 17:14:26 +00:00
Author
Owner
Copy Link
Copy Source

Verdict: bad RAM, quarantined; card/driver exonerated. Crash cluster resolved pending DIMM replacement.

  • Root cause: defective physical memory region ~0xFDFC00000 (identical pfn neighborhood fdfc07/fdfdc8/fdfde8 across 3 boots, 3 different processes incl. one only READING fusionio — classic fixed-address DIMM failure, not DMA scatter).
  • Mitigation ACTIVE: GRUB_BADRAM="0xFDF000000,0xFFFFFFFFFF000000" — 16 MB e820-unusable hole, verified in the memory map.
  • Reproduction test after quarantine: 2×10 GB buffered writes through the historical crash path (page cache → writeback → iomemory-vsl4 → fioa) at 1.3-1.6 GB/s — ZERO corruption events, no crash. The iomemory-vsl4/kernel-6.8 hypothesis from earlier forensics is retired; the card is cleared for the remaining /mnt/fusionio/spikersoft app data until Workstream A (#413) drains it.
  • Also completed while resolving this: MinIO data root migrated to SERVER's NVMe (infra PR #55) with checksum-verified copy — bulk-write dependency on the aging card removed regardless.

Remaining on this ticket: overnight memtest86+ pass to map the full extent of the bad module (extend GRUB_BADRAM if more regions found), then DIMM replacement at leisure. Downgrading from CRITICAL — the box survived deliberate reproduction of its killer workload.

**Verdict: bad RAM, quarantined; card/driver exonerated. Crash cluster resolved pending DIMM replacement.** - Root cause: defective physical memory region ~0xFDFC00000 (identical pfn neighborhood fdfc07/fdfdc8/fdfde8 across 3 boots, 3 different processes incl. one only READING fusionio — classic fixed-address DIMM failure, not DMA scatter). - Mitigation ACTIVE: GRUB_BADRAM="0xFDF000000,0xFFFFFFFFFF000000" — 16 MB e820-unusable hole, verified in the memory map. - **Reproduction test after quarantine: 2×10 GB buffered writes through the historical crash path (page cache → writeback → iomemory-vsl4 → fioa) at 1.3-1.6 GB/s — ZERO corruption events, no crash.** The iomemory-vsl4/kernel-6.8 hypothesis from earlier forensics is retired; the card is cleared for the remaining /mnt/fusionio/spikersoft app data until Workstream A (#413) drains it. - Also completed while resolving this: MinIO data root migrated to SERVER's NVMe (infra PR #55) with checksum-verified copy — bulk-write dependency on the aging card removed regardless. Remaining on this ticket: overnight memtest86+ pass to map the full extent of the bad module (extend GRUB_BADRAM if more regions found), then DIMM replacement at leisure. Downgrading from CRITICAL — the box survived deliberate reproduction of its killer workload.
spikerj commented 2026-07-29 15:52:50 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — root cause found and mitigated; the hardware fix is outstanding. Staying open.

Recorded in repo:

  • minio/docker-stack.yml:20-26 documents the bad RAM at a fixed pfn neighbourhood, the GRUB_BADRAM e820 hole, and the exoneration of the storage card.
  • docs/minio-data-root-cutover-runbook.md:29-32 carries explicit verification steps (grep -i badram /etc/default/grub, dmesg | grep -iE "e820|badram").
  • Commits 0964c1e / d65ce4f (PR #55, data root → NVMe), 1a8fa0d / 2c7b1c7 (PR #72, back to fusionio), 7cc81b1 (guarded deploy that refuses an empty data root).

That's a good outcome for the diagnosis half — the cause is pinned to specific hardware rather than left as "the node is flaky".

Remaining — this ticket's own closing items, neither of which has any record in any repo:

  1. Overnight memtest86+ to map the full extent of the bad module, and extend GRUB_BADRAM if more regions turn up. The current hole covers the one neighbourhood observed; nothing establishes it's the only one.
  2. DIMM replacement. Until then the mitigation is a workaround that survives only as long as the grub config does — a kernel or bootloader update that regenerates cmdline without the BADRAM entry silently re-exposes the bad pages.

Whether GRUB_BADRAM is currently live on the host is live state, not repo state. Settling evidence: cat /proc/cmdline / grep BADRAM /etc/default/grub, plus dmesg | grep -c 'Bad page state' counted since 2026-07-13.

One cross-reference worth making, since it may be the same root cause: #482 / #561 track Keycloak Postgres pg_toast_2619 corruption on this same host, and that repair has never been run. Page-level corruption in a database on a machine with confirmed bad RAM is a plausible common cause. The SHOW data_checksums; check listed on #482 would help settle whether these are one incident or two.

Audited against `origin/master` — **root cause found and mitigated; the hardware fix is outstanding.** Staying open. **Recorded in repo:** - `minio/docker-stack.yml:20-26` documents the bad RAM at a fixed pfn neighbourhood, the `GRUB_BADRAM` e820 hole, and the exoneration of the storage card. - `docs/minio-data-root-cutover-runbook.md:29-32` carries explicit verification steps (`grep -i badram /etc/default/grub`, `dmesg | grep -iE "e820|badram"`). - Commits `0964c1e` / `d65ce4f` (PR #55, data root → NVMe), `1a8fa0d` / `2c7b1c7` (PR #72, back to fusionio), `7cc81b1` (guarded deploy that refuses an empty data root). That's a good outcome for the diagnosis half — the cause is pinned to specific hardware rather than left as "the node is flaky". **Remaining — this ticket's own closing items, neither of which has any record in any repo:** 1. **Overnight memtest86+** to map the full extent of the bad module, and extend `GRUB_BADRAM` if more regions turn up. The current hole covers the one neighbourhood observed; nothing establishes it's the only one. 2. **DIMM replacement.** Until then the mitigation is a workaround that survives only as long as the grub config does — a kernel or bootloader update that regenerates cmdline without the BADRAM entry silently re-exposes the bad pages. Whether `GRUB_BADRAM` is currently live on the host is live state, not repo state. Settling evidence: `cat /proc/cmdline` / `grep BADRAM /etc/default/grub`, plus `dmesg | grep -c 'Bad page state'` counted since 2026-07-13. One cross-reference worth making, since it may be the same root cause: **#482 / #561** track Keycloak Postgres `pg_toast_2619` corruption on this same host, and that repair has never been run. Page-level corruption in a database on a machine with confirmed bad RAM is a plausible common cause. The `SHOW data_checksums;` check listed on #482 would help settle whether these are one incident or two.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#503