spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes #552

New Issue
Closed
opened 2026-07-14 02:23:50 +00:00 by spikerj · 9 comments
spikerj commented 2026-07-14 02:23:50 +00:00
Owner
Copy Link
Copy Source

QA Team — found during QA log watch 2026-07-14 ~02:20Z.

Node outage: docker node ls shows dreamstream4: Status=Down, ManagerStatus=Unreachable (it is a manager). All other nodes Ready — including dreamstream6, which has recovered from its own outage (#539). This looks like the same node-drop pattern as #539, now on a different (and manager) node. Quorum currently fine (leader laptop-server, SERVER + 4090 reachable), but a second manager loss would threaten it.

Fallout observed — keycloak_postgres-keycloak:

  • Its postgres task had been Running on dreamstream4 for 3 days. When the node dropped, swarm rescheduled.
  • First target dreamstream6 rejected it 3x ~01:20Z: No such image: postgres:16@sha256:be01cf82... — the same "task rejected because the node can't materialize the image" signature as the (closed) #511, though here the pull from Docker Hub by digest failed rather than a registry push race. This also corroborates the container-DNS/pull problem on ds6 noted in #539's thread.
  • It then landed on dreamstream7, Running ~01:20Z.
  • The dead dreamstream4 task still reports DesiredState=Shutdown / CurrentState=Running (3 days ago) — unconfirmable because the node is unreachable — so the service shows 2/1 replicas.

Data-integrity question (please verify): this is Keycloak's database and it just moved nodes. If its volume is node-local, the dreamstream7 instance may have started from an empty/stale data dir (Keycloak container itself shows 1/1, but that doesn't prove realm/user data survived). If dreamstream4 comes back with its old task alive, two postgreses with diverged data could exist. Worth checking Keycloak logins/realm data now and defining which node's volume is authoritative before ds4 rejoins.

Refs: #539 (dreamstream6 outage — same pattern, that node is now Ready again), #511 (closed — same 'No such image' reject-loop failure signature during (re)scheduling).

**QA Team** — found during QA log watch 2026-07-14 ~02:20Z. **Node outage:** `docker node ls` shows **dreamstream4: Status=Down, ManagerStatus=Unreachable** (it is a manager). All other nodes Ready — including dreamstream6, which has recovered from its own outage (#539). This looks like the same node-drop pattern as #539, now on a different (and manager) node. Quorum currently fine (leader laptop-server, SERVER + 4090 reachable), but a second manager loss would threaten it. **Fallout observed — keycloak_postgres-keycloak:** - Its postgres task had been Running on **dreamstream4 for 3 days**. When the node dropped, swarm rescheduled. - First target **dreamstream6** rejected it **3x ~01:20Z**: `No such image: postgres:16@sha256:be01cf82...` — the same "task rejected because the node can't materialize the image" signature as the (closed) #511, though here the pull from Docker Hub by digest failed rather than a registry push race. This also corroborates the container-DNS/pull problem on ds6 noted in #539's thread. - It then landed on **dreamstream7**, Running ~01:20Z. - The dead dreamstream4 task still reports `DesiredState=Shutdown / CurrentState=Running (3 days ago)` — unconfirmable because the node is unreachable — so the service shows **2/1 replicas**. **Data-integrity question (please verify):** this is Keycloak's *database* and it just moved nodes. If its volume is node-local, the dreamstream7 instance may have started from an empty/stale data dir (Keycloak container itself shows 1/1, but that doesn't prove realm/user data survived). If dreamstream4 comes back with its old task alive, two postgreses with diverged data could exist. Worth checking Keycloak logins/realm data now and defining which node's volume is authoritative before ds4 rejoins. **Refs:** #539 (dreamstream6 outage — same pattern, that node is now Ready again), #511 (closed — same 'No such image' reject-loop failure signature during (re)scheduling).
spikerj commented 2026-07-14 02:35:16 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — URGENT escalation 2026-07-14 ~02:45Z. The data-integrity question from this ticket is now CONFIRMED as an active failure.

1. Node status: dreamstream4 rejoined ~02:40Z (all nodes Ready). Root cause per the ops alert email at 02:32Z: kernel error on dreamstream4 (system-remediation caught it). Rejoin fallout was transient ('cannot create a swarm scoped network' rejects, then clean starts); the stale zombie tasks (keycloak-postgres, jetson-influx-4) were reaped without resuming — no split-brain occurred.

2. THE URGENT PART — keycloak postgres on dreamstream7 is writing against a broken mount and showing corruption:
Its data dir is bind /mnt/keycloak/database → /var/lib/postgresql/data — a shared/network mount, and on ds7 that mount is dead. Postgres has been erroring once per second since ~02:29Z:

ERROR: could not seek to end of file "global/1262": Transport endpoint is not connected
ERROR: could not open file "base/16384/16439": Transport endpoint is not connected
ERROR: missing chunk number 0 for toast value 131293 in pg_toast_2619   <-- corruption (pg_statistic toast)

Keycloak upstairs sees it too (GenericJDBCException on session-purge and ClearExpiredEvents scheduled tasks; agroal pool closing connections in VALIDATION state). ids.spikersoft.com OIDC still answers 200, so from the outside everything looks fine while the DB rots underneath — logins likely work off cache/partially until they don't.

Suggested immediate action (not taken by QA): stop letting it write — fix/remount /mnt/keycloak on dreamstream7 (or constrain the service back to a node with a healthy mount), then run integrity checks (VACUUM/REINDEX pg_statistic for the toast error, full dump-restore check). Every minute it runs like this increases corruption risk to realm/user data.

3. Related: the main influxDB_influxdb is pinned to dreamstream4 and was hard-down for the entire outage (~01:15–02:40Z) while docker service ls showed a stale 1/1 — see #554 for the fleet-wide fallout that caused and the monitoring gap it exposes.

**QA Team** — URGENT escalation 2026-07-14 ~02:45Z. The data-integrity question from this ticket is now CONFIRMED as an active failure. **1. Node status:** dreamstream4 rejoined ~02:40Z (all nodes Ready). Root cause per the ops alert email at 02:32Z: **kernel error on dreamstream4** (system-remediation caught it). Rejoin fallout was transient ('cannot create a swarm scoped network' rejects, then clean starts); the stale zombie tasks (keycloak-postgres, jetson-influx-4) were reaped without resuming — no split-brain occurred. **2. THE URGENT PART — keycloak postgres on dreamstream7 is writing against a broken mount and showing corruption:** Its data dir is `bind /mnt/keycloak/database → /var/lib/postgresql/data` — a shared/network mount, and on ds7 that mount is **dead**. Postgres has been erroring **once per second** since ~02:29Z: ``` ERROR: could not seek to end of file "global/1262": Transport endpoint is not connected ERROR: could not open file "base/16384/16439": Transport endpoint is not connected ERROR: missing chunk number 0 for toast value 131293 in pg_toast_2619 <-- corruption (pg_statistic toast) ``` Keycloak upstairs sees it too (GenericJDBCException on session-purge and ClearExpiredEvents scheduled tasks; agroal pool closing connections in VALIDATION state). **`ids.spikersoft.com` OIDC still answers 200**, so from the outside everything looks fine while the DB rots underneath — logins likely work off cache/partially until they don't. **Suggested immediate action (not taken by QA):** stop letting it write — fix/remount `/mnt/keycloak` on dreamstream7 (or constrain the service back to a node with a healthy mount), then run integrity checks (`VACUUM/REINDEX pg_statistic` for the toast error, full dump-restore check). Every minute it runs like this increases corruption risk to realm/user data. **3. Related:** the main `influxDB_influxdb` is pinned to dreamstream4 and was hard-down for the entire outage (~01:15–02:40Z) while `docker service ls` showed a stale 1/1 — see #554 for the fleet-wide fallout that caused and the monitoring gap it exposes.
spikerj commented 2026-07-14 02:38:36 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — de-escalation update 2026-07-14 ~02:50Z (follow-up to the URGENT note above):

  • The postgres transport errors stopped at 02:29:51Z and have not recurred; timing tracks dreamstream4's recovery. Working hypothesis: /mnt/keycloak is served/backed by dreamstream4, so the ds4 kernel panic didn't just kill the postgres task that lived there — it also broke the data mount out from under the failed-over replacement on ds7. If true, the keycloak DB has the same single-node-storage SPOF pattern as InfluxDB in #554, just one level down (storage instead of service). Worth confirming where /mnt/keycloak actually comes from.
  • Keycloak app layer is quiet since then too — no new JDBC errors after 02:29:51Z.
  • Still open: the corruption evidence. missing chunk number 0 for toast value 131293 in pg_toast_2619 hit during the window (pg_statistic's toast — may be repairable via VACUUM/REINDEX of pg_statistic, but it proves reads were served garbage). The failing query came from Keycloak's ClearExpiredEvents scheduled task, which re-runs periodically — QA is watching for recurrence; if it re-fires, corruption is durable and needs the dump/restore-level check. Either way a one-time integrity pass on this DB is still recommended before trusting it long-term.
**QA Team** — de-escalation update 2026-07-14 ~02:50Z (follow-up to the URGENT note above): - The postgres transport errors **stopped at 02:29:51Z** and have not recurred; timing tracks dreamstream4's recovery. Working hypothesis: **`/mnt/keycloak` is served/backed by dreamstream4**, so the ds4 kernel panic didn't just kill the postgres task that lived there — it also broke the data mount out from under the failed-over replacement on ds7. If true, the keycloak DB has the same single-node-storage SPOF pattern as InfluxDB in #554, just one level down (storage instead of service). Worth confirming where /mnt/keycloak actually comes from. - Keycloak app layer is quiet since then too — no new JDBC errors after 02:29:51Z. - **Still open: the corruption evidence.** `missing chunk number 0 for toast value 131293 in pg_toast_2619` hit during the window (pg_statistic's toast — may be repairable via VACUUM/REINDEX of pg_statistic, but it proves reads were served garbage). The failing query came from Keycloak's ClearExpiredEvents scheduled task, which re-runs periodically — QA is watching for recurrence; if it re-fires, corruption is durable and needs the dump/restore-level check. Either way a one-time integrity pass on this DB is still recommended before trusting it long-term.
spikerj commented 2026-07-14 02:42:23 +00:00
Author
Owner
Copy Link
Copy Source

Cross-linking evidence from tonight: dreamstream4 being Down is very likely the cause of the GlusterFS /mnt/infrastructure distress Joey observed ~02:14-02:25Z (and possibly of the brief full-edge outage ~02:30Z).

  • ds4 is (per this ticket) a swarm manager that dropped; if it also carries a Gluster brick for the /mnt/infrastructure replicated volume (DS1-7), every FUSE lookup on the surviving mounts pays pending-heal checks while a replica is absent.
  • Observed signature during backend run 6613's deploy step on DS1: a single cd /mnt/infrastructure/<stack> + docker stack deploy (a few KB of reads) took 6 minutes; the six-stack loop blew drone-ssh's 10m command_timeout after only 2 stacks (backend PR #268, merged, raises it to 30m + adds ff-only pull). Tiny metadata ops blocking for minutes = degraded-replica/heal-pressure behavior, not throughput load — nothing new writes bulk data to that volume (tonight's ~20 GB of image pushes go Gitea→MinIO on SERVER NVMe, #498).
  • Once ds4 is back: sudo gluster volume status (all bricks Online) then sudo gluster volume heal <vol> info — expect a backlog draining; deploy latency on /mnt/infrastructure should normalize as it empties. If ds4 stays down long, heal info output is the thing to watch before trusting any CI deploy step that reads the shared checkout.
  • Keycloak postgres failover fallout (ds6 "No such image" rejects) matches the #539 pull-path problem — worth keeping that thread separate from the Gluster angle.

— macbook-claude-session

Cross-linking evidence from tonight: **dreamstream4 being Down is very likely the cause of the GlusterFS `/mnt/infrastructure` distress Joey observed ~02:14-02:25Z** (and possibly of the brief full-edge outage ~02:30Z). - ds4 is (per this ticket) a swarm manager that dropped; if it also carries a Gluster brick for the `/mnt/infrastructure` replicated volume (DS1-7), every FUSE lookup on the surviving mounts pays pending-heal checks while a replica is absent. - Observed signature during backend run [6613](https://git.spikersoft.com/spikerj/spikersoft-backend/actions/runs/6613)'s deploy step on DS1: a single `cd /mnt/infrastructure/<stack>` + `docker stack deploy` (a few KB of reads) took **6 minutes**; the six-stack loop blew drone-ssh's 10m `command_timeout` after only 2 stacks (backend PR #268, merged, raises it to 30m + adds ff-only pull). Tiny metadata ops blocking for minutes = degraded-replica/heal-pressure behavior, not throughput load — nothing new writes bulk data to that volume (tonight's ~20 GB of image pushes go Gitea→MinIO on SERVER NVMe, #498). - Once ds4 is back: `sudo gluster volume status` (all bricks Online) then `sudo gluster volume heal <vol> info` — expect a backlog draining; deploy latency on /mnt/infrastructure should normalize as it empties. If ds4 stays down long, `heal info` output is the thing to watch before trusting any CI deploy step that reads the shared checkout. - Keycloak postgres failover fallout (ds6 "No such image" rejects) matches the #539 pull-path problem — worth keeping that thread separate from the Gluster angle. — macbook-claude-session
spikerj commented 2026-07-14 02:44:48 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — predictive heads-up 2026-07-14 ~02:55Z: the kernel-error class that preceded dreamstream4's panic is actively accumulating on other nodes. system-remediation in the last 4h:

  • dreamstream1: occurrences=53 by 02:29Z and climbing (alert emails now on cooldown/suppressed — the counter keeps rising quietly)
  • SERVER: 8 kernel-error mentions
  • dreamstream4 itself was at occurrences=33 when it went down

If these are the same fault signature (dsX are Jetson-class aarch64 boards — shared kernel/hardware profile), dreamstream1 looks like the next candidate for a ds4-style crash. Recommend someone pulls dmesg/journalctl -k on dreamstream1 while it's still up — and notes that alert cooldown means email volume understates the error rate; the 02:32Z email for ds4 arrived at occurrence 33, i.e., ~the moment it died. ds1 is already past that count.

**QA Team** — predictive heads-up 2026-07-14 ~02:55Z: the kernel-error class that preceded dreamstream4's panic is **actively accumulating on other nodes**. system-remediation in the last 4h: - **dreamstream1: occurrences=53** by 02:29Z and climbing (alert emails now on cooldown/suppressed — the counter keeps rising quietly) - **SERVER: 8 kernel-error mentions** - dreamstream4 itself was at occurrences=33 when it went down If these are the same fault signature (dsX are Jetson-class aarch64 boards — shared kernel/hardware profile), dreamstream1 looks like the next candidate for a ds4-style crash. Recommend someone pulls `dmesg`/`journalctl -k` on dreamstream1 while it's still up — and notes that alert cooldown means email volume *understates* the error rate; the 02:32Z email for ds4 arrived at occurrence 33, i.e., ~the moment it died. ds1 is already past that count.
spikerj commented 2026-07-14 02:49:03 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — corruption verdict 2026-07-14 ~03:00Z: durable, not transient. The missing chunk number 0 for toast value 131293 in pg_toast_2619 error re-fired at 02:44:50Z — well after the mount recovered (~02:30Z) — on the next ClearExpiredEvents run. So the missing toast chunk is really gone from disk.

Silver lining: pg_toast_2619 is the toast table for pg_statistic — the planner's derived statistics catalog, which is safely rebuildable. Repair recipe (as postgres superuser on the keycloak DB):

DELETE FROM pg_statistic;  -- or target just the corrupt rows if preferred
ANALYZE;                   -- rebuilds all stats fresh

(If DELETE itself trips on the corrupt toast row: REINDEX TABLE pg_toast.pg_toast_2619; first, or delete per-table rows until the offender is gone.)

Caveat: pg_statistic being the visible casualty doesn't guarantee it's the only one — it's just the table the scheduled task happens to read. A one-time sweep (pg_dump > /dev/null catches most toast/page-level read errors across user tables) is cheap insurance before declaring realm/user data clean. Until repaired, expect the ClearExpiredEvents/session-purge scheduled tasks to keep failing every cycle — expired events/sessions are accumulating.

**QA Team** — corruption verdict 2026-07-14 ~03:00Z: **durable, not transient.** The `missing chunk number 0 for toast value 131293 in pg_toast_2619` error re-fired at **02:44:50Z** — well after the mount recovered (~02:30Z) — on the next ClearExpiredEvents run. So the missing toast chunk is really gone from disk. **Silver lining:** `pg_toast_2619` is the toast table for **`pg_statistic`** — the planner's derived statistics catalog, which is safely rebuildable. Repair recipe (as postgres superuser on the keycloak DB): ```sql DELETE FROM pg_statistic; -- or target just the corrupt rows if preferred ANALYZE; -- rebuilds all stats fresh ``` (If DELETE itself trips on the corrupt toast row: `REINDEX TABLE pg_toast.pg_toast_2619;` first, or delete per-table rows until the offender is gone.) **Caveat:** pg_statistic being the *visible* casualty doesn't guarantee it's the *only* one — it's just the table the scheduled task happens to read. A one-time sweep (`pg_dump > /dev/null` catches most toast/page-level read errors across user tables) is cheap insurance before declaring realm/user data clean. Until repaired, expect the ClearExpiredEvents/session-purge scheduled tasks to keep failing every cycle — expired events/sessions are accumulating.
spikerj commented 2026-07-14 02:59:04 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — kernel-error signature identified 2026-07-14 ~03:15Z (kernel log pulled from a dreamstream node by ops):

The fleet 'kernel errors' are PCIe AER corrected-error bursts on the NVMe link — root port 10de:10e5 ↔ Micron NVMe 1344:5416, logging Receiver Error, Bad TLP, Replay Timer Timeout, REPLAY_NUM Rollover, all severity=Corrected. Confirmed on dreamstream2 (burst Jul 12 18:27 EDT); with ds1's active counter (53+) and ds4's crash, at least three nodes share it → fleet-wide marginal PCIe/NVMe links (same hardware batch?), not one sick node. This failure class escalating from Corrected→Uncorrected = a drive dropping off the bus, which is a plausible mechanism for both the ds4 panic (this ticket) and the '/mnt/fusionio drive blip' in closed #504/#505.

Recommended fixes (in order):

  1. pcie_aspm=off on the APPEND line of /boot/extlinux/extlinux.conf on the dreamstreams (ASPM transitions are the classic trigger for exactly this error pair on Jetson carriers) — cheap, reversible.
  2. Reseat NVMe modules/adapters on ds1 first (active counter); suspect the common batch.
  3. Check Micron firmware rev + media-error count: smartctl -a /dev/nvme0.
  4. Confirm signature on ds1 (journalctl -k -p err -n 50) and ds4's crash boot (journalctl -k -b -1 -p err) — on ds4, look for escalation to severity=Uncorrected right before the panic.
**QA Team** — kernel-error signature identified 2026-07-14 ~03:15Z (kernel log pulled from a dreamstream node by ops): The fleet 'kernel errors' are **PCIe AER corrected-error bursts on the NVMe link** — root port `10de:10e5` ↔ Micron NVMe `1344:5416`, logging `Receiver Error`, `Bad TLP`, `Replay Timer Timeout`, `REPLAY_NUM Rollover`, all `severity=Corrected`. Confirmed on **dreamstream2** (burst Jul 12 18:27 EDT); with ds1's active counter (53+) and ds4's crash, at least three nodes share it → fleet-wide marginal PCIe/NVMe links (same hardware batch?), not one sick node. This failure class escalating from Corrected→Uncorrected = a drive dropping off the bus, which is a plausible mechanism for both the ds4 panic (this ticket) and the '/mnt/fusionio drive blip' in closed #504/#505. **Recommended fixes (in order):** 1. `pcie_aspm=off` on the `APPEND` line of `/boot/extlinux/extlinux.conf` on the dreamstreams (ASPM transitions are the classic trigger for exactly this error pair on Jetson carriers) — cheap, reversible. 2. Reseat NVMe modules/adapters on ds1 first (active counter); suspect the common batch. 3. Check Micron firmware rev + media-error count: `smartctl -a /dev/nvme0`. 4. Confirm signature on ds1 (`journalctl -k -p err -n 50`) and ds4's crash boot (`journalctl -k -b -1 -p err`) — on ds4, look for escalation to `severity=Uncorrected` right before the panic.
spikerj commented 2026-07-14 03:01:34 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — major timeline update 2026-07-14 ~03:20Z, from dreamstream1's kernel journal (pulled by ops):

dreamstream1 crashed and rebooted tonight at 02:28:06Z — its journal begins at that boot. That completes the 02:27–02:31Z sequence QA watched live without realizing the node itself bounced: 02:27:48Z SignalR/Redis PUBLISH timeouts (the shard member on ds1 going silent) → ds1 down ~02:27:50Z → boot 02:28:06Z → redis shard failover completes 02:28:27Z → 'Crash loop: init.scope on dreamstream1' incident 02:29:04Z → Redis FAIL cleared 02:31:23Z. It recovered in under a minute, so docker node ls never caught it Down between QA sweeps — a live demonstration of the alerting blindness in #555.

Evidence gap: journald is volatile on the dreamstreams (journal starts at boot). Everything ds1 logged before dying — the 53 counted kernel errors, any PCIe Corrected→Uncorrected escalation like ds2's pattern — is gone. Recommend fleet-wide: mkdir -p /var/log/journal && systemctl restart systemd-journald so the next crash leaves a death note. Until then the PCIe/NVMe hypothesis (from ds2's log, earlier note) can't be confirmed or ruled out for ds1/ds4.

New lead on the storage side: ds1's boot log shows CIFS VFS: cifs_mount failed w/return code = -101 (ENETUNREACH — mount raced the network at boot). So these nodes carry CIFS mounts — and 'Transport endpoint is not connected' (exactly what postgres hit on ds7) is the signature error of a dead CIFS session. If /mnt/keycloak is CIFS, the whole storage story unifies: the SMB-serving host going down (ds4?) kills sessions fleet-wide. Please check on a dreamstream: grep -i cifs /etc/fstab — (a) where do these shares come from, (b) do the entries have _netdev,x-systemd.automount (without them, boot-time failures stay failed until manual remount, which may be why ds7's session was already dead when postgres failed over onto it).

(Also for the record: the harmless cgroup2: unknown option "nsdelegate" boot line is noise — systemd passing an option the L4T kernel predates; ignore in error counts.)

**QA Team** — major timeline update 2026-07-14 ~03:20Z, from dreamstream1's kernel journal (pulled by ops): **dreamstream1 crashed and rebooted tonight at 02:28:06Z** — its journal begins at that boot. That completes the 02:27–02:31Z sequence QA watched live without realizing the node itself bounced: 02:27:48Z SignalR/Redis PUBLISH timeouts (the shard member on ds1 going silent) → ds1 down ~02:27:50Z → boot 02:28:06Z → redis shard failover completes 02:28:27Z → 'Crash loop: init.scope on dreamstream1' incident 02:29:04Z → Redis FAIL cleared 02:31:23Z. It recovered in under a minute, so `docker node ls` never caught it Down between QA sweeps — a live demonstration of the alerting blindness in #555. **Evidence gap: journald is volatile on the dreamstreams** (journal starts at boot). Everything ds1 logged before dying — the 53 counted kernel errors, any PCIe Corrected→Uncorrected escalation like ds2's pattern — is gone. **Recommend fleet-wide: `mkdir -p /var/log/journal && systemctl restart systemd-journald`** so the next crash leaves a death note. Until then the PCIe/NVMe hypothesis (from ds2's log, earlier note) can't be confirmed or ruled out for ds1/ds4. **New lead on the storage side:** ds1's boot log shows `CIFS VFS: cifs_mount failed w/return code = -101` (ENETUNREACH — mount raced the network at boot). So these nodes carry **CIFS mounts** — and 'Transport endpoint is not connected' (exactly what postgres hit on ds7) is the signature error of a dead CIFS session. If `/mnt/keycloak` is CIFS, the whole storage story unifies: the SMB-serving host going down (ds4?) kills sessions fleet-wide. Please check on a dreamstream: `grep -i cifs /etc/fstab` — (a) where do these shares come from, (b) do the entries have `_netdev,x-systemd.automount` (without them, boot-time failures stay failed until manual remount, which may be why ds7's session was already dead when postgres failed over onto it). (Also for the record: the harmless `cgroup2: unknown option "nsdelegate"` boot line is noise — systemd passing an option the L4T kernel predates; ignore in error counts.)
spikerj commented 2026-07-14 03:03:29 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — new casualty + revised storage hypothesis 2026-07-14 ~03:25Z:

Seq was hit too. On dreamstream2, Seq's event store threw Transport endpoint is not connected (os error 107) writing /data/Stream/* between ~02:28 and 02:31Z, including error creating a writable ingest buffer — i.e., the central log store was dropping/deferring ingest during that window, plus a disrupted indexing pass. Recovered by 02:31:52Z, UI healthy now. (Meta-note: the fleet's log system failing during the incident is why Seq-based dashboards may show a suspicious absence of errors for 02:28–02:31Z.)

Revised hypothesis on where the CIFS shares live: the mount-death window now correlates with dreamstream1's crash, not dreamstream4's outage:

  • Seq (/data on ds2): errors 02:28–02:31Z — exactly ds1's down window (02:27:48 → boot 02:28:06 → stable ~02:31)
  • keycloak-postgres (/mnt/keycloak on ds7): observed errors 02:24:14–02:29:51Z — fits a ds1 stall-then-crash; nothing ties them to ds4's 01:15Z drop
  • ds1 itself mounts CIFS at boot (earlier note), and ops' shell prompt shows /mnt/infrastructure — consistent with ds1 both consuming and possibly serving the SMB shares

If the SMB server is on dreamstream1, then ds1's crash — not ds4's — broke storage out from under keycloak-postgres and Seq simultaneously, and ds4's role in the postgres story was only forcing the failover placement. The grep -i cifs /etc/fstab check requested earlier would settle it (plus smbstatus/systemctl list-units '*smb*' on ds1). Either way the architectural point sharpens: two databases and the log store all depend on SMB shares from a single Jetson that crashed once tonight already.

**QA Team** — new casualty + revised storage hypothesis 2026-07-14 ~03:25Z: **Seq was hit too.** On dreamstream2, Seq's event store threw `Transport endpoint is not connected (os error 107)` writing `/data/Stream/*` between ~02:28 and 02:31Z, including `error creating a writable ingest buffer` — i.e., **the central log store was dropping/deferring ingest during that window**, plus a disrupted indexing pass. Recovered by 02:31:52Z, UI healthy now. (Meta-note: the fleet's log system failing during the incident is why Seq-based dashboards may show a suspicious *absence* of errors for 02:28–02:31Z.) **Revised hypothesis on where the CIFS shares live:** the mount-death window now correlates with **dreamstream1's crash, not dreamstream4's outage**: - Seq (/data on ds2): errors 02:28–02:31Z — exactly ds1's down window (02:27:48 → boot 02:28:06 → stable ~02:31) - keycloak-postgres (/mnt/keycloak on ds7): observed errors 02:24:14–02:29:51Z — fits a ds1 stall-then-crash; nothing ties them to ds4's 01:15Z drop - ds1 itself mounts CIFS at boot (earlier note), and ops' shell prompt shows `/mnt/infrastructure` — consistent with ds1 both consuming and possibly **serving** the SMB shares If the SMB server is on dreamstream1, then ds1's crash — not ds4's — broke storage out from under keycloak-postgres and Seq simultaneously, and ds4's role in the postgres story was only forcing the failover placement. The `grep -i cifs /etc/fstab` check requested earlier would settle it (plus `smbstatus`/`systemctl list-units '*smb*'` on ds1). Either way the architectural point sharpens: **two databases and the log store all depend on SMB shares from a single Jetson that crashed once tonight already.**
spikerj commented 2026-07-14 06:32:26 +00:00
Author
Owner
Copy Link
Copy Source

Recovered. The node is Ready / Active / Reachable, it is carrying tasks again, and the whole manager set is reachable with a stable leader.

The concrete casualty in this ticket is also cleared: the Keycloak tier is fully 1/1 across keycloak, postgres-keycloak, rabbitmq and backups. No stale over-provisioned task remains, and the No such image reject-loop (the #511 signature) is gone.

Closing on the observed state, with one honest caveat: the root cause of the outage itself was never captured. By the time it was investigated the node had already come back, and nothing recorded why it left. That gap is not incidental — it is exactly what #555 is about (node outages are invisible to ops alerting; alerts arrive after recovery carrying no diagnostic payload). Until that lands, a node flap will keep being diagnosed after the evidence has evaporated. That is the durable fix; reopening this ticket would not produce it.

— macbook-claude-session

Recovered. The node is **Ready / Active / Reachable**, it is carrying tasks again, and the whole manager set is reachable with a stable leader. The concrete casualty in this ticket is also cleared: the Keycloak tier is **fully 1/1** across keycloak, postgres-keycloak, rabbitmq and backups. No stale over-provisioned task remains, and the `No such image` reject-loop (the #511 signature) is gone. Closing on the observed state, with one honest caveat: **the root cause of the outage itself was never captured.** By the time it was investigated the node had already come back, and nothing recorded *why* it left. That gap is not incidental — it is exactly what #555 is about (node outages are invisible to ops alerting; alerts arrive after recovery carrying no diagnostic payload). Until that lands, a node flap will keep being diagnosed after the evidence has evaporated. That is the durable fix; reopening this ticket would not produce it. — macbook-claude-session
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#552