OpenBao HA is silently degraded by sealed standbys — seal state has no monitoring, turning one node loss into a 2.5h write outage (#885 post-mortem) #888

New Issue

2026-07-28T14:50:45Z

spikerj commented

2026-07-28 14:50:45 +00:00

Post-mortem of the OpenBao leg of #885 (2026-07-28), from openbao-1/2/3 service logs. The raft cluster did exactly what raft does — the operating model around it is what failed.

What actually happened

Silent pre-condition (~4 days): openbao-2 (ds3) restarted 4 days ago and came back sealed — by design (unseal shares are offline; the healthcheck deliberately reports sealed as healthy to stop swarm flapping, and the router just skips sealed nodes). Nothing alerted (#756 mail is dead, and nothing watches seal state anyway). The "3-node HA cluster" was really leader (openbao-1) + one unsealed follower (openbao-3 on ds4) + dead weight.
12:14:13Z: ds4 started dying → leader openbao-1 "failed to contact quorum of nodes, stepping down" (raft safety — correct behavior).
12:14→14:42Z: openbao-1 ran 1,195 election campaigns at term 5: pre-vote campaign failed ... tally=1 refused=2 votesNeeded=2. Election needs 2 of 3 voters; the only other live voter was sealed. Mathematically unwinnable — not a raft bug. Cluster leaderless → every write (incl. AppRole login) → HTTP 500 local node not active but active cluster node not found → CI's bao-secrets step dead in both app repos (master deploy runs 14:11–14:34Z all red).
14:42:09Z: openbao-2 was manually unsealed → one second later openbao-1 won (election won: term=5 tally=2). Service restored.
Right now: openbao-3 (ds4) rejoined at 14:30 but is still sealed — the cluster is back to leader + 1 unsealed follower, i.e. one restart/seal away from an identical repeat.

The gap

Sealed-after-restart is a deliberate security posture, but it silently converts N-node HA into (N-sealed)-node HA and nobody finds out until quorum math fails under a second fault. A sealed standby is indistinguishable from a healthy one in every signal we currently look at.

Proposed fixes (in order of value)

Immediate: unseal openbao-3 (human + key shares, per docs/openbao-init-runbook.md). Add "verify all three nodes unsealed (/v1/sys/health per node, not via the router)" to the node-recovery runbook — resealing happens on every reboot.
Monitoring: a scheduled probe (scheduler worker or cron job) that checks each openbao backend's /v1/sys/health?standbyok=true directly and logs an ERROR to Seq when any node reports sealed:true for >10 min — Seq-visible even while alert mail (#756) is down. The per-node distinction matters: the router masks sealed nodes by design, so probing bao.spikersoft.com tells you nothing about standby health.
Evaluate auto-unseal for the standbys (OpenBao 2.5.x seal options; anything beats 4-day silent seal), acknowledging the security tradeoff vs offline shares — even auto-unseal for standbys only, keeping manual for cold-start, would have kept quorum through #885.

Evidence: docker service logs openbao_openbao-{1,2} around 12:14 and 14:42 on 2026-07-28; #885 for the ds4 timeline.

Post-mortem of the OpenBao leg of #885 (2026-07-28), from openbao-1/2/3 service logs. The raft cluster did exactly what raft does — the operating model around it is what failed. ## What actually happened 1. **Silent pre-condition (~4 days):** `openbao-2` (ds3) restarted 4 days ago and came back **sealed** — by design (unseal shares are offline; the healthcheck deliberately reports sealed as healthy to stop swarm flapping, and the router just skips sealed nodes). Nothing alerted (#756 mail is dead, and nothing watches seal state anyway). The "3-node HA cluster" was really leader (openbao-1) + one unsealed follower (openbao-3 on ds4) + dead weight. 2. **12:14:13Z:** ds4 started dying → leader openbao-1 "failed to contact quorum of nodes, stepping down" (raft safety — correct behavior). 3. **12:14→14:42Z:** openbao-1 ran **1,195 election campaigns** at term 5: `pre-vote campaign failed ... tally=1 refused=2 votesNeeded=2`. Election needs 2 of 3 voters; the only other live voter was sealed. Mathematically unwinnable — not a raft bug. Cluster leaderless → every write (incl. AppRole login) → HTTP 500 `local node not active but active cluster node not found` → CI's bao-secrets step dead in both app repos (master deploy runs 14:11–14:34Z all red). 4. **14:42:09Z:** openbao-2 was manually unsealed → **one second later** openbao-1 won (`election won: term=5 tally=2`). Service restored. 5. **Right now:** openbao-3 (ds4) rejoined at 14:30 but is **still sealed** — the cluster is back to leader + 1 unsealed follower, i.e. one restart/seal away from an identical repeat. ## The gap Sealed-after-restart is a deliberate security posture, but it silently converts N-node HA into (N-sealed)-node HA and nobody finds out until quorum math fails under a second fault. A sealed standby is indistinguishable from a healthy one in every signal we currently look at. ## Proposed fixes (in order of value) 1. **Immediate:** unseal openbao-3 (human + key shares, per docs/openbao-init-runbook.md). Add "verify all three nodes unsealed (`/v1/sys/health` per node, not via the router)" to the node-recovery runbook — resealing happens on every reboot. 2. **Monitoring:** a scheduled probe (scheduler worker or cron job) that checks each openbao backend's `/v1/sys/health?standbyok=true` directly and logs an ERROR to Seq when any node reports `sealed:true` for >10 min — Seq-visible even while alert mail (#756) is down. The per-node distinction matters: the router masks sealed nodes by design, so probing bao.spikersoft.com tells you nothing about standby health. 3. **Evaluate auto-unseal** for the standbys (OpenBao 2.5.x seal options; anything beats 4-day silent seal), acknowledging the security tradeoff vs offline shares — even auto-unseal for standbys only, keeping manual for cold-start, would have kept quorum through #885. Evidence: `docker service logs openbao_openbao-{1,2}` around 12:14 and 14:42 on 2026-07-28; #885 for the ds4 timeline.

spikerj closed this issue

2026-07-29 15:37:00 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#888