SpikerSoft.Secrets: add bounded retry to OpenBao AppRole login + KV read #607

New Issue

2026-07-16T03:22:51Z

spikerj commented

2026-07-16 03:22:51 +00:00

Problem

SpikerSoft.Secrets (the OpenBao config overlay, Phase 2 of the secrets epic #543/#546) does its AppRole login + KV reads exactly once at startup with no retry. On any transient failure it either falls back to env/appsettings (non-strict, default) or throws (Secrets:Strict=true).

Observed live 2026-07-16 during the InfluxDashboard rollout: the overlay hit a transient 504 Gateway Timeout at the AppRole-login endpoint and fell back to appsettings; a plain docker service update --force re-roll then loaded all 8 values cleanly. The AppRole-login endpoint behind the bao router is a known intermittent 504/429 source (see the CI login-burst incident).

Why it matters

Non-strict workers (Scheduler, Notifications, InfluxDashboard, CodeExecution): a transient 504 = silent fallback to committed appsettings until the next roll. Tolerable but means the flip silently didn't take.
Strict=true services (the API): a transient 504 at boot fails startup. Start-first deploys protect availability (the old task keeps serving), but API deploys can intermittently churn/rollback for no real reason.

Proposed fix

Add a small bounded retry (e.g. 3–5 attempts, exponential backoff + jitter, capped at a few seconds total) around:

VaultSecretSource.LoginAsync (the AppRole login), and
the per-path KV v2 reads.

Retry only on transient conditions (timeouts, 5xx incl. 502/503/504, 429); do not retry genuine auth failures (400/403) or a 404 (which is the normal not-yet-populated-path skip). Keep it behind the existing Secrets:TimeoutSeconds budget or add a Secrets:MaxRetries. Covered by the existing FakeVaultHandler test seam (script a 504-then-200 sequence).

Epic #543 / #546. Hardens every service on the vault, the Strict ones most.

## Problem `SpikerSoft.Secrets` (the OpenBao config overlay, Phase 2 of the secrets epic #543/#546) does its AppRole login + KV reads **exactly once at startup with no retry**. On any transient failure it either falls back to env/appsettings (non-strict, default) or throws (`Secrets:Strict=true`). Observed live 2026-07-16 during the InfluxDashboard rollout: the overlay hit a transient **504 Gateway Timeout at the AppRole-login endpoint** and fell back to appsettings; a plain `docker service update --force` re-roll then loaded all 8 values cleanly. The AppRole-login endpoint behind the bao router is a known intermittent 504/429 source (see the CI login-burst incident). ## Why it matters - **Non-strict workers** (Scheduler, Notifications, InfluxDashboard, CodeExecution): a transient 504 = silent fallback to committed appsettings until the next roll. Tolerable but means the flip silently didn't take. - **`Strict=true` services (the API)**: a transient 504 at boot **fails startup**. Start-first deploys protect availability (the old task keeps serving), but API deploys can intermittently churn/rollback for no real reason. ## Proposed fix Add a small bounded retry (e.g. 3–5 attempts, exponential backoff + jitter, capped at a few seconds total) around: - `VaultSecretSource.LoginAsync` (the AppRole login), and - the per-path KV v2 reads. Retry only on transient conditions (timeouts, 5xx incl. 502/503/504, 429); do **not** retry genuine auth failures (400/403) or a 404 (which is the normal not-yet-populated-path skip). Keep it behind the existing `Secrets:TimeoutSeconds` budget or add a `Secrets:MaxRetries`. Covered by the existing `FakeVaultHandler` test seam (script a 504-then-200 sequence). Epic #543 / #546. Hardens every service on the vault, the Strict ones most.

spikerj commented

2026-07-16 03:58:58 +00:00

Resolved in spikersoft-backend PR #314 (merged to master). Added bounded retry to the SpikerSoft.Secrets OpenBao overlay — VaultSecretSource now retries AppRole login + KV reads on transient failures only (HTTP 429 / 5xx and connection/timeout errors, respecting caller cancellation), NOT on 400/403/404, with exponential backoff + jitter (default MaxRetries=4). Covered by VaultSecretSourceRetryTests (504-then-200, 429-then-200, exhaustion, 403/400 fail-fast, 404 no-retry). This removes the transient-504-at-login fallback-to-appsettings seen on the InfluxDashboard deploy and hardens strict-mode (API) boots. Ships in every image on its next normal deploy. Closing.

Resolved in spikersoft-backend PR #314 (merged to `master`). Added bounded retry to the `SpikerSoft.Secrets` OpenBao overlay — `VaultSecretSource` now retries AppRole login + KV reads on transient failures only (HTTP 429 / 5xx and connection/timeout errors, respecting caller cancellation), NOT on 400/403/404, with exponential backoff + jitter (default `MaxRetries=4`). Covered by `VaultSecretSourceRetryTests` (504-then-200, 429-then-200, exhaustion, 403/400 fail-fast, 404 no-retry). This removes the transient-504-at-login fallback-to-appsettings seen on the InfluxDashboard deploy and hardens strict-mode (API) boots. Ships in every image on its next normal deploy. Closing.

spikerj closed this issue

2026-07-16 03:58:58 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#607