[Epic][Security][Infra] Central secrets management — OpenBao vault + PKI, retire scattered passwords #543
Assignees
spikerj (Joseph Spiker)
Clear assignees
spikerj (Joseph Spiker)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: spikerj/spikersoft-issues#543
Reference in New Issue
Block a user
Passwords/tokens today live in per-repo Gitea Actions secrets (7+ new *_S3_SECRET_KEY from #413 alone), deploy-shell exports (MinIO root), appsettings defaults (RabbitMQ dockerUser/dockerPassword!), and at least one token COMMITTED to git (the shared act_runner registration token, gitea-act-runner/docker-stack.yml — its own comment tracks rotation in README §9).
Decision: OpenBao (Linux Foundation fork of Vault; MPL open source — Vault itself is BUSL now). Rationale: PKI secrets engine covers the "certificate enforced" requirement natively (internal CA → replace generate-internal-certs.sh → mTLS later); AppRole gives per-service/per-runner machine identities with least-privilege policies (same model we just proved with scoped MinIO keys, but centralized + audited + rotatable); Raft HA scales 1→3 nodes; Vault-compatible API = mature .NET client (VaultSharp) + Keycloak OIDC for human SSO. Considered: Infisical (nicer UI, weaker PKI/EE-gated bits), Vaultwarden (right tool for HUMAN passwords — optional later addition, not the machine backbone), Conjur (stagnant).
Phases (tickets follow the #515 pattern):
Root-of-trust rule: Bao's own bootstrap (unseal shares, initial root token) lives OFFLINE, never in Bao, never in git.
QA Team — inventory input for this epic, 2026-07-14: a full fact-checking audit of every spikersoft-backend project (README rebuild, 100+ agents reading every csproj/appsettings/Program.cs) surfaced live-looking credentials committed in appsettings.json files. Types and locations only (values intentionally not repeated here):
SpikerSoft.Api/appsettings.json: Twilio AuthToken, SMTP password,Encryption:MasterKey, InfluxDB token, Keycloak client secrets, DNS-server API token, MediatR license key.spikersoft-backendcommitted identically in all three appsettings variants of at least one project.uploads-svc,metadata-svc) present in swarm service env (visible to anyone withdocker service inspect)..env-10Nfiles on the shared mount carry plaintext InfluxDB creds (already flagged in #554's thread).Suggested handling within the epic: treat all of the above as compromised-by-default (they're in git history regardless of what happens to the working tree) — rotation list for Phase 1/2, and the appsettings files should end up with empty placeholders + README guidance, which the new per-project READMEs now provide hooks for. The full per-project question list from the audit (includes several more config-hygiene items) is with @spikerj.
QA Team — triage update 2026-07-14: per spikerj, the committed appsettings credentials inventoried above are known development values — no emergency rotation needed. The inventory stands as the epic's migration checklist (these are the keys that must come from OpenBao instead of appsettings when their phases land), just without the compromised-by-default urgency in the earlier note.
Epic status roll-up — audited every phase against
origin/master. Phase 2's seam is live fleet-wide; Phases 3 and 4 have not started, and the "retire scattered passwords" half is where the gap is.openbao/docker-stack.yml,docs/openbao-init-runbook.md)SpikerSoft.Secretshost-builder seam (#546)Phase 2's seam is genuinely done and done well.
EventHandlerHostBuilder.cs:76callsAddBaoSecrets(), so all ~30 workers inherit it; the three hosts outside that builder call it directly (SpikerSoft.Api/Program.cs:53,GpuCoordinator/Program.cs:26,NxCacheServer/Program.cs:27). 27 stacks setSecrets__UseBao=true, 26 mount the AppRole creds, andopenbao/provision-services-approle.shmakes it reproducible. That's the hard part.But six stacks were left behind, and two of them are traps rather than oversights:
spikersoft-embeddings/docker-stack-gpu.ymlandspikersoft-quiz-generation/docker-stack-gpu.ymlare GPU variants of services whose main stack has the seam. Once Stage 4 blanks the committed plaintext, deploying a GPU variant boots with empty credentials and will look like a config bug. Details on #546.Phase 3 has no artifact at all.
git grep -rniE "pki|internal ca|mtls"acrossopenbao/*anddocs/*.mdreturns nothing outside the secrets-management doc itself. No CA role, no cert issuance, no mTLS on the stateful tier.The honest state of "retire scattered passwords." This epic's headline promise is the part that hasn't landed, and I have concrete numbers from auditing #633 today: six sets of committed plaintext credentials remain — four in backend
appsettings*.json(the Gitea account password, InfluxDB admin token, Keycloak client secret, RabbitMQdockerUser/dockerPassword), plus five literals inmailserver/docker-stack.ymlthat were outside every previous sweep's scope. One partial win: PR #498 blanked every committed Seq key exceptCodeExecution/appsettings.Example.json.Two things make this more urgent than an infrastructure-hygiene epic normally is:
spikerj/*image is freely downloadable. If any of those six credential sets was ever in a build context, it's in a publicly-pullable layer.appsettings.jsonfiles viasonar-scan.yml:75-79, and the suppressions are file-scoped rather than value-scoped. So new secrets in those files won't be flagged either.Suggested sequencing: close #546's six-stack gap (cheap, and it's a live trap), then #633's rotation — before Phase 3, and before #563 deploys another service carrying a committed secret. PKI is the right long-term direction but it doesn't reduce today's exposure.