[Go-Live Checklist][Security] Rotate + Bao-migrate the dev-environment secrets before production #633
Assignees
spikerj (Joseph Spiker)
Clear assignees
spikerj (Joseph Spiker)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: spikerj/spikersoft-issues#633
Reference in New Issue
Block a user
From a careful pass over all 40 open SonarQube vulnerabilities on api.spikersoft.com (2026-07-17). Values verified against the working tree at master — this is the confirmed-real subset (3 false positives were closed in SonarQube with comments; the rest are stale-line issues a fresh scan will re-baseline). No secret values are reproduced here — locations only.
Committed real credentials (rotate first, then move consumption to OpenBao per #546; deleting from HEAD is not enough — they are burned into git history):
spikerj—SpikerSoft.EventHandlers.CodeExecution/appsettings.json(Git:Authentication:HTTPS) andappsettings.Production.json. This is a personal admin credential used by the student-repo provisioner. Rotate the account password; give the provisioner a dedicated machine account or scoped token (Gitea:AccessTokenis already supported by the provisioner config and currently empty).SpikerSoft.EventHandlers.InfluxDashboard/appsettings.json+appsettings.Production.json(InfluxDB + InfluxDashboard sections). Rotate in InfluxDB, store insecret/services/....spikersoft-backend—SpikerSoft.AI.MCPServer/appsettings.json+appsettings.Production.json. Regenerate in Keycloak.dockerUser/dockerPassword— verified in spikersoft-infrastructure (rabbitmq/docker-stack.yml,keycloak/docker-stack.yml) and consumed verbatim inSpikerSoft.EventHandlers.DockerMonitor/appsettings.Production.jsonand as fallback defaults in the two Python services (Trellis3D.Python,ImageDescription.Python). These read as local-dev placeholders but ARE the production broker credentials — rotating them touches every publisher/consumer, so coordinate with a deploy window.Suggested order: 1 (personal credential, highest blast radius) → 3 → 2 → 5 → 4. Each rotation should land the new value in OpenBao (
secret/services/<svc>), extend the provisioning script, and delete the literal from appsettings (services already call AddBaoSecrets — #546 Stage 0).Related: #546 (Bao service-secrets migration), #545 (CI secrets migration). Filed from the SonarQube triage loop at spikerj's direction.
Closure audit (2026-07-17 05:50): all 37 still-open SonarQube vulnerabilities on the backend are accounted for — none unexamined. 13 are stale SpikerSoft.Api/appsettings.json lines whose secrets already moved to Bao (#546 Stage 0) and will auto-close on the first post-#632 rescan; the remaining 24 map exactly onto this ticket's five rotation items (CodeExecution Gitea password + worker ApiKeys, InfluxDashboard token, MCPServer client secret, the shared Seq ApiKey family, prod-RabbitMQ dockerUser/dockerPassword incl. the two Python env-defaults). The 3 genuine false positives were resolved in SonarQube with reasoning comments. Net: once this ticket's rotations are done and a rescan runs, the backend vulnerability list reads zero-noise.
Addition found while mapping the SonarQube stack for the #632 CE investigation: spikersoft-infrastructure/sonarqube/docker-stack.yml carries the Postgres credentials in plaintext (POSTGRES_USER/POSTGRES_PASSWORD + the same pair in SONAR_JDBC_*). Internal-overlay exposure only, but same policy violation class as the rest of this ticket — fold into the rotation pass as item 6 (rotate in postgres, move to a swarm secret or Bao-hydrated env per the secrets-management doc).
Partial progress (PR #357, merged): the CodeExecution appsettings.Example.json no longer carries live secrets — the real prod RabbitMQ password and the live Production Seq ApiKey it leaked are replaced with placeholders matching the file's convention. This was the one member of the secret set safe to fix WITHOUT a value rotation (example files aren't loaded at runtime). The 24 runtime-config secrets still need value rotation + Bao-sourced removal, which stays open on this ticket and needs your admin access.
Reclassified per spikerj (2026-07-17): this is an isolated dev environment, so the hardcoded secrets carry no production exposure and the 24 SonarQube findings have been ACCEPTED (not false-positived — they are genuinely hardcoded, just not a live risk here) with that rationale recorded on each. The one example-file finding was actually fixed (PR #357).
This ticket stays OPEN but is no longer an urgent security cleanup — it is now the GO-LIVE ROTATION CHECKLIST: before production, rotate + move to OpenBao (SpikerSoft.Secrets already wires AddBaoSecrets on the hosts): the shared Seq ApiKey (~8 services), InfluxDB token, Keycloak spikersoft-backend client secret, MCPServer client secret, the SonarQube postgres creds (infra stack), and the dockerUser/dockerPassword prod RabbitMQ/Mongo credentials. Not blocking day-to-day work.
[Security][Backend] Rotate and migrate real secrets committed in appsettings files (SonarQube S6418/S2068 — verified true positives)to [Go-Live Checklist][Security] Rotate + Bao-migrate the dev-environment secrets before productionSweep note: PR #357 scrubbed the live secrets from the example config, but the ticket's ROTATION + Bao-migration halves have no merged evidence — leaving open (rotation is the part that matters post-leak).
Re-audited all five items against
origin/master. One is done, four are untouched. Paths and value lengths only below — no values reproduced, per the secrets policy.spikerjdockerUser/dockerPassword1 — Gitea personal credential. Still present.
SpikerSoft.EventHandlers.CodeExecution/appsettings.json→/Git/Authentication/HTTPS/Passwordis NON-EMPTY (len 11). The recommended replacement path is still unused:/Gitea/AccessTokenin the same file is EMPTY. This was ranked highest blast radius and nothing has moved.2 — InfluxDB admin token. Still present.
SpikerSoft.EventHandlers.InfluxDashboard/appsettings.json→/InfluxDB/TokenNON-EMPTY (len 88).3 — Keycloak client secret. Still present.
SpikerSoft.AI.MCPServer/appsettings.json→/SpikerSoftApi/ClientSecretNON-EMPTY (len 32).4 — Seq keys: swept, with one miss — and it's the one this ticket specifically warned about. PR #498 (
e8ba7bff, merge8b1387b8) blanked every committedSeq:ApiKeyacross the fleet. I enumerated everyappsettings*.jsonon master: all are EMPTY except one —SpikerSoft.EventHandlers.CodeExecution/appsettings.Example.json→/Seq/ApiKeyNON-EMPTY (len 16)which is exactly the file the body flagged: "the Example file carries the same value as Production, so the 'example' is a live key."
One correction to that claim, since I checked rather than assumed: I hashed the current Example value and the Production value as it stood immediately before the blanking commit — they differ. So the Example key is a distinct value, not a copy of Production. That does not make it safe; it means an unidentified 16-char key is still committed, and the sweep that removed all its siblings skipped it. Whether it is still valid can only be settled in Seq itself, not from git.
5 — RabbitMQ credentials. Still present, and demonstrably still live.
/Rabbit/Passwordis NON-EMPTY (len 14) inSpikerSoft.EventHandlers.CodeExecution/appsettings.json, andSpikerSoft.Api/appsettings.jsoncarries both/Rabbit/Passwordand/RabbitMQ/Passwordat the same length. Independent confirmation these are the production credentials: during the 2026-07-29 outage (#895) the broker was rebuilt fromRABBITMQ_DEFAULT_USER/PASSand every consumer reconnected — i.e.dockerUseris still the account the whole fleet authenticates with.Worth noting alongside this:
SpikerSoft.Api/appsettings.jsonis otherwise clean —InfluxDB/Token,Keycloak/*/secret,Stripe/*,Twilio/AuthToken,Email/Password,DnsServer/Tokenare all EMPTY with/Secrets/Address+/Secrets/Mountwired for Bao. So the pattern works; it just hasn't been applied to the four services above.Staying open. Suggested order is unchanged (1 → 3 → 2 → 5 → 4), with the Example-file key folded into step 4 — or pulled forward, since it's a one-line deletion.
New finding — a sixth item, not in the original five. Turned up while auditing #610. Locations only below, no values.
spikersoft-infrastructure/mailserver/docker-stack.ymlcommits five literal plaintext credentials. I checked the shape of each rather than assuming — none use${VAR:?...}env indirection, they are all inline literals:DBPASSRSPAMD_PASSWORDDBPASSMYSQL_RANDOM_ROOT_PASSWORDMYSQL_PASSWORDFingerprinting them (SHA-256, first 8 hex) shows
:108DBPASSand:180MYSQL_PASSWORDare the same value — consistent with one mail-database password written in two places, which means rotating it requires touching both or the stack breaks.This matters for the Go-Live scope for two reasons beyond "another committed secret":
spikersoft-backendentirely. The original five were all in backendappsettings*.json, so a sweep scoped to that repo — which is how the Seq cleanup in PR #498 was run — would never have found these. Worth confirming whether any other infra stack files carry literals; this one was found incidentally.Same treatment as the rest: rotate first (they're in git history, so deleting from HEAD does not revoke them), land the new values in OpenBao, then switch the stack to the
${VAR:?...}required-variable form already used successfully attraefik/docker-stack.yml:64— that pattern fails the deploy closed rather than silently shipping an empty credential.Related: #610 (centralize Let's Encrypt certs in Bao) names this same file's plaintext as an adjacent gap and is itself NOT DONE.