spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Security][CI] OpenBao Phase 1: Gitea Actions secrets → AppRole fetch; rotate the committed runner token (epic #543) #545

New Issue
Closed
opened 2026-07-13 21:07:24 +00:00 by spikerj · 9 comments
spikerj commented 2026-07-13 21:07:24 +00:00
Owner
Copy Link
Copy Source

Replace per-repo Actions secrets (DOCKER_PASSWORD, the seven S3_SECRET_KEYs, ACCESS_TOKEN_GITEA, DISCORD, HF_TOKEN) with a fetch-from-Bao step at job start: one AppRole bootstrap identity per repo (role_id as the sole Actions secret; secret_id delivered via response-wrapping or runner-local file), policies scoping each repo to its paths. FIRST WIN regardless of the rest: rotate the act_runner registration token that is committed in gitea-act-runner/docker-stack.yml (README §9) and move the new one to Bao.

Replace per-repo Actions secrets (DOCKER_PASSWORD, the seven *_S3_SECRET_KEYs, ACCESS_TOKEN_GITEA, DISCORD_*, HF_TOKEN) with a fetch-from-Bao step at job start: one AppRole bootstrap identity per repo (role_id as the sole Actions secret; secret_id delivered via response-wrapping or runner-local file), policies scoping each repo to its paths. FIRST WIN regardless of the rest: rotate the act_runner registration token that is committed in gitea-act-runner/docker-stack.yml (README §9) and move the new one to Bao.
spikerj commented 2026-07-13 21:12:21 +00:00
Author
Owner
Copy Link
Copy Source

Infra PR #62 (stacked on #61): four least-privilege CI policies + idempotent AppRole provisioning script — per-repo Actions secrets shrink to BAO_ROLE_ID/BAO_SECRET_ID. Exemplar workflow fetch step (to convert repos after provisioning):

TOKEN=$(curl -s -X POST $BAO_ADDR/v1/auth/approle/login -d "{"role_id":"$BAO_ROLE_ID","secret_id":"$BAO_SECRET_ID"}" | jq -r .auth.client_token)
VAL=$(curl -s -H "X-Vault-Token: $TOKEN" $BAO_ADDR/v1/secret/data/ci/backend/minio/scanner | jq -r .data.data.secret_key)
echo "::add-mask::$VAL"; echo "SCANNER_S3_SECRET_KEY=$VAL" >> $GITHUB_ENV

Workflow conversions land per-repo once #61+#62 are deployed/provisioned.

Infra PR #62 (stacked on #61): four least-privilege CI policies + idempotent AppRole provisioning script — per-repo Actions secrets shrink to BAO_ROLE_ID/BAO_SECRET_ID. Exemplar workflow fetch step (to convert repos after provisioning): TOKEN=$(curl -s -X POST $BAO_ADDR/v1/auth/approle/login -d "{\"role_id\":\"$BAO_ROLE_ID\",\"secret_id\":\"$BAO_SECRET_ID\"}" | jq -r .auth.client_token) VAL=$(curl -s -H "X-Vault-Token: $TOKEN" $BAO_ADDR/v1/secret/data/ci/backend/minio/scanner | jq -r .data.data.secret_key) echo "::add-mask::$VAL"; echo "SCANNER_S3_SECRET_KEY=$VAL" >> $GITHUB_ENV Workflow conversions land per-repo once #61+#62 are deployed/provisioned.
spikerj commented 2026-07-15 16:19:15 +00:00
Author
Owner
Copy Link
Copy Source

Progress update (2026-07-15) — Phase 1 underway; first service fully cut over.

  • bao-secrets composite action (AppRole login → export KV v2 secrets into the job env, masked).
  • New PUBLIC repo spikerj/gitea-actions now hosts the action. Required because CI runners can't clone the private spikersoft-infrastructure repo (job token is repo-scoped → "Repository not found"); the action is pure logic (no secrets), so it's safe public. Reference by full URL so Gitea resolves it locally instead of github.com.
  • Provisioning scripts on master: provision-ci-approles.sh (per-repo AppRoles + least-priv policies), bootstrap-secrets.sh (populate secret/ci/shared/*), MinIO CI-key rotation/provisioning (infra PRs #88–#94).
  • Notifications = pilot: deployed green end-to-end via Bao (build → deploy → discord notify), no plaintext Actions secrets in the path. Fixed a resolution bug where a bare uses: owner/repo/... hit github.com (backend PR #300).
  • Batch 1 (backend PR #301): 16 shared-secret workflows converted via a validated transformer.

Remaining before this can close:

  • Convert the rest: 6 structural variants + tests.yml, then 8 special-secret workflows (S3 → secret/ci/backend/minio/<svc>, Sonar, and the wasm runner token).
  • Rotate the committed act_runner registration token → secret/infra/act-runner/registration (still open).
  • Delete the plaintext Actions secrets + strip plaintext from committed stack files, only once every consumer of each is migrated.
  • Remove the now-superseded action copy from the private infra repo.
**Progress update (2026-07-15) — Phase 1 underway; first service fully cut over.** - **`bao-secrets` composite action** (AppRole login → export KV v2 secrets into the job env, masked). - **New PUBLIC repo `spikerj/gitea-actions`** now hosts the action. Required because CI runners can't clone the *private* `spikersoft-infrastructure` repo (job token is repo-scoped → "Repository not found"); the action is pure logic (no secrets), so it's safe public. Reference by full URL so Gitea resolves it locally instead of github.com. - Provisioning scripts on master: `provision-ci-approles.sh` (per-repo AppRoles + least-priv policies), `bootstrap-secrets.sh` (populate `secret/ci/shared/*`), MinIO CI-key rotation/provisioning (infra PRs #88–#94). - **Notifications = pilot: deployed green end-to-end via Bao** (build → deploy → discord notify), no plaintext Actions secrets in the path. Fixed a resolution bug where a bare `uses: owner/repo/...` hit github.com (backend PR #300). - **Batch 1 (backend PR #301): 16 shared-secret workflows converted** via a validated transformer. Remaining before this can close: - Convert the rest: 6 structural variants + `tests.yml`, then 8 special-secret workflows (S3 → `secret/ci/backend/minio/<svc>`, Sonar, and the `wasm` runner token). - **Rotate the committed act_runner registration token** → `secret/infra/act-runner/registration` (still open). - Delete the plaintext Actions secrets + strip plaintext from committed stack files, only once every consumer of each is migrated. - Remove the now-superseded action copy from the private infra repo.
spikerj commented 2026-07-15 18:02:30 +00:00
Author
Owner
Copy Link
Copy Source

Phase 1 rollout progress (spikersoft-backend)

Batch Workflows PR State
pilot notifications #300 merged, green e2e
1 13 shared-secret-only #301 merged
2 6 variants (single-arch/tests) #302 merged
3a 7 S3-key (metadata/scanner/uploads/videos) #303 merged (closed #581)
3b api, blog-media-processor, scheduler #304 open

Batch 3b was unblocked by provisioning the api/blogs/scheduler MinIO svc-users and writing their keys to secret/ci/backend/minio/{api,blogs,scheduler}. All three now fetch their *_S3_SECRET_KEY (and the shared registry/deploy/discord secrets) from OpenBao; the leftover half-migrated env: *_S3_SECRET_KEY blocks on blog-media/scheduler (which pointed at now-absent Actions secrets) are gone.

Reliability fixes folded in along the way: the CI login-burst 504/429 (batch tokens + retry action + loosened quota) and #590 on the three batch-3b workflows (cancel-in-progress gated to non-master refs).

Remaining before Actions secrets can be deleted:

  • Batch 4 specials: sonar-scan (needs a Bao path for SONARQUBE_HOST/TOKEN) and wasm (rotate the committed act_runner RUNNER_TOKEN).
  • Then: delete the plaintext Actions secrets, strip plaintext from committed stack files, and remove the superseded action copy in the private infra repo.
  • #590 remains open — the cancel-in-progress bug is fleet-wide; only the workflows touched by these batches are fixed so far. A dedicated sweep will cover the rest.
**Phase 1 rollout progress (spikersoft-backend)** | Batch | Workflows | PR | State | |---|---|---|---| | pilot | notifications | #300 | merged, green e2e | | 1 | 13 shared-secret-only | #301 | merged | | 2 | 6 variants (single-arch/tests) | #302 | merged | | 3a | 7 S3-key (metadata/scanner/uploads/videos) | #303 | merged (closed #581) | | 3b | api, blog-media-processor, scheduler | #304 | **open** | Batch 3b was unblocked by provisioning the `api`/`blogs`/`scheduler` MinIO svc-users and writing their keys to `secret/ci/backend/minio/{api,blogs,scheduler}`. All three now fetch their `*_S3_SECRET_KEY` (and the shared registry/deploy/discord secrets) from OpenBao; the leftover half-migrated `env: *_S3_SECRET_KEY` blocks on blog-media/scheduler (which pointed at now-absent Actions secrets) are gone. Reliability fixes folded in along the way: the CI login-burst 504/429 (batch tokens + retry action + loosened quota) and **#590** on the three batch-3b workflows (cancel-in-progress gated to non-master refs). **Remaining before Actions secrets can be deleted:** - Batch 4 specials: `sonar-scan` (needs a Bao path for SONARQUBE_HOST/TOKEN) and `wasm` (rotate the committed act_runner RUNNER_TOKEN). - Then: delete the plaintext Actions secrets, strip plaintext from committed stack files, and remove the superseded action copy in the private infra repo. - **#590** remains open — the cancel-in-progress bug is fleet-wide; only the workflows touched by these batches are fixed so far. A dedicated sweep will cover the rest.
spikerj commented 2026-07-15 18:04:38 +00:00
Author
Owner
Copy Link
Copy Source

Batch 3b merged — spikersoft-backend PR #304 to master (merge d152c626). api/blog-media-processor/scheduler now fetch every secret (registry, deploy, discord, and their *_S3_SECRET_KEY) from OpenBao. All S3-key backend workflows are migrated. Remaining: batch 4 specials (sonar-scan needs a Bao path for SONARQUBE_*; wasm needs the committed act_runner RUNNER_TOKEN rotated), then delete the plaintext Actions secrets + strip plaintext from committed stack files + remove the superseded infra action copy.

**Batch 3b merged** — spikersoft-backend PR #304 to `master` (merge `d152c626`). api/blog-media-processor/scheduler now fetch every secret (registry, deploy, discord, and their `*_S3_SECRET_KEY`) from OpenBao. All S3-key backend workflows are migrated. Remaining: batch 4 specials (sonar-scan needs a Bao path for SONARQUBE_*; wasm needs the committed act_runner RUNNER_TOKEN rotated), then delete the plaintext Actions secrets + strip plaintext from committed stack files + remove the superseded infra action copy.
spikerj commented 2026-07-15 18:21:29 +00:00
Author
Owner
Copy Link
Copy Source

Batch 4 up (PR #306) — sonar-scan + wasm, the last two backend workflows on plaintext Actions secrets. Bao paths populated on the swarm via infra provision-batch4-secrets.sh (infra PR #100): secret/ci/backend/sonar (host, token) and the rotated secret/ci/shared/gitea token for wasm's package publish.

Once #306 merges green, no spikersoft-backend workflow references a plaintext Actions secret. Remaining Phase 1 tail:

  1. Delete the now-orphaned Actions secrets (SONARQUBE_HOST/TOKEN, RUNNER_TOKEN — the last after wasm runs green once on the rotated token; then the shared DOCKER_PASSWORD/DREAMSTREAM1_IP/SSH_USERNAME/PASSWORD/DISCORD_*/*_S3_SECRET_KEY once the other repos are also migrated).
  2. Strip plaintext from committed stack files + remove the superseded action copy in the private infra repo.
  3. Day-one step 6: revoke the initial root token (SSO admin verified).
**Batch 4 up (PR #306)** — sonar-scan + wasm, the last two backend workflows on plaintext Actions secrets. Bao paths populated on the swarm via infra `provision-batch4-secrets.sh` (infra PR #100): `secret/ci/backend/sonar` (host, token) and the rotated `secret/ci/shared/gitea` token for wasm's package publish. Once #306 merges green, **no spikersoft-backend workflow references a plaintext Actions secret.** Remaining Phase 1 tail: 1. Delete the now-orphaned Actions secrets (`SONARQUBE_HOST/TOKEN`, `RUNNER_TOKEN` — the last after wasm runs green once on the rotated token; then the shared `DOCKER_PASSWORD`/`DREAMSTREAM1_IP`/`SSH_USERNAME`/`PASSWORD`/`DISCORD_*`/`*_S3_SECRET_KEY` once the other repos are also migrated). 2. Strip plaintext from committed stack files + remove the superseded action copy in the private infra repo. 3. Day-one step 6: revoke the initial root token (SSO admin verified).
spikerj commented 2026-07-15 18:49:14 +00:00
Author
Owner
Copy Link
Copy Source

Phase 1 code-complete & CI-verified (2026-07-15). All backend workflows fetch from OpenBao; batch 4 (sonar-scan + wasm, PR #306) merged and both ran green — wasm published its bundle on the rotated secret/ci/shared/gitea token (the publish curls use --fail, so a green job = a working token). The #590-swept workflows are also re-deploying green through the Bao flow.

Cleanup PRs merged: infra #101 (removed the superseded private-repo action copy) and #102 (runner registration token → shell-env interpolation; 9 inline copies purged from the committed stack file).

Remaining to fully close #545:

  1. Delete the now-orphaned plaintext Actions secrets on spikersoft-backend (zero workflow references confirmed): DOCKER_PASSWORD, DREAMSTREAM1_IP, SSH_USERNAME, PASSWORD, DISCORD_ID, DISCORD_TOKEN, SONARQUBE_HOST, SONARQUBE_TOKEN, SCANNER_S3_SECRET_KEY, METADATA_S3_SECRET_KEY, UPLOADS_S3_SECRET_KEY, RUNNER_TOKEN. Keep BAO_ROLE_ID/BAO_SECRET_ID.
  2. Rotate the burned runner registration token at Gitea > Admin > Actions > Runners (the old value is still in git history).

Note: the shared secrets remain live for the other repos (angular, artpipe) until those migrate — that's separate follow-up, not #545.

**Phase 1 code-complete & CI-verified (2026-07-15).** All backend workflows fetch from OpenBao; batch 4 (sonar-scan + wasm, PR #306) merged and both ran green — wasm published its bundle on the rotated `secret/ci/shared/gitea` token (the publish curls use `--fail`, so a green job = a working token). The #590-swept workflows are also re-deploying green through the Bao flow. Cleanup PRs merged: infra #101 (removed the superseded private-repo action copy) and #102 (runner registration token → shell-env interpolation; 9 inline copies purged from the committed stack file). **Remaining to fully close #545:** 1. Delete the now-orphaned plaintext Actions secrets on spikersoft-backend (zero workflow references confirmed): `DOCKER_PASSWORD`, `DREAMSTREAM1_IP`, `SSH_USERNAME`, `PASSWORD`, `DISCORD_ID`, `DISCORD_TOKEN`, `SONARQUBE_HOST`, `SONARQUBE_TOKEN`, `SCANNER_S3_SECRET_KEY`, `METADATA_S3_SECRET_KEY`, `UPLOADS_S3_SECRET_KEY`, `RUNNER_TOKEN`. Keep `BAO_ROLE_ID`/`BAO_SECRET_ID`. 2. Rotate the burned runner registration token at Gitea > Admin > Actions > Runners (the old value is still in git history). Note: the shared secrets remain live for the *other* repos (angular, artpipe) until those migrate — that's separate follow-up, not #545.
spikerj commented 2026-07-15 18:52:05 +00:00
Author
Owner
Copy Link
Copy Source

All 12 orphaned plaintext Actions secrets deleted from spikersoft-backend (Gitea API). The repo now holds only BAO_ROLE_ID + BAO_SECRET_ID; every real secret lives in OpenBao.

#545 is effectively complete for spikersoft-backend. The only residual is a manual, human-held action: rotate the burned runner registration token at Gitea > Admin > Actions > Runners (its old value is still in git history — PR #102 only stopped committing new copies). Leaving this ticket OPEN until that rotation is confirmed done.

Follow-on (separate scope): migrate spikersoft-angular + spikersoft-artpipe CI to the same bao-secrets pattern, which is what still keeps the shared secret values live in those repos.

**All 12 orphaned plaintext Actions secrets deleted** from spikersoft-backend (Gitea API). The repo now holds only `BAO_ROLE_ID` + `BAO_SECRET_ID`; every real secret lives in OpenBao. **#545 is effectively complete for spikersoft-backend.** The only residual is a manual, human-held action: rotate the burned runner registration token at Gitea > Admin > Actions > Runners (its old value is still in git history — PR #102 only stopped committing new copies). Leaving this ticket OPEN until that rotation is confirmed done. Follow-on (separate scope): migrate spikersoft-angular + spikersoft-artpipe CI to the same bao-secrets pattern, which is what still keeps the *shared* secret values live in those repos.
spikerj commented 2026-07-15 20:09:53 +00:00
Author
Owner
Copy Link
Copy Source

spikersoft-angular fully migrated & cleaned up (2026-07-15). PR #192 (workflows → bao-secrets) merged and green on master; the 3 angular-specific paths secret/ci/angular/{seq,e2e,sonar} populated (infra provision-angular-secrets.sh, PRs #103+#104), and all 10 orphaned plaintext Actions secrets deleted — the repo now holds only BAO_ROLE_ID/BAO_SECRET_ID, matching backend.

Two identities (e2e.staff + e2e.student) are stored in secret/ci/angular/e2e; the suite runs single-identity (staff) for now — the role split is tracked in #605.

Phase 1 remaining:

  1. spikersoft-artpipe — the last repo to migrate (ci-artpipe AppRole + policy already exist).
  2. Rotate the burned runner registration token at Gitea > Admin > Actions > Runners (human-held).
**spikersoft-angular fully migrated & cleaned up (2026-07-15).** PR #192 (workflows → bao-secrets) merged and green on master; the 3 angular-specific paths `secret/ci/angular/{seq,e2e,sonar}` populated (infra provision-angular-secrets.sh, PRs #103+#104), and all 10 orphaned plaintext Actions secrets deleted — the repo now holds only `BAO_ROLE_ID`/`BAO_SECRET_ID`, matching backend. Two identities (e2e.staff + e2e.student) are stored in `secret/ci/angular/e2e`; the suite runs single-identity (staff) for now — the role split is tracked in #605. **Phase 1 remaining:** 1. **spikersoft-artpipe** — the last repo to migrate (ci-artpipe AppRole + policy already exist). 2. Rotate the burned runner registration token at Gitea > Admin > Actions > Runners (human-held).
spikerj commented 2026-07-15 20:22:28 +00:00
Author
Owner
Copy Link
Copy Source

Phase 1 complete. All three repos migrated to the OpenBao bao-secrets AppRole fetch — spikersoft-backend, spikersoft-angular (PR #192), spikersoft-artpipe (PR #21) — every CI credential now resolves from OpenBao at runtime; the repos carry only BAO_ROLE_ID/BAO_SECRET_ID. The committed act_runner registration token was de-inlined to shell-env (infra #102) and rotated at its Gitea source. Related reliability work landed alongside: CI login-burst 504 fix (batch tokens + retry action + quota), and #590 (closed). Superseded private-repo action copy removed (#101).

Residual hygiene (non-blocking): spikersoft-artpipe still has two orphaned plaintext Actions secrets (DOCKER_PASSWORD, ACCESS_TOKEN_GITEA) that no workflow references — pending deletion to match backend/angular. Closing Phase 1; deferred sub-work tracked separately (#605 two-role e2e). Phases 2 (service-config secrets) + 3 (PKI) continue under epic #543.

Phase 1 complete. **All three repos** migrated to the OpenBao `bao-secrets` AppRole fetch — spikersoft-backend, spikersoft-angular (PR #192), spikersoft-artpipe (PR #21) — every CI credential now resolves from OpenBao at runtime; the repos carry only `BAO_ROLE_ID`/`BAO_SECRET_ID`. The committed act_runner registration token was de-inlined to shell-env (infra #102) and **rotated** at its Gitea source. Related reliability work landed alongside: CI login-burst 504 fix (batch tokens + retry action + quota), and #590 (closed). Superseded private-repo action copy removed (#101). Residual hygiene (non-blocking): spikersoft-artpipe still has two orphaned plaintext Actions secrets (`DOCKER_PASSWORD`, `ACCESS_TOKEN_GITEA`) that no workflow references — pending deletion to match backend/angular. Closing Phase 1; deferred sub-work tracked separately (#605 two-role e2e). Phases 2 (service-config secrets) + 3 (PKI) continue under epic #543.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#545