[Security][Infra] OpenBao: centralize public Let's Encrypt cert storage/distribution in Bao (Traefik acme.json → KV; mailserver + consumers pull) (epic #543) #610

New Issue

2026-07-16T05:26:42Z

spikerj commented

2026-07-16 05:26:42 +00:00

Part of the OpenBao epic #543. This is the public-cert half that Phase 3 (#547) explicitly does not cover, and it is currently untracked.

Problem

Public-facing TLS certificates (Let's Encrypt via Traefik resolver myresolver) are not centralized — they live as loose host files with a fragile, undocumented sync:

Traefik acquires + renews them via ACME (tls + http challenge) and writes them to a local host file: /mnt/traefik/data/acme.json → /le/acme.json (traefik/docker-stack.yml).
Mailserver (mailserver2/mailserver) reads a separate host dir /mnt/mailserver/acme → /etc/letsencrypt/acme and decodes acme.json to extract the mail.spikersoft.com cert+key on container start (mailserver/docker-stack.yml:72).
The step that copies Traefik's acme.json into the mailserver's acme dir is a host-level sync not represented in any stack file — undocumented, and it forces every cert consumer to bind-mount the same host path and co-locate on the node.

Scope clarification (important)

OpenBao's PKI engine (#547) is an internal CA for internal mTLS — it cannot issue publicly-trusted certs, so Traefik + mail must stay Let's Encrypt. This ticket is therefore about centralizing the storage + distribution of the LE certs in Bao, not their issuance. LE remains the issuer; Bao becomes the single source of truth for the resulting cert material and every consumer pulls from Bao instead of a shared host file.

Proposed design

Publish — a small sync (Traefik post-renewal hook, or a scheduled one-shot swarm job pinned to the Traefik node) parses /mnt/traefik/data/acme.json and writes per-domain material to Bao KV, e.g. secret/certs/public/<domain> = { fullchain, privkey } (optionally the raw acme.json under secret/certs/public/acme for consumers that decode themselves). Values are write-only; never logged.
Policy + AppRole — a least-privilege certs-public-read policy (read secret/data/certs/public/* only), delivered as a scoped AppRole via Docker swarm secrets bao_certs_role_id / bao_certs_secret_id, following the svc-codeexecution least-privilege pattern from #546.
Consume (mailserver first) — replace the host acme bind-mount with an init step that authenticates to Bao (mounted role/secret id), pulls the mail.spikersoft.com cert+key, writes them where the image expects, then hands off to the normal entrypoint. Dual-run per house pattern: if Bao is unreachable, fall back to the existing host acme dir so mail TLS never hard-breaks.
Renewal propagation — on Traefik renewal → KV updated → consumers refresh (restart or lightweight watch). Document renewal→propagation latency and the fallback.

Non-goals

Issuing public certs from Bao PKI (not possible — an internal CA isn't publicly trusted). LE stays the issuer.
Internal mTLS / internal CA — that is #547.

Acceptance criteria

Traefik's LE certs are mirrored into Bao KV automatically on issue/renew (verify a renewal propagates end-to-end).
The mailserver obtains its mail.spikersoft.com cert+key from Bao on start, with no dependency on the manual host-file copy, and a documented non-strict fallback to the host acme dir.
Least-privilege AppRole/policy scoped to secret/certs/public/* only (not the broad services role).
Runbook: how the sync works, how to rotate the AppRole, the fallback behavior, renewal cadence.

Epic #543; sibling to Phase 3 #547 (internal PKI/mTLS).
Least-privilege scoped-AppRole pattern established in #546 (svc-codeexecution).
Separate but adjacent gap: the mailserver stack still commits plaintext Casey51423! (DBPASS + RSPAMD_PASSWORD, mailserver/docker-stack.yml:38-39). The mailserver is not a .NET app, so the SpikerSoft.Secrets overlay from #546 can't reach it — it needs Docker swarm secrets or Bao-at-deploy injection. Track under #546's plaintext retirement or a dedicated ticket.

Part of the OpenBao epic #543. This is the **public-cert** half that Phase 3 (#547) explicitly does *not* cover, and it is currently untracked. ## Problem Public-facing TLS certificates (Let's Encrypt via Traefik resolver `myresolver`) are **not centralized** — they live as loose host files with a fragile, undocumented sync: - **Traefik** acquires + renews them via ACME (tls + http challenge) and writes them to a **local host file**: `/mnt/traefik/data/acme.json` → `/le/acme.json` (`traefik/docker-stack.yml`). - **Mailserver** (`mailserver2/mailserver`) reads a **separate** host dir `/mnt/mailserver/acme` → `/etc/letsencrypt/acme` and decodes `acme.json` to extract the `mail.spikersoft.com` cert+key **on container start** (`mailserver/docker-stack.yml:72`). - The step that copies Traefik's `acme.json` into the mailserver's acme dir is a **host-level sync not represented in any stack file** — undocumented, and it forces every cert consumer to bind-mount the same host path and co-locate on the node. ## Scope clarification (important) OpenBao's PKI engine (#547) is an **internal CA for internal mTLS** — it **cannot** issue publicly-trusted certs, so Traefik + mail **must stay Let's Encrypt**. This ticket is therefore about centralizing the **storage + distribution** of the LE certs in Bao, **not** their issuance. LE remains the issuer; Bao becomes the single source of truth for the resulting cert material and every consumer pulls from Bao instead of a shared host file. ## Proposed design 1. **Publish** — a small sync (Traefik post-renewal hook, or a scheduled one-shot swarm job pinned to the Traefik node) parses `/mnt/traefik/data/acme.json` and writes per-domain material to Bao KV, e.g. `secret/certs/public/<domain>` = `{ fullchain, privkey }` (optionally the raw `acme.json` under `secret/certs/public/acme` for consumers that decode themselves). Values are write-only; never logged. 2. **Policy + AppRole** — a least-privilege `certs-public-read` policy (`read secret/data/certs/public/*` only), delivered as a **scoped** AppRole via Docker swarm secrets `bao_certs_role_id` / `bao_certs_secret_id`, following the svc-codeexecution least-privilege pattern from #546. 3. **Consume (mailserver first)** — replace the host acme bind-mount with an init step that authenticates to Bao (mounted role/secret id), pulls the `mail.spikersoft.com` cert+key, writes them where the image expects, then hands off to the normal entrypoint. **Dual-run per house pattern**: if Bao is unreachable, fall back to the existing host acme dir so mail TLS never hard-breaks. 4. **Renewal propagation** — on Traefik renewal → KV updated → consumers refresh (restart or lightweight watch). Document renewal→propagation latency and the fallback. ## Non-goals - Issuing public certs from Bao PKI (not possible — an internal CA isn't publicly trusted). LE stays the issuer. - Internal mTLS / internal CA — that is #547. ## Acceptance criteria - [ ] Traefik's LE certs are mirrored into Bao KV automatically on issue/renew (verify a renewal propagates end-to-end). - [ ] The mailserver obtains its `mail.spikersoft.com` cert+key from Bao on start, with **no** dependency on the manual host-file copy, and a documented non-strict fallback to the host acme dir. - [ ] Least-privilege AppRole/policy scoped to `secret/certs/public/*` only (not the broad `services` role). - [ ] Runbook: how the sync works, how to rotate the AppRole, the fallback behavior, renewal cadence. ## Related - Epic #543; sibling to Phase 3 #547 (internal PKI/mTLS). - Least-privilege scoped-AppRole pattern established in #546 (svc-codeexecution). - **Separate but adjacent gap:** the mailserver stack still commits plaintext `Casey51423!` (`DBPASS` + `RSPAMD_PASSWORD`, `mailserver/docker-stack.yml:38-39`). The mailserver is not a .NET app, so the SpikerSoft.Secrets overlay from #546 can't reach it — it needs Docker swarm secrets or Bao-at-deploy injection. Track under #546's plaintext retirement or a dedicated ticket.

spikerj referenced this issue

2026-07-29 15:57:12 +00:00

[Go-Live Checklist][Security] Rotate + Bao-migrate the dev-environment secrets before production #633

spikerj commented

2026-07-29 15:59:26 +00:00

Audited against origin/master — NOT DONE. Zero implementing change; all four acceptance criteria unstarted. Recording the greps so this doesn't need re-deriving:

No KV path, policy or AppRole: git grep -rn "certs/public\|certs-public-read\|bao_certs_role_id" origin/master → no matches.
No acme.json → Bao sync hook or one-shot job: git ls-tree -r --name-only origin/master | grep -i "acme\|cert" returns only redis-cluster/tls/gen-test-certs.sh, which is unrelated internal test certs.
Consumer unchanged: mailserver/docker-stack.yml:72 still binds /mnt/mailserver/acme:/etc/letsencrypt/acme — the undocumented host-file copy this ticket exists to replace is still the live mechanism.
No runbook, no referencing PRs, and the ticket has zero comments since filing.

On the adjacent gap this ticket's body flags — the plaintext credentials in the same file — I checked and it's worse than "still there". mailserver/docker-stack.yml commits five literal credentials: :38 DBPASS, :39 RSPAMD_PASSWORD, :108 DBPASS, :177 MYSQL_RANDOM_ROOT_PASSWORD, :180 MYSQL_PASSWORD. I verified each is a genuine inline literal rather than ${VAR:?} env indirection, and SHA-256 fingerprinting shows :108 and :180 hold the same value (so rotating needs both touched or the stack breaks).

I've filed that as a sixth item on #633 (the Go-Live secrets ticket), since it's rotation work rather than cert-distribution work. Flagging the linkage here because whoever picks up this ticket will be editing that exact file — worth doing both in one pass rather than touching mailserver/docker-stack.yml twice.

No work has landed; notes were accurate.

Audited against `origin/master` — **NOT DONE. Zero implementing change; all four acceptance criteria unstarted.** Recording the greps so this doesn't need re-deriving: - **No KV path, policy or AppRole:** `git grep -rn "certs/public\|certs-public-read\|bao_certs_role_id" origin/master` → **no matches**. - **No acme.json → Bao sync hook or one-shot job:** `git ls-tree -r --name-only origin/master | grep -i "acme\|cert"` returns only `redis-cluster/tls/gen-test-certs.sh`, which is unrelated internal test certs. - **Consumer unchanged:** `mailserver/docker-stack.yml:72` still binds `/mnt/mailserver/acme:/etc/letsencrypt/acme` — the undocumented host-file copy this ticket exists to replace is still the live mechanism. - No runbook, no referencing PRs, and the ticket has zero comments since filing. **On the adjacent gap this ticket's body flags** — the plaintext credentials in the same file — I checked and it's worse than "still there". `mailserver/docker-stack.yml` commits **five** literal credentials: `:38 DBPASS`, `:39 RSPAMD_PASSWORD`, `:108 DBPASS`, `:177 MYSQL_RANDOM_ROOT_PASSWORD`, `:180 MYSQL_PASSWORD`. I verified each is a genuine inline literal rather than `${VAR:?}` env indirection, and SHA-256 fingerprinting shows `:108` and `:180` hold the same value (so rotating needs both touched or the stack breaks). I've filed that as a sixth item on **#633** (the Go-Live secrets ticket), since it's rotation work rather than cert-distribution work. Flagging the linkage here because whoever picks up this ticket will be editing that exact file — worth doing both in one pass rather than touching `mailserver/docker-stack.yml` twice. No work has landed; notes were accurate.

spikerj referenced this issue

2026-07-29 16:02:56 +00:00

[Critical][Ops][Notifications] Ops alerting is silently dead — every system-remediation-alert email fails SMTP AUTH 535; critical disk/OOM alerts never delivered #756

spikerj referenced this issue

2026-07-29 18:22:34 +00:00

[Epic][Security][Infra] Central secrets management — OpenBao vault + PKI, retire scattered passwords #543

spikerj referenced this issue

2026-07-29 18:22:48 +00:00

[Security][Infra] OpenBao Phase 3: PKI — internal CA, replace CI cert generation, mTLS for the stateful tier (epic #543) #547

Sign in to join this conversation.