[infra] Bump Keycloak 26.6 → 26.7 (keycloak/docker-stack.yml) #467

New Issue

2026-07-11T02:13:45Z

spikerj commented

2026-07-11 02:13:45 +00:00

Goal

Bump the Keycloak image in spikersoft-infrastructure/keycloak/docker-stack.yml from quay.io/keycloak/keycloak:26.6 to :26.7.

26.7.0 compatibility review (no backend code changes required)

Reviewed the 26.7.0 release notes and upgrading guide migration section against our integration:

Our usage is standard OIDC: JWT bearer validation via .well-known/openid-configuration + realm_access.roles → claims (SpikerSoft.Api/Infrastructure/Authentication.cs), and Admin REST calls with grant_type=client_credentials (KeycloakAdminService, health check, MCP auth). User provisioning goes through the Admin API (POST /admin/realms/{realm}/users with a password credential), not Keycloak's self-registration screen.
Breaking changes reviewed — none apply: view-system role removed (unused), Identity Provider alias now immutable (we don't manage IdPs via REST), X509 client auth CA subject DN (not used), service accounts excluded from user-by-ID queries (we only fetch real user UUIDs), Organizations brief representation / invitation exact-match filters (Organizations unused), Dynamic→Parameterized Scopes rename (not used).
Notable behavior changes reviewed — not impactful: "Verify email required before credentials setup during self-registration" doesn't affect us (we create users + set password via Admin API); session cookie SHA-384 and shutdown-timeout extension are UI/runtime-only; DPoP/implicit + token-exchange sender-constraint changes — we use neither.
Removed: experimental token-exchange-external-internal:v2 — not enabled here.

Residual risk to verify post-deploy

The custom event-listener SPI provider mounted at /opt/keycloak/providers (the KK_TO_RMQ_* keycloak→RabbitMQ listener). Event-listener SPI is stable across this minor bump, but confirm Keycloak loads the provider cleanly on startup (no Failed to load provider / ProviderNotFoundException in logs) after deploy.

Rollout

One-node-at-a-time (single replica here) — Postgres schema auto-migrates on start; DB backup job already runs.
Rollback: revert the image tag to :26.6 and redeploy.

## Goal Bump the Keycloak image in `spikersoft-infrastructure/keycloak/docker-stack.yml` from `quay.io/keycloak/keycloak:26.6` to `:26.7`. ## 26.7.0 compatibility review (no backend code changes required) Reviewed the [26.7.0 release notes](https://www.keycloak.org/docs/latest/release_notes/index.html) and [upgrading guide](https://www.keycloak.org/docs/latest/upgrading/) migration section against our integration: - **Our usage** is standard OIDC: JWT bearer validation via `.well-known/openid-configuration` + `realm_access.roles` → claims (`SpikerSoft.Api/Infrastructure/Authentication.cs`), and Admin REST calls with `grant_type=client_credentials` (`KeycloakAdminService`, health check, MCP auth). User provisioning goes through the Admin API (`POST /admin/realms/{realm}/users` with a password credential), not Keycloak's self-registration screen. - **Breaking changes reviewed — none apply:** `view-system` role removed (unused), Identity Provider alias now immutable (we don't manage IdPs via REST), X509 client auth CA subject DN (not used), service accounts excluded from user-by-ID queries (we only fetch real user UUIDs), Organizations brief representation / invitation exact-match filters (Organizations unused), Dynamic→Parameterized Scopes rename (not used). - **Notable behavior changes reviewed — not impactful:** "Verify email required before credentials setup during self-registration" doesn't affect us (we create users + set password via Admin API); session cookie SHA-384 and shutdown-timeout extension are UI/runtime-only; DPoP/implicit + token-exchange sender-constraint changes — we use neither. - **Removed:** experimental `token-exchange-external-internal:v2` — not enabled here. ## Residual risk to verify post-deploy The custom event-listener SPI provider mounted at `/opt/keycloak/providers` (the `KK_TO_RMQ_*` keycloak→RabbitMQ listener). Event-listener SPI is stable across this minor bump, but confirm Keycloak loads the provider cleanly on startup (no `Failed to load provider` / `ProviderNotFoundException` in logs) after deploy. ## Rollout - One-node-at-a-time (single replica here) — Postgres schema auto-migrates on start; DB backup job already runs. - Rollback: revert the image tag to `:26.6` and redeploy.

spikerj commented

2026-07-11 02:21:55 +00:00

Re-verification: Traefik reverse-proxy contract vs the 26.7 guide

Re-checked keycloak/docker-stack.yml against Configuring a reverse proxy (the guide that gained the Traefik blueprints in 26.7). The bump remains safe; no backend code changes. Findings:

Confirmed correct (unchanged)

KC_PROXY_HEADERS=xforwarded + KC_HTTP_ENABLED=true is exactly the documented contract for edge termination at Traefik.
Traefik overwrites X-Forwarded-* from untrusted clients by default (we don't widen forwardedHeaders.trustedIPs), satisfying the "proxy must overwrite, not append" requirement.
Only port 8080 is proxied; the 9000 management port (health/metrics) is not exposed through Traefik, per the guide.
Sticky cookie + single replica + KC_CACHE=local — affinity guidance is moot but harmless.

Dead/duplicate config removed on the PR branch (`bfd5d7e`) — all inert, zero behavior change

KC_PROXY=edge and KC_PROXY_ADDRESS_FORWARDING — the proxy option was removed from Keycloak (26.x) and the latter is WildFly-era; both were already silently ignored on 26.6.
Legacy KEYCLOAK_ADMIN/KEYCLOAK_ADMIN_PASSWORD — superseded by the KC_BOOTSTRAP_ADMIN_* pair also present.
Duplicate KC_HOSTNAME YAML key (last-wins made it a no-op).
keycloak-redirect middleware + its router.middlewares label — the label was declared twice and the later compresstraefik declaration always won, so the redirect was never applied (router only listens on websecure anyway). Effective Traefik config unchanged.
Duplicate PGPASSWORD key in backups-keycloak — strict YAML parsers reject the file (docker compose config failed on it; passes now).

Follow-up hardening candidates from the guide (not in this PR)

Set KC_PROXY_TRUSTED_ADDRESSES so forwarded headers are only honored from Traefik (weak but cheap defense; needs a stable proxy CIDR on the overlay).
Don't expose /admin/ and /realms/master/ publicly — guide recommends internal-only; today the Traefik router forwards all paths for ids.spikersoft.com.
KEYCLOAK_LOGLEVEL=DEBUG in production — noisy; drop to INFO.
Plaintext secrets in the stack file — already tracked in #404.

## Re-verification: Traefik reverse-proxy contract vs the 26.7 guide Re-checked `keycloak/docker-stack.yml` against [Configuring a reverse proxy](https://www.keycloak.org/server/reverseproxy) (the guide that gained the Traefik blueprints in 26.7). **The bump remains safe; no backend code changes.** Findings: ### Confirmed correct (unchanged) - `KC_PROXY_HEADERS=xforwarded` + `KC_HTTP_ENABLED=true` is exactly the documented contract for edge termination at Traefik. - Traefik overwrites `X-Forwarded-*` from untrusted clients by default (we don't widen `forwardedHeaders.trustedIPs`), satisfying the "proxy must overwrite, not append" requirement. - Only port 8080 is proxied; the 9000 management port (health/metrics) is not exposed through Traefik, per the guide. - Sticky cookie + single replica + `KC_CACHE=local` — affinity guidance is moot but harmless. ### Dead/duplicate config removed on the PR branch (`bfd5d7e`) — all inert, zero behavior change - `KC_PROXY=edge` and `KC_PROXY_ADDRESS_FORWARDING` — the `proxy` option was removed from Keycloak (26.x) and the latter is WildFly-era; both were already silently ignored on 26.6. - Legacy `KEYCLOAK_ADMIN`/`KEYCLOAK_ADMIN_PASSWORD` — superseded by the `KC_BOOTSTRAP_ADMIN_*` pair also present. - Duplicate `KC_HOSTNAME` YAML key (last-wins made it a no-op). - `keycloak-redirect` middleware + its `router.middlewares` label — the label was declared **twice** and the later `compresstraefik` declaration always won, so the redirect was never applied (router only listens on `websecure` anyway). Effective Traefik config unchanged. - Duplicate `PGPASSWORD` key in `backups-keycloak` — strict YAML parsers reject the file (`docker compose config` failed on it; passes now). ### Follow-up hardening candidates from the guide (not in this PR) 1. Set `KC_PROXY_TRUSTED_ADDRESSES` so forwarded headers are only honored from Traefik (weak but cheap defense; needs a stable proxy CIDR on the overlay). 2. Don't expose `/admin/` and `/realms/master/` publicly — guide recommends internal-only; today the Traefik router forwards all paths for `ids.spikersoft.com`. 3. `KEYCLOAK_LOGLEVEL=DEBUG` in production — noisy; drop to INFO. 4. Plaintext secrets in the stack file — already tracked in #404.

spikerj commented

2026-07-11 23:32:26 +00:00

Resolved on spikersoft-infrastructure master: commit 4113331 bumps keycloak/docker-stack.yml to quay.io/keycloak/keycloak:26.7, and bfd5d7e reconciles the stack with the 26.7 reverse-proxy guide (PR #16 was closed as superseded by these direct commits — its compatibility review still applies: no breaking changes touch our OIDC/Admin-API usage). Ops reminder for the next deploy window: sync /mnt/infrastructure and redeploy the keycloak stack, then confirm the KK_TO_RMQ event-listener provider loads cleanly on startup. Closing.

spikerj closed this issue

2026-07-11 23:32:26 +00:00

spikerj referenced this issue

2026-07-24 00:57:51 +00:00

GPU coordinator queue deadlocked after #467 node-routing deploy — Embeddings pinned to disabled SERVER lane wedges all GPU tenants #822

spikerj referenced this issue

2026-07-29 15:47:29 +00:00

GPU coordinator queue deadlocked after #467 node-routing deploy — Embeddings pinned to disabled SERVER lane wedges all GPU tenants #822

spikerj referenced this issue

2026-07-29 15:59:41 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting #582

spikerj referenced this issue