Authenticated E2E walk: rpg-game, dungeon-crawler, duality-game fail with runtime TypeErrors (pre-existing on master) #726

Open
opened 2026-07-20 01:32:52 +00:00 by spikerj · 3 comments
Owner

Found by the E2E authenticated walk while gating the #722 E6 defer work (spikersoft-angular branch perf/a22-e6-defer-threejs-722). All three failures reproduce on a clean master build (verified 2026-07-19 by baseline run before the E6 changes), so they are pre-existing, not caused by the defer wrappers.

Symptom

pnpm e2e --grep "authenticated walk" — console-error assertions fail on:

  • /rpg-gameERROR TypeError: Cannot read properties of undefined (reading 'nativeElement') and ... (reading 'render')
  • /dungeon-crawlerError loading inventory: TypeError: Cannot read properties of undefined (reading 'playSound')
  • /duality-game — console errors (untouched by any E6 work; failing alongside the others)

The nativeElement signature on rpg-game looks like the E3 signal-query migration bug class (#722 E3 gotchas — viewChild setter side effects / guarded reads; cf. spikersoft-angular#439 for the art-studio equivalent).

Also observed in the same run (probably environmental, listing for completeness): 9 tool-playground tests fail with WebSocket connection to 'wss://api.spikersoft.com/codeExecutionHub' failed: HTTP Authentication failed, plus asset-approval, docker-info, reading-journey console errors. These may overlap with the e2e cred drift (#628) or whatever the current master test triage is covering.

Repro

pnpm build
pnpm e2e --grep "loads rpg-game|loads dungeon-crawler|loads duality-game"

Part of epic #307.

Found by the E2E authenticated walk while gating the #722 E6 defer work (spikersoft-angular branch `perf/a22-e6-defer-threejs-722`). **All three failures reproduce on a clean master build** (verified 2026-07-19 by baseline run before the E6 changes), so they are pre-existing, not caused by the defer wrappers. ## Symptom `pnpm e2e --grep "authenticated walk"` — console-error assertions fail on: - **/rpg-game** — `ERROR TypeError: Cannot read properties of undefined (reading 'nativeElement')` and `... (reading 'render')` - **/dungeon-crawler** — `Error loading inventory: TypeError: Cannot read properties of undefined (reading 'playSound')` - **/duality-game** — console errors (untouched by any E6 work; failing alongside the others) The `nativeElement` signature on rpg-game looks like the E3 signal-query migration bug class (#722 E3 gotchas — viewChild setter side effects / guarded reads; cf. spikersoft-angular#439 for the art-studio equivalent). Also observed in the same run (probably environmental, listing for completeness): 9 tool-playground tests fail with `WebSocket connection to 'wss://api.spikersoft.com/codeExecutionHub' failed: HTTP Authentication failed`, plus asset-approval, docker-info, reading-journey console errors. These may overlap with the e2e cred drift (#628) or whatever the current master test triage is covering. ## Repro ``` pnpm build pnpm e2e --grep "loads rpg-game|loads dungeon-crawler|loads duality-game" ``` Part of epic #307.
Author
Owner

Audited at file level against origin/masterNOT DONE, but the good news is the tests were not weakened. I specifically checked for the pin-rather-than-fix pattern that has turned up repeatedly in this codebase (#809–#812), and it is not present here.

The coverage is fully intact:

  • All three routes are still in the manifest with real coverage — e2e/support/route-manifest.ts:111 (/rpg-game), :121 (/dungeon-crawler), :122 (/duality-game), all access: "auth".
  • authenticated-walk.spec.ts:27 generates a test per manifest route, and :47 asserts expect(unexpectedConsoleErrors(consoleErrors)).toEqual([]).
  • No game-specific exemption exists. EXPECTED_CONSOLE_NOISE (e2e/support/fixtures.ts:28-41) contains exactly three entries: the /api/system/version 429 (the walk's own rate-limit, scoped to that endpoint), and two environment-only SSL/service-worker lines (NG05604). Nothing game-related, nothing broad.
  • Most importantly, unexpectedConsoleErrors at :49 never filters [pageerror] entries — the filter is entry.startsWith("[pageerror]") || !noise.test(entry). Runtime TypeErrors arrive as [pageerror] (:63) and are unconditionally treated as failures.
  • No .skip on any of the three. The only test.skip in the walk is the credentials gate at :23.

So nobody quietly silenced these. The specs would still fail today if run — the TypeErrors simply haven't been fixed.

The thing worth acting on, though: that credentials gate at :23 is test.skip(!hasUserCredentials(), NO_CREDENTIALS_REASON). Per #628 (e2e.staff credential drift, open since 2026-07-17 with zero comments), the nightly's staff password no longer authenticates — and e2e-nightly.yml:52-53 feeds exactly that password in. So the entire authenticated walk has been self-skipping, which means these three failures have not actually been observed for roughly twelve days.

That has two consequences for this ticket: its "pre-existing on master" claim is unverified against current code, and if someone did fix one of the three, nothing would have reported it. Worth re-running the walk with working credentials before investing in a fix — the failure list may have moved.

Remaining: fix #628 first so the walk actually executes, then triage the three TypeErrors against fresh output. I've noted on #628 separately that the suite should fail rather than skip when credentials are expected — a silent skip in CI is indistinguishable from a pass, which is precisely why this went unnoticed.

Audited at file level against `origin/master` — **NOT DONE, but the good news is the tests were not weakened.** I specifically checked for the pin-rather-than-fix pattern that has turned up repeatedly in this codebase (#809–#812), and it is **not** present here. **The coverage is fully intact:** - All three routes are still in the manifest with real coverage — `e2e/support/route-manifest.ts:111` (`/rpg-game`), `:121` (`/dungeon-crawler`), `:122` (`/duality-game`), all `access: "auth"`. - `authenticated-walk.spec.ts:27` generates a test per manifest route, and `:47` asserts `expect(unexpectedConsoleErrors(consoleErrors)).toEqual([])`. - **No game-specific exemption exists.** `EXPECTED_CONSOLE_NOISE` (`e2e/support/fixtures.ts:28-41`) contains exactly three entries: the `/api/system/version` 429 (the walk's own rate-limit, scoped to that endpoint), and two environment-only SSL/service-worker lines (`NG05604`). Nothing game-related, nothing broad. - Most importantly, `unexpectedConsoleErrors` at `:49` **never filters `[pageerror]` entries** — the filter is `entry.startsWith("[pageerror]") || !noise.test(entry)`. Runtime TypeErrors arrive as `[pageerror]` (`:63`) and are unconditionally treated as failures. - No `.skip` on any of the three. The only `test.skip` in the walk is the credentials gate at `:23`. So nobody quietly silenced these. The specs would still fail today if run — the TypeErrors simply haven't been fixed. **The thing worth acting on, though:** that credentials gate at `:23` is `test.skip(!hasUserCredentials(), NO_CREDENTIALS_REASON)`. Per **#628** (e2e.staff credential drift, open since 2026-07-17 with zero comments), the nightly's staff password no longer authenticates — and `e2e-nightly.yml:52-53` feeds exactly that password in. **So the entire authenticated walk has been self-skipping, which means these three failures have not actually been observed for roughly twelve days.** That has two consequences for this ticket: its "pre-existing on master" claim is unverified against current code, and if someone *did* fix one of the three, nothing would have reported it. Worth re-running the walk with working credentials before investing in a fix — the failure list may have moved. **Remaining:** fix #628 first so the walk actually executes, then triage the three TypeErrors against fresh output. I've noted on #628 separately that the suite should **fail** rather than skip when credentials are expected — a silent skip in CI is indistinguishable from a pass, which is precisely why this went unnoticed.
Author
Owner

Nightly triage 2026-08-01 → 08-03: this is the whole nightly, and it splits three ways

The nightly has never been green. Every event: schedule run of e2e-nightly.yml back to 2026-07-21 (oldest in the 20 pages of run history checked) has failed — 13 straight nights, 13–19 failures each, all in the [authenticated] project. Runs 19308 (08-01) and 19312 (08-03) have byte-identical failure lists; 19309 (08-02) is those same 14 plus a flaky /stream. The Discord webhook has fired every morning for ~2 weeks.

The failure set is the one filed here on 07-20. Below is the split, since this ticket's original text guesses the tool failures are "probably environmental" and possibly #628both are wrong.

#628 is not involved — now closed

The specs self-skip wholesale when E2E_USER_* is unset, and they are running. Every failing test also passes the not.toContain("ids.spikersoft.com") assertion, so the storage state is honored and the Keycloak login works. Verified directly: the e2e.staff password grant returns 200 with staff in realm_access.roles. #628 closed 2026-08-03.

Bucket 1 — the browser isn't attaching the token (10 of 14, every night)

8 tool playgrounds + /reading-journey. Probed prod with a valid e2e.staff token:

Endpoint with staff token anonymous
/api/Lessons/progress 200 401
/api/Profile/areas-of-interest 200 401
/codeExecutionHub/negotiate 200 401

CI's failure statuses match the anonymous column exactly. Backend healthy, endpoints exist, account has the roles. This is frontend token attachment — not authz, not a backend bug.

jwtInterceptor attaches Authorization only when keyCloak.authenticated is truthy at request time. Three candidate mechanisms, none yet confirmed:

  • (a) KEYCLOAK_INIT_TIMEOUT_MS = 8000 (keycloak-bounded-init.ts) — on timeout bootstrap proceeds unauthenticated by design (the #345 anti-blank-shell net). Against: the 401 branch calls keyCloak.login(), which should redirect, and no test bounced.
  • (b) 300s access-token lifetime vs. failures ~11 min into the run; the isTokenExpired() branch refreshes without redirecting, which fits the no-bounce observation.
  • (c) withAutoRefreshToken({ onInactivityTimeout: "logout", sessionTimeout: 300000 }) is live. UserActivityService counts only click/keydown/mousemove/scroll/touchstart — the walk performs none. A logout doesn't redirect to login either. Caveat: each test gets a fresh page so the timer restarts per load; the surviving variant is that one long-idle page tears down the server-side SSO session and poisons every worker after it, which would explain why failures cluster at the end of a 12.4-min run.

Discriminator: does the failing request carry an Authorization header at all, and if so has its exp passed. Needs a HAR/trace — a local run is in progress.

⚠️ Log silence is not evidence here. [keycloak] init did not settle and jwtInterceptor: 401 appear zero times in the CI logs, but both are console.warn and the consoleErrors fixture (e2e/support/fixtures.ts:56) collects only msg.type() === "error". Their absence is expected under all three hypotheses.

Bucket 2 — two endpoints that do not exist (2 of 14, every night) — a live prod defect

  • /docker-info → 404 on /api/DockerSwarm/services
  • /asset-approval → 404 on /api/Card

Not auth. Both return 404 with a valid staff token as well as anonymously, so it isn't authz-masking. Of 85 *Controller.cs in spikersoft-backend there is no DockerSwarmController and no CardController — only the DTOs (SpikerSoft.Data/DTOs/DockerSwarm*Dto.cs). Call sites: _services/asset/asset.service.ts:19,25 and _services/docker.service.spec.ts:25. Both services' unit tests pass because they mock HttpClient — false coverage over an endpoint that was never built.

These two screens are broken in production. Worth splitting into its own ticket; it is not an E2E problem.

Bucket 3 — this ticket's actual scope (3 of 14)

/rpg-game (nativeElement, then render), /dungeon-crawler (playSound, plus a staff-model fetch returning <!doctype HTML instead of JSON → missing static asset hitting the SPA fallback), /duality-game. These would fail with a perfect token.

Noise floor

/stream (3/7 nights), github-sweeper, snake-game, snake-scroller, clue-for-sql, commit-snake, intro — 1/7 each. Ignore until the persistent 13 are fixed.

Recommendation

The nightly is currently a smoke alarm nobody hears — two weeks of ignored pings is worse than no alarm. Bucket 2 is the cheapest real win and is a live prod bug; bucket 1 is one fix that clears 10 of 14 once the trace picks between (a)/(b)/(c).

## Nightly triage 2026-08-01 → 08-03: this is the whole nightly, and it splits three ways **The nightly has never been green.** Every `event: schedule` run of `e2e-nightly.yml` back to 2026-07-21 (oldest in the 20 pages of run history checked) has failed — 13 straight nights, 13–19 failures each, all in the `[authenticated]` project. Runs 19308 (08-01) and 19312 (08-03) have **byte-identical** failure lists; 19309 (08-02) is those same 14 plus a flaky `/stream`. The Discord webhook has fired every morning for ~2 weeks. The failure set is the one filed here on 07-20. Below is the split, since this ticket's original text guesses the tool failures are "probably environmental" and possibly #628 — **both are wrong.** ### #628 is not involved — now closed The specs self-skip wholesale when `E2E_USER_*` is unset, and they are running. Every failing test also passes the `not.toContain("ids.spikersoft.com")` assertion, so the storage state is honored and the Keycloak login works. Verified directly: the `e2e.staff` password grant returns 200 with `staff` in `realm_access.roles`. #628 closed 2026-08-03. ### Bucket 1 — the browser isn't attaching the token (10 of 14, every night) 8 tool playgrounds + `/reading-journey`. Probed prod with a valid `e2e.staff` token: | Endpoint | with staff token | anonymous | | --- | --- | --- | | `/api/Lessons/progress` | **200** | 401 | | `/api/Profile/areas-of-interest` | **200** | 401 | | `/codeExecutionHub/negotiate` | **200** | 401 | **CI's failure statuses match the anonymous column exactly.** Backend healthy, endpoints exist, account has the roles. This is frontend token attachment — not authz, not a backend bug. `jwtInterceptor` attaches `Authorization` only when `keyCloak.authenticated` is truthy at request time. Three candidate mechanisms, none yet confirmed: - **(a)** `KEYCLOAK_INIT_TIMEOUT_MS = 8000` (`keycloak-bounded-init.ts`) — on timeout bootstrap proceeds **unauthenticated by design** (the #345 anti-blank-shell net). *Against:* the 401 branch calls `keyCloak.login()`, which should redirect, and no test bounced. - **(b)** 300s access-token lifetime vs. failures ~11 min into the run; the `isTokenExpired()` branch refreshes without redirecting, which fits the no-bounce observation. - **(c)** `withAutoRefreshToken({ onInactivityTimeout: "logout", sessionTimeout: 300000 })` is live. `UserActivityService` counts only click/keydown/mousemove/scroll/touchstart — the walk performs none. A logout doesn't redirect to login either. *Caveat:* each test gets a fresh page so the timer restarts per load; the surviving variant is that one long-idle page tears down the **server-side** SSO session and poisons every worker after it, which would explain why failures cluster at the end of a 12.4-min run. Discriminator: does the failing request carry an `Authorization` header at all, and if so has its `exp` passed. Needs a HAR/trace — a local run is in progress. ⚠️ **Log silence is not evidence here.** `[keycloak] init did not settle` and `jwtInterceptor: 401` appear zero times in the CI logs, but both are `console.warn` and the `consoleErrors` fixture (`e2e/support/fixtures.ts:56`) collects only `msg.type() === "error"`. Their absence is expected under all three hypotheses. ### Bucket 2 — two endpoints that do not exist (2 of 14, every night) — **a live prod defect** - `/docker-info` → 404 on `/api/DockerSwarm/services` - `/asset-approval` → 404 on `/api/Card` Not auth. Both return **404 with a valid staff token as well as anonymously**, so it isn't authz-masking. Of 85 `*Controller.cs` in `spikersoft-backend` there is no `DockerSwarmController` and no `CardController` — only the DTOs (`SpikerSoft.Data/DTOs/DockerSwarm*Dto.cs`). Call sites: `_services/asset/asset.service.ts:19,25` and `_services/docker.service.spec.ts:25`. Both services' unit tests pass because they mock `HttpClient` — false coverage over an endpoint that was never built. **These two screens are broken in production.** Worth splitting into its own ticket; it is not an E2E problem. ### Bucket 3 — this ticket's actual scope (3 of 14) `/rpg-game` (`nativeElement`, then `render`), `/dungeon-crawler` (`playSound`, plus a staff-model fetch returning `<!doctype` HTML instead of JSON → missing static asset hitting the SPA fallback), `/duality-game`. These would fail with a perfect token. ### Noise floor `/stream` (3/7 nights), `github-sweeper`, `snake-game`, `snake-scroller`, `clue-for-sql`, `commit-snake`, `intro` — 1/7 each. Ignore until the persistent 13 are fixed. ### Recommendation The nightly is currently a smoke alarm nobody hears — two weeks of ignored pings is worse than no alarm. Bucket 2 is the cheapest real win and is a live prod bug; bucket 1 is one fix that clears 10 of 14 once the trace picks between (a)/(b)/(c).
Author
Owner

Bucket 1 root cause CONFIRMED — it is (a), the 8s bounded-init timeout

Ran the failing specs locally with --project=authenticated --trace on. Reproduces in 44.5 seconds, so nothing time-dependent is involved — that rules out (b) token expiry and (c) the inactivity logout outright. The trace captures the whole chain:

15:02:13.2   page loads; shell fetches ALL carry a token (exp …633) → 200
             ↓  7.7s
[warning] [keycloak] init did not settle within 8000ms; proceeding with app
          bootstrap (unauthenticated / deferred re-auth) … See #345.
[log]     Keycloak token available: false
[log]     Keycloak authenticated: false
[log]     Access token factory called, token length: 0

15:02:21.03  BURST WITH NO Authorization HEADER:
   200  ABSENT  /api/system/version
   200  ABSENT  /api/Lessons/reading?language=C%23
   401  ABSENT  /api/Lessons/progress        ← the failure
    -1  ABSENT  /codeExecutionHub            ← the WS failure
[warning] jwtInterceptor: 401 Unauthorized - redirecting to login

15:02:21.25  init settles; token length 1618; authenticated: true
   200  PRESENT /api/Lessons/progress  (exp …641)  ← identical request, now fine
   101  PRESENT /codeExecutionHub?access_token=…    ← WS connects

Causal chain

KEYCLOAK_INIT_TIMEOUT_MS = 8000 (_providers/keycloak-bounded-init.ts) elapses → the bounded initializer resolves and lets bootstrap proceed unauthenticated by design (the #345 anti-blank-shell net) → the routed feature's fetches fire while keyCloak.authenticated === false, so jwtInterceptor attaches no header → /api/Lessons/progress 401s → LessonCatalogService logs console.error → the walk's zero-console-errors assertion fails.

The app then self-heals ~200ms later when init() finally settles. That is why the shell paints, the toolbar shows, and nothing bounces to the IdP — everything the specs assert about rendering is fine. The header is ABSENT, not expired.

The load-bearing wrong assumption

From keycloak-bounded-init.ts's own docstring:

"Normal login (callback branch) and anonymous (silent iframe reports no session) settle well within the timeout."

The session-restore path (storage-state / hard refresh) does not settle within 8s — and that is precisely the Playwright case. It reproduced on a local machine against prod ids.spikersoft.com, so this is not CI-runner slowness.

Two things worth noting beyond the test

  1. jwtInterceptor did call keyCloak.login() on the 401 (see the warning in the trace). It happened to produce no navigation here, but that is a latent redirect hazard for real users on a slow session-restore, not just a test artifact.
  2. Any real user whose init() misses 8s gets the same ~200ms unauthenticated window — a spurious 401 against whatever the landing route fetches.

Fix directions (not implemented — needs a decision)

  • Raise the bound — one-line, but trades against the #345 blank-shell guarantee and doesn't remove the window, just narrows the odds.
  • Gate authenticated fetches on auth-readiness rather than letting them fire tokenless. The bounded init exists so the shell can paint; feature code arguably should not hit authenticated endpoints until auth resolves. Bigger change, actually fixes it.
  • Make jwtInterceptor await an in-flight init before deciding isLoggedIn. Narrowest real fix.

Happy to take whichever; flagging rather than picking since #345 constrains the first option.

Repro

pnpm run build
pnpm exec playwright test --project=authenticated \
  --grep "opens tool csharp-playground" --workers=1 --trace on

Then read 0-trace.network for Authorization headers — note the suite's own consoleErrors fixture (e2e/support/fixtures.ts:56) collects only msg.type() === "error", so both keycloak warnings above are invisible in normal runs and in CI logs.

## ✅ Bucket 1 root cause CONFIRMED — it is (a), the 8s bounded-init timeout Ran the failing specs locally with `--project=authenticated --trace on`. **Reproduces in 44.5 seconds**, so nothing time-dependent is involved — that rules out (b) token expiry and (c) the inactivity logout outright. The trace captures the whole chain: ``` 15:02:13.2 page loads; shell fetches ALL carry a token (exp …633) → 200 ↓ 7.7s [warning] [keycloak] init did not settle within 8000ms; proceeding with app bootstrap (unauthenticated / deferred re-auth) … See #345. [log] Keycloak token available: false [log] Keycloak authenticated: false [log] Access token factory called, token length: 0 15:02:21.03 BURST WITH NO Authorization HEADER: 200 ABSENT /api/system/version 200 ABSENT /api/Lessons/reading?language=C%23 401 ABSENT /api/Lessons/progress ← the failure -1 ABSENT /codeExecutionHub ← the WS failure [warning] jwtInterceptor: 401 Unauthorized - redirecting to login 15:02:21.25 init settles; token length 1618; authenticated: true 200 PRESENT /api/Lessons/progress (exp …641) ← identical request, now fine 101 PRESENT /codeExecutionHub?access_token=… ← WS connects ``` ### Causal chain `KEYCLOAK_INIT_TIMEOUT_MS = 8000` (`_providers/keycloak-bounded-init.ts`) elapses → the bounded initializer resolves and lets bootstrap proceed **unauthenticated by design** (the #345 anti-blank-shell net) → the routed feature's fetches fire while `keyCloak.authenticated === false`, so `jwtInterceptor` attaches no header → `/api/Lessons/progress` 401s → `LessonCatalogService` logs `console.error` → the walk's zero-console-errors assertion fails. **The app then self-heals ~200ms later** when `init()` finally settles. That is why the shell paints, the toolbar shows, and nothing bounces to the IdP — everything the specs assert about rendering is fine. The header is **ABSENT, not expired**. ### The load-bearing wrong assumption From `keycloak-bounded-init.ts`'s own docstring: > *"Normal login (callback branch) and anonymous (silent iframe reports no session) settle well within the timeout."* The **session-restore** path (storage-state / hard refresh) does **not** settle within 8s — and that is precisely the Playwright case. It reproduced on a local machine against prod `ids.spikersoft.com`, so this is not CI-runner slowness. ### Two things worth noting beyond the test 1. `jwtInterceptor` **did** call `keyCloak.login()` on the 401 (see the warning in the trace). It happened to produce no navigation here, but that is a latent redirect hazard for real users on a slow session-restore, not just a test artifact. 2. Any real user whose `init()` misses 8s gets the same ~200ms unauthenticated window — a spurious 401 against whatever the landing route fetches. ### Fix directions (not implemented — needs a decision) - **Raise the bound** — one-line, but trades against the #345 blank-shell guarantee and doesn't remove the window, just narrows the odds. - **Gate authenticated fetches on auth-readiness** rather than letting them fire tokenless. The bounded init exists so the *shell* can paint; feature code arguably should not hit authenticated endpoints until auth resolves. Bigger change, actually fixes it. - **Make `jwtInterceptor` await an in-flight init** before deciding `isLoggedIn`. Narrowest real fix. Happy to take whichever; flagging rather than picking since #345 constrains the first option. ### Repro ``` pnpm run build pnpm exec playwright test --project=authenticated \ --grep "opens tool csharp-playground" --workers=1 --trace on ``` Then read `0-trace.network` for `Authorization` headers — note the suite's own `consoleErrors` fixture (`e2e/support/fixtures.ts:56`) collects only `msg.type() === "error"`, so both keycloak warnings above are invisible in normal runs and in CI logs.
Sign in to join this conversation.