spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Epic] Full-coverage Playwright E2E walk of the Angular app with live Jaeger + Seq telemetry #307

New Issue
Open
opened 2026-07-01 19:14:35 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-01 19:14:35 +00:00
Owner
Copy Link
Copy Source

Vision

Drive a Playwright suite that walks the entire Angular component/route surface and performs every meaningful user action, starting exactly as an anonymous visitor landing on the site and expanding into authenticated, parent, and staff/admin journeys — toward 100% E2E coverage of the app's routes and interactive flows.

While the suite runs, programmatically pull telemetry from Jaeger (traces) and Seq (logs) so that every discovered failure comes with backend visibility (which service, which span, which log line), turning the E2E run into a full-system observability harness rather than just a UI checker.

Work proceeds from master in spikersoft-angular (E2E lives there today) with a small supporting change in spikersoft-backend for verification test-hooks (see §4). Fixes discovered along the way are filed as their own tickets and linked back here.

Current base (what already exists)

  • spikersoft-angular/playwright.config.ts — serves the prebuilt dist/spikersoft/browser on http://127.0.0.1:4200 via serve; testDir: ./e2e/playwright.
  • e2e/playwright/example.spec.ts (app-loads smoke) and video-call.spec.ts (two-context fake-media smoke).
  • CI job e2e-smoke in .gitea/workflows/main.yml (prebuilds prod bundle, installs chromium, runs the video call grep; continue-on-error: true).
  • Gap: no data-testid/data-e2e selector convention across the app (only the diagram dev-tool lib uses data-testid). Robust selectors are a prerequisite for stable full coverage.

Route surface to cover (from routing audit of projects/spikersoft/src/routes.ts)

  • Public/anonymous: /, /intro, /mission, /our-hardware, /legacy-systems, /develop, /contact, /syllabus, /team, /employment, /links(/:id), /blog, /sponsor (+ donate/profile), /tree-of-knowledge, /geography/*, /brain-forge, /cortex-quest, /wisdom-well, /chess, /ai, /classes, /cpu-dashboard, public coding topics, and the /tools shell + ~25 named-outlet dev tools, /parental-approve/:token, /organization/confirm, /** (404).
  • Auth-gated (AuthGuard → Keycloak): /home, /services, /reading-journey, /reader/:bookId, /info-vault, /fundraisers, /blog/create, most /*-game routes, /calendar, /video-call, /chat, /profile, /tracker, Marks (/mark/*).
  • Special guards: ReadingInterestGuard (/reading-journey), ParentGuard (/parent-dashboard), RoleGuard Admin/Staff (/admin/*, /keycloak-admin), unsavedChangesGuard (/profile).

Coverage is tracked against a committed route manifest so "100%" is measurable (every route visited + its primary actions exercised).

The hard part: signup verification (email code + Twilio SMS)

Registration is a 4-step stepper (registration-stepper.component) ending in email + phone verification:

  • POST /api/authentication/initiate-pre-registration → creates a Mongo PreRegistration with emailVerificationCode (6 digits) + emailVerificationToken.
  • POST /api/authentication/verify-pre-registration-email (code or token) → triggers Twilio Verify SMS.
  • POST /api/authentication/verify-pre-registration-phone → checks the code with Twilio Verify (code is NOT stored locally).
  • POST /api/authentication/complete-pre-registration → creates the Keycloak user once fully verified.

Email is automatable today (read emailVerificationCode/emailVerificationToken from Mongo PreRegistrations, or hit the GET verify link). SMS is not — no bypass exists.

Proposed test hook (needs confirmation, backend change): in non-prod only, thread the existing SendVerificationCodeCommand.CustomCode through pre-registration behind an env flag (e.g. PreRegistration:E2E:Enabled + E2E_SMS_CODE), OR provide an E2E-profile fake ITwilioVerifyService that accepts a fixed code, OR a test-only endpoint that marks PhoneVerified for a known pre-registration id. Must be impossible to enable in Production. Related existing flags: Authentication:UseTestAuthentication + TestAuthenticationHandler (API auth bypass for authenticated tests), PreRegistration:Bypass* (cooldowns/abuse only — not verification).

For authenticated journeys generally, obtain a Keycloak token via direct grant against a dedicated non-prod test realm/user (or reuse UseTestAuthentication for API-level setup), and use Playwright storage-state to skip the interactive login on most specs.

Telemetry integration (Jaeger + Seq)

Build a reusable e2e/telemetry client used by specs and fixtures:

  • Jaeger (query API on the internal UI, http://<node-ip>:16686): GET /api/services, GET /api/traces?service=SpikerSoft%20API&start&end&tags&lookback, GET /api/traces/:id. Correlate by test time-window + service + tags (and, once the SPA emits W3C traceparent, by trace id).
  • Seq (https://seq.spikersoft.com): GET /api/events?filter=<seq-query>&count=N with X-Seq-ApiKey. Filter by ServiceName, Environment, level, and the test's correlation id.
  • Endpoints/keys via env (JAEGER_QUERY_URL, SEQ_URL, SEQ_API_KEY). A Playwright fixture snapshots "errors in Seq / error spans in Jaeger during this test" and attaches them to the report on failure.
  • (Stretch) Have the SPA propagate a per-session correlation header so browser actions map 1:1 to backend traces.

Phased plan (each phase = its own linked sub-issue)

  • P0 — Harness & telemetry: page-object base, route manifest + coverage reporter, JaegerClient/SeqClient, failure-time telemetry fixture, CI wiring (promote e2e-smoke beyond continue-on-error for the anonymous suite). Decide target environment (dedicated staging vs prod-read-only).
  • P1 — Anonymous walk: every public route loads, primary nav works, no console/JS errors, key CTAs, /tools + every dev-tool, 404. (No backend mutations.)
  • P2 — Selector hardening: introduce a data-testid convention across the components the walk touches.
  • P3 — Registration & verification: land the verification test-hook (§4), then full signup happy-path + validation/error paths; create a reusable "fresh verified user" fixture.
  • P4 — Authenticated core: login/logout/profile, reading journey, book reader, calendar, chat, AI, lessons + code playgrounds, games.
  • P5 — Parent & family: parental approval, parent dashboard, child-account flows.
  • P6 — Staff/Admin: /admin/* review surfaces + /keycloak-admin named-outlet UI (role-scoped test users).
  • P7 — Realtime/media: video-call two-party, gameserver, streaming (build on existing fake-media scaffold).
  • P8 — Coverage gate: route-manifest coverage report in CI; ratchet toward 100%.

Definition of done

  • Every route in the manifest is visited and its primary actions exercised by a passing spec.
  • Signup (incl. email + SMS verification) is fully automated via a safe, non-prod-only hook.
  • Each spec has a telemetry fixture that surfaces correlated Jaeger spans + Seq logs on failure.
  • CI runs the suite (anonymous suite blocking; authenticated suites at least nightly) with a coverage report.

Open decisions (need a call before/at P0)

  1. Target environment for E2E: stand up a dedicated staging stack, or run read-mostly against prod with disposable test accounts? (affects data-safety of P3+.)
  2. Verification hook shape (CustomCode+flag vs fake service vs test endpoint) — security review required so it can never arm in Production.
  3. Whether to invest in SPA → W3C traceparent propagation now (best trace correlation) or defer.
## Vision Drive a Playwright suite that **walks the entire Angular component/route surface and performs every meaningful user action**, starting exactly as an anonymous visitor landing on the site and expanding into authenticated, parent, and staff/admin journeys — toward **100% E2E coverage of the app's routes and interactive flows**. While the suite runs, **programmatically pull telemetry from Jaeger (traces) and Seq (logs)** so that every discovered failure comes with backend visibility (which service, which span, which log line), turning the E2E run into a full-system observability harness rather than just a UI checker. > Work proceeds from `master` in `spikersoft-angular` (E2E lives there today) with a small supporting change in `spikersoft-backend` for verification test-hooks (see §4). Fixes discovered along the way are filed as their own tickets and linked back here. --- ## Current base (what already exists) - `spikersoft-angular/playwright.config.ts` — serves the prebuilt `dist/spikersoft/browser` on `http://127.0.0.1:4200` via `serve`; `testDir: ./e2e/playwright`. - `e2e/playwright/example.spec.ts` (app-loads smoke) and `video-call.spec.ts` (two-context fake-media smoke). - CI job `e2e-smoke` in `.gitea/workflows/main.yml` (prebuilds prod bundle, installs chromium, runs the `video call` grep; `continue-on-error: true`). - **Gap:** no `data-testid`/`data-e2e` selector convention across the app (only the diagram dev-tool lib uses `data-testid`). Robust selectors are a prerequisite for stable full coverage. --- ## Route surface to cover (from routing audit of `projects/spikersoft/src/routes.ts`) - **Public/anonymous:** `/`, `/intro`, `/mission`, `/our-hardware`, `/legacy-systems`, `/develop`, `/contact`, `/syllabus`, `/team`, `/employment`, `/links(/:id)`, `/blog`, `/sponsor` (+ donate/profile), `/tree-of-knowledge`, `/geography/*`, `/brain-forge`, `/cortex-quest`, `/wisdom-well`, `/chess`, `/ai`, `/classes`, `/cpu-dashboard`, public coding topics, and the `/tools` shell + ~25 named-outlet dev tools, `/parental-approve/:token`, `/organization/confirm`, `/**` (404). - **Auth-gated (AuthGuard → Keycloak):** `/home`, `/services`, `/reading-journey`, `/reader/:bookId`, `/info-vault`, `/fundraisers`, `/blog/create`, most `/*-game` routes, `/calendar`, `/video-call`, `/chat`, `/profile`, `/tracker`, Marks (`/mark/*`). - **Special guards:** `ReadingInterestGuard` (`/reading-journey`), `ParentGuard` (`/parent-dashboard`), `RoleGuard` Admin/Staff (`/admin/*`, `/keycloak-admin`), `unsavedChangesGuard` (`/profile`). Coverage is tracked against a committed **route manifest** so "100%" is measurable (every route visited + its primary actions exercised). --- ## The hard part: signup verification (email code + Twilio SMS) Registration is a 4-step stepper (`registration-stepper.component`) ending in email + phone verification: - `POST /api/authentication/initiate-pre-registration` → creates a Mongo `PreRegistration` with `emailVerificationCode` (6 digits) + `emailVerificationToken`. - `POST /api/authentication/verify-pre-registration-email` (code or token) → triggers Twilio Verify SMS. - `POST /api/authentication/verify-pre-registration-phone` → checks the code with **Twilio Verify** (code is NOT stored locally). - `POST /api/authentication/complete-pre-registration` → creates the Keycloak user once fully verified. **Email is automatable today** (read `emailVerificationCode`/`emailVerificationToken` from Mongo `PreRegistrations`, or hit the GET verify link). **SMS is not** — no bypass exists. **Proposed test hook (needs confirmation, backend change):** in non-prod only, thread the existing `SendVerificationCodeCommand.CustomCode` through pre-registration behind an env flag (e.g. `PreRegistration:E2E:Enabled` + `E2E_SMS_CODE`), OR provide an E2E-profile fake `ITwilioVerifyService` that accepts a fixed code, OR a test-only endpoint that marks `PhoneVerified` for a known pre-registration id. Must be **impossible to enable in Production**. Related existing flags: `Authentication:UseTestAuthentication` + `TestAuthenticationHandler` (API auth bypass for authenticated tests), `PreRegistration:Bypass*` (cooldowns/abuse only — not verification). For **authenticated journeys generally**, obtain a Keycloak token via direct grant against a dedicated non-prod test realm/user (or reuse `UseTestAuthentication` for API-level setup), and use Playwright storage-state to skip the interactive login on most specs. --- ## Telemetry integration (Jaeger + Seq) Build a reusable `e2e/telemetry` client used by specs and fixtures: - **Jaeger** (query API on the internal UI, `http://<node-ip>:16686`): `GET /api/services`, `GET /api/traces?service=SpikerSoft%20API&start&end&tags&lookback`, `GET /api/traces/:id`. Correlate by test time-window + service + tags (and, once the SPA emits W3C `traceparent`, by trace id). - **Seq** (`https://seq.spikersoft.com`): `GET /api/events?filter=<seq-query>&count=N` with `X-Seq-ApiKey`. Filter by `ServiceName`, `Environment`, level, and the test's correlation id. - Endpoints/keys via env (`JAEGER_QUERY_URL`, `SEQ_URL`, `SEQ_API_KEY`). A Playwright fixture snapshots "errors in Seq / error spans in Jaeger during this test" and attaches them to the report on failure. - (Stretch) Have the SPA propagate a per-session correlation header so browser actions map 1:1 to backend traces. --- ## Phased plan (each phase = its own linked sub-issue) - [ ] **P0 — Harness & telemetry:** page-object base, route manifest + coverage reporter, `JaegerClient`/`SeqClient`, failure-time telemetry fixture, CI wiring (promote `e2e-smoke` beyond continue-on-error for the anonymous suite). Decide target environment (dedicated staging vs prod-read-only). - [ ] **P1 — Anonymous walk:** every public route loads, primary nav works, no console/JS errors, key CTAs, `/tools` + every dev-tool, 404. (No backend mutations.) - [ ] **P2 — Selector hardening:** introduce a `data-testid` convention across the components the walk touches. - [ ] **P3 — Registration & verification:** land the verification test-hook (§4), then full signup happy-path + validation/error paths; create a reusable "fresh verified user" fixture. - [ ] **P4 — Authenticated core:** login/logout/profile, reading journey, book reader, calendar, chat, AI, lessons + code playgrounds, games. - [ ] **P5 — Parent & family:** parental approval, parent dashboard, child-account flows. - [ ] **P6 — Staff/Admin:** `/admin/*` review surfaces + `/keycloak-admin` named-outlet UI (role-scoped test users). - [ ] **P7 — Realtime/media:** video-call two-party, gameserver, streaming (build on existing fake-media scaffold). - [ ] **P8 — Coverage gate:** route-manifest coverage report in CI; ratchet toward 100%. ## Definition of done - Every route in the manifest is visited and its primary actions exercised by a passing spec. - Signup (incl. email + SMS verification) is fully automated via a safe, non-prod-only hook. - Each spec has a telemetry fixture that surfaces correlated Jaeger spans + Seq logs on failure. - CI runs the suite (anonymous suite blocking; authenticated suites at least nightly) with a coverage report. ## Open decisions (need a call before/at P0) 1. **Target environment** for E2E: stand up a dedicated staging stack, or run read-mostly against prod with disposable test accounts? (affects data-safety of P3+.) 2. **Verification hook** shape (CustomCode+flag vs fake service vs test endpoint) — security review required so it can never arm in Production. 3. Whether to invest in **SPA → W3C traceparent** propagation now (best trace correlation) or defer.
spikerj added the enhancement label 2026-07-01 19:14:35 +00:00
spikerj commented 2026-07-01 21:18:13 +00:00
Author
Owner
Copy Link
Copy Source

Phase sub-issues + status

Phase Issue Status
P0 Harness & telemetry #313 Largely done (PR spikersoft-angular#90); CI env wiring + coverage reporter remain
P1 Anonymous walk #314 Route-load coverage done, 25/25 green; per-page actions + tools walk remain
P2 data-testid convention #315 Not started
P3 Registration & verification #318 Not started (SMS hook is the blocker)
P4 Authenticated core #316 Not started
P5 Parent & family #317 Not started
P6 Staff/Admin #319 Not started
P7 Realtime & media #320 Not started
P8 Coverage gate #321 Not started

Bugs found by the suite so far

  • #308 /geography crash on API failure — fixed in spikersoft-angular#90
  • #309 /ai blank render — triaged, harness assertion issue, resolved in spikersoft-angular#90
  • #310 /blog crash on API failure — fixed in spikersoft-angular#90
  • #311 429s carry no CORS headers (middleware order) — fix open in spikersoft-backend#50
  • #312 GET /api/interests 401s anonymously — fix open in spikersoft-backend#50

Environment decisions locked in

  • Live-is-test for now; safe-mode split deferred.
  • E2E origin is https://localhost:4200 (already in Keycloak redirect URIs + API CORS allowlist; https://127.0.0.1:4200 also added to Keycloak). The suite runs API-connected.
## Phase sub-issues + status | Phase | Issue | Status | |---|---|---| | P0 Harness & telemetry | #313 | Largely done (PR spikersoft-angular#90); CI env wiring + coverage reporter remain | | P1 Anonymous walk | #314 | Route-load coverage done, **25/25 green**; per-page actions + tools walk remain | | P2 data-testid convention | #315 | Not started | | P3 Registration & verification | #318 | Not started (SMS hook is the blocker) | | P4 Authenticated core | #316 | Not started | | P5 Parent & family | #317 | Not started | | P6 Staff/Admin | #319 | Not started | | P7 Realtime & media | #320 | Not started | | P8 Coverage gate | #321 | Not started | ## Bugs found by the suite so far - #308 `/geography` crash on API failure — **fixed** in spikersoft-angular#90 - #309 `/ai` blank render — triaged, harness assertion issue, **resolved** in spikersoft-angular#90 - #310 `/blog` crash on API failure — **fixed** in spikersoft-angular#90 - #311 429s carry no CORS headers (middleware order) — **fix open** in spikersoft-backend#50 - #312 `GET /api/interests` 401s anonymously — **fix open** in spikersoft-backend#50 ## Environment decisions locked in - Live-is-test for now; safe-mode split deferred. - E2E origin is `https://localhost:4200` (already in Keycloak redirect URIs + API CORS allowlist; `https://127.0.0.1:4200` also added to Keycloak). The suite runs API-connected.
spikerj commented 2026-07-03 17:42:42 +00:00
Author
Owner
Copy Link
Copy Source

Phase status refresh (2026-07-03)

Phase Issue Status
P0 Harness & telemetry #313 Complete pending merge (angular PR #105: coverage reporter, telemetry env, blocking e2e-anonymous CI job)
P1 Anonymous walk #314 Nearly done (PR #104 merged: tools walk, 404, primary actions). Remaining: id-dependent click-throughs (needs seeded data) + console-error tightening (unblocked — #311/#312 closed)
P2 data-testid convention #315 Closed
P3 Registration & verification #318 Not started (SMS hook blocker)
P4 Authenticated core #316 Not started — next up; manifest now classifies all 43 auth routes + 9 runner/auth tools for it
P5 Parent & family #317 Not started
P6 Staff/Admin #319 Not started — manifest now lists all 15 admin routes + 12 keycloak-admin outlet views
P7 Realtime & media #320 Not started
P8 Coverage gate #321 Baseline established: route-coverage.json reports 32% of 97 declared routes walked; drift gate already blocking in CI

New bug from the P0 reporter: #340 (/keycloak-admin missing RoleGuard).

## Phase status refresh (2026-07-03) | Phase | Issue | Status | |---|---|---| | P0 Harness & telemetry | #313 | **Complete pending merge** (angular PR #105: coverage reporter, telemetry env, blocking `e2e-anonymous` CI job) | | P1 Anonymous walk | #314 | Nearly done (PR #104 merged: tools walk, 404, primary actions). Remaining: id-dependent click-throughs (needs seeded data) + console-error tightening (**unblocked** — #311/#312 closed) | | P2 data-testid convention | #315 | **Closed** | | P3 Registration & verification | #318 | Not started (SMS hook blocker) | | P4 Authenticated core | #316 | Not started — next up; manifest now classifies all 43 auth routes + 9 runner/auth tools for it | | P5 Parent & family | #317 | Not started | | P6 Staff/Admin | #319 | Not started — manifest now lists all 15 admin routes + 12 keycloak-admin outlet views | | P7 Realtime & media | #320 | Not started | | P8 Coverage gate | #321 | Baseline established: route-coverage.json reports **32% of 97 declared routes walked**; drift gate already blocking in CI | New bug from the P0 reporter: #340 (/keycloak-admin missing RoleGuard).
spikerj commented 2026-07-03 20:03:37 +00:00
Author
Owner
Copy Link
Copy Source

Phase status refresh (2026-07-03, evening)

Phase Issue Status
P0 Harness & telemetry #313 Closed (PR #105 + #106 merged; SEQ_API_KEY repo secret still to be created)
P1 Anonymous walk #314 Console-error hard-fail merged (PR #106); only the id-dependent click-throughs remain (seeded-data dependent)
P2 data-testid convention #315 Closed
P3 Registration & verification #318 Not started (SMS hook blocker)
P4 Authenticated core #316 Foundation merged (PR #107): storage-state Keycloak login, 38-route + 9-tool authenticated walk, nightly full-suite workflow. Blocked on a seeded live-realm test user (E2E_USER_USERNAME/E2E_USER_PASSWORD in .env.e2e + repo secrets); journeys next
P5 Parent & family #317 Not started
P6 Staff/Admin #319 Not started (needs a staff-role test account; /keycloak-admin now properly RoleGuard-gated)
P7 Realtime & media #320 Not started
P8 Coverage gate #321 Anonymous baseline 32%; jumps to ~72% once P4 creds activate the authenticated walk

Bugs from the suite closed/open today

  • #342 blockly ViewChild crash — fixed (PR #106)
  • #343 /tree-of-knowledge anonymous 401 noise — fixed (PR #106)
  • #340 /keycloak-admin missing RoleGuard — fixed (PR #108)
  • #341 rate limiter 503→429 — fix open (backend PR #59); the two /api/system/version allowlist entries in EXPECTED_CONSOLE_NOISE come out after it deploys
  • #344 (new, infra) SonarQube CE task fails server-side on every run — filed so the permanently-red context stops masking real regressions
## Phase status refresh (2026-07-03, evening) | Phase | Issue | Status | |---|---|---| | P0 Harness & telemetry | #313 | **Closed** (PR #105 + #106 merged; `SEQ_API_KEY` repo secret still to be created) | | P1 Anonymous walk | #314 | Console-error hard-fail merged (PR #106); only the id-dependent click-throughs remain (seeded-data dependent) | | P2 data-testid convention | #315 | Closed | | P3 Registration & verification | #318 | Not started (SMS hook blocker) | | P4 Authenticated core | #316 | **Foundation merged** (PR #107): storage-state Keycloak login, 38-route + 9-tool authenticated walk, nightly full-suite workflow. Blocked on a seeded live-realm test user (`E2E_USER_USERNAME`/`E2E_USER_PASSWORD` in `.env.e2e` + repo secrets); journeys next | | P5 Parent & family | #317 | Not started | | P6 Staff/Admin | #319 | Not started (needs a staff-role test account; `/keycloak-admin` now properly RoleGuard-gated) | | P7 Realtime & media | #320 | Not started | | P8 Coverage gate | #321 | Anonymous baseline 32%; jumps to ~72% once P4 creds activate the authenticated walk | ## Bugs from the suite closed/open today - #342 blockly ViewChild crash — **fixed** (PR #106) - #343 /tree-of-knowledge anonymous 401 noise — **fixed** (PR #106) - #340 /keycloak-admin missing RoleGuard — **fixed** (PR #108) - #341 rate limiter 503→429 — **fix open** (backend PR #59); the two `/api/system/version` allowlist entries in `EXPECTED_CONSOLE_NOISE` come out after it deploys - #344 (new, infra) SonarQube CE task fails server-side on every run — filed so the permanently-red context stops masking real regressions
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#307