spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

backend: bring SpikerSoft.Tests.Integration back online — prod-parity Redis fixtures, rot fixes, G7 RESP3/AllowAdmin coverage #444

New Issue
Closed
opened 2026-07-07 02:35:27 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-07 02:35:27 +00:00
Owner
Copy Link
Copy Source

Migrated from spikersoft-backend#163 (filed there by mistake — all tickets belong in this tracker).

Repo: spikersoft-backend · Branch: test/integration-modernization-163

Context

SpikerSoft.Tests.Integration has rotted and is effectively offline (didn't compile on master). Meanwhile the highest-risk recent changes are exactly the kind only integration tests can verify:

  • NuGet G7 (spikersoft-backend PR #155) upgraded StackExchange.Redis 2.13.1 → 3.0.11 (RESP3 by default, AllowAdmin gating on Execute() admin commands) and NRedisStack → 1.6.0. The PR's two post-merge test-plan items (live cluster health check, FT.SEARCH under RESP3) were unverified.
  • ~11,300 unit tests are mock-based and structurally cannot catch protocol/infra behavior.

Problems found

  1. Wrong Redis everywhere. Production runs a 6-node redis/redis-stack-server:latest cluster with RediSearch/ReJSON/TS/Bloom loaded; fixtures used redis:7-alpine (no modules) and dead-end Bitnami images.
  2. Compile rot: DecompileSimpleIntegrationTests referenced APIs that no longer exist (3× CS1061 on master).
  3. Stale docs/infra: README from Oct 2025; redundant in-container test-runner compose service.
  4. API not testable in-process (discovered during validation): several production behaviors broke WebApplicationFactory hosting — see checklist.

Progress

  • Single-node RedisFixture → redis/redis-stack-server:latest (prod parity, modules loaded)
  • RedisClusterTestFixture → 6-node cluster in one redis-stack container (fixes Windows Docker networking + duplicate-node detection; cluster-node-timeout 15s)
  • DecompileSimpleIntegrationTests rewritten against current DecompilationService API
  • New G7-closing tests: FT.SEARCH KNN under RESP3, RedisVectorIndexInitializationService idempotent index creation, Redis.OM round-trip, cluster health checks with AllowAdmin
  • TestEnvironmentFixture actually starts Mongo 8 + delayed-message RabbitMQ + redis-stack via Testcontainers; publishes dynamic endpoints as env vars + in-memory config for WebApplicationFactory hosts
  • Compose cleanup (single-container cluster, no test-runner) + helper scripts + README rewrite
  • API fix: Art Studio guardrail/options registrations moved out of the HostedServices:Enabled gate (MediatR handlers need them; DI validation failed in test hosts)
  • API fix: RabbitMQ:*/Rabbit:* config both overridden by fixture (API reads RabbitMQ:* first; baked-in rabbitmq hostname won before)
  • API fix: https-redirect middleware config-gated (HttpsRedirect:Enabled) — TestServer's http client had every POST 302→GET (405)
  • API fix: GeoNames postal seed config-gated (Seeding:PostalCodes:Enabled) — external download + ~60s retry backoff on every fresh test DB
  • API fix: RedisRateLimitMiddleware now honors the already-bound RateLimiting:Enabled kill switch (TestServer requests share one fingerprint and tripped 3/300s limits)
  • PreRegistration tests updated for required dateOfBirth + no-Twilio environment — 7/7 green
  • Full-suite validation run (in progress; Books auth assertions + Decompile performance tests still failing, DLQ blocking-wait hang under investigation)
  • PR into master
  • CI: wire the suite into Gitea Actions as a separate, initially non-blocking job (Docker-enabled runner required)

Notes

  • Testcontainers 4.13 requires the image in the builder constructor — migrated.
  • Prod pins redis/redis-stack-server:latest; consider pinning to a digest, and note redis-stack-server is on a deprecation path in favor of redis:8 (follow-up infra ticket).
  • Related: spikersoft-backend PR #155 (G7), spikersoft-infrastructure redis-cluster stack.
Migrated from spikersoft-backend#163 (filed there by mistake — all tickets belong in this tracker). Repo: `spikersoft-backend` · Branch: `test/integration-modernization-163` ## Context `SpikerSoft.Tests.Integration` has rotted and is effectively offline (didn't compile on master). Meanwhile the highest-risk recent changes are exactly the kind only integration tests can verify: - **NuGet G7** (spikersoft-backend PR #155) upgraded StackExchange.Redis 2.13.1 → 3.0.11 (RESP3 by default, `AllowAdmin` gating on `Execute()` admin commands) and NRedisStack → 1.6.0. The PR's two post-merge test-plan items (live cluster health check, FT.SEARCH under RESP3) were unverified. - ~11,300 unit tests are mock-based and structurally cannot catch protocol/infra behavior. ## Problems found 1. **Wrong Redis everywhere.** Production runs a 6-node `redis/redis-stack-server:latest` cluster with RediSearch/ReJSON/TS/Bloom loaded; fixtures used `redis:7-alpine` (no modules) and dead-end Bitnami images. 2. **Compile rot**: `DecompileSimpleIntegrationTests` referenced APIs that no longer exist (3× CS1061 on master). 3. **Stale docs/infra**: README from Oct 2025; redundant in-container `test-runner` compose service. 4. **API not testable in-process** (discovered during validation): several production behaviors broke `WebApplicationFactory` hosting — see checklist. ## Progress - [x] Single-node `RedisFixture` → `redis/redis-stack-server:latest` (prod parity, modules loaded) - [x] `RedisClusterTestFixture` → 6-node cluster in one redis-stack container (fixes Windows Docker networking + duplicate-node detection; `cluster-node-timeout` 15s) - [x] `DecompileSimpleIntegrationTests` rewritten against current `DecompilationService` API - [x] New G7-closing tests: FT.SEARCH KNN under RESP3, `RedisVectorIndexInitializationService` idempotent index creation, Redis.OM round-trip, cluster health checks with `AllowAdmin` - [x] `TestEnvironmentFixture` actually starts Mongo 8 + delayed-message RabbitMQ + redis-stack via Testcontainers; publishes dynamic endpoints as env vars + in-memory config for `WebApplicationFactory` hosts - [x] Compose cleanup (single-container cluster, no test-runner) + helper scripts + README rewrite - [x] API fix: Art Studio guardrail/options registrations moved out of the `HostedServices:Enabled` gate (MediatR handlers need them; DI validation failed in test hosts) - [x] API fix: `RabbitMQ:*`/`Rabbit:*` config both overridden by fixture (API reads `RabbitMQ:*` first; baked-in `rabbitmq` hostname won before) - [x] API fix: https-redirect middleware config-gated (`HttpsRedirect:Enabled`) — TestServer's http client had every POST 302→GET (405) - [x] API fix: GeoNames postal seed config-gated (`Seeding:PostalCodes:Enabled`) — external download + ~60s retry backoff on every fresh test DB - [x] API fix: `RedisRateLimitMiddleware` now honors the already-bound `RateLimiting:Enabled` kill switch (TestServer requests share one fingerprint and tripped 3/300s limits) - [x] PreRegistration tests updated for required `dateOfBirth` + no-Twilio environment — 7/7 green - [ ] Full-suite validation run (in progress; Books auth assertions + Decompile performance tests still failing, DLQ blocking-wait hang under investigation) - [ ] PR into master - [ ] CI: wire the suite into Gitea Actions as a separate, initially non-blocking job (Docker-enabled runner required) ## Notes - Testcontainers 4.13 requires the image in the builder constructor — migrated. - Prod pins `redis/redis-stack-server:latest`; consider pinning to a digest, and note redis-stack-server is on a deprecation path in favor of `redis:8` (follow-up infra ticket). - Related: spikersoft-backend PR #155 (G7), spikersoft-infrastructure redis-cluster stack.
spikerj commented 2026-07-07 04:25:15 +00:00
Author
Owner
Copy Link
Copy Source

PR up: spikerj/spikersoft-backend#167 — full suite green (100 passed / 0 failed / 3 skipped, ~2m30s, self-contained Testcontainers).

Beyond the checklist above, the validation run surfaced and fixed:

  • Test auth: config-flag selection doesn't work under minimal hosting (factory config invisible to Program.Main); the old process-wide env var authenticated every host and broke all 401 tests. Now a per-factory UseTestAuthentication() extension via ConfigureTestServices; bare clients keep real JWT auth.
  • Decompile RPC: /api/decompile/* dispatches over RabbitMQ to the decompile worker — the fixture now hosts DecompileRpcConsumerHostedService + Workers.Decompile handlers in-process against the test broker (fixed 8 tests that each burned the 60s RPC timeout).
  • Suite hang root cause: RedisL2CacheServiceTests had a duplicate IClassFixture<TestEnvironmentFixture> next to the collection fixture; its second container set published process env vars that outlived disposal, so every later host connected to dead containers (Mongo connection-refused cascade + 30s server-selection stalls).
  • Assorted test rot: wrong client in the decompile 401 test, GET /api/book vs /api/book/all, missing ObjectId/enum JSON converters client-side, DLQ tests consuming a queue the infrastructure never creates and asserting header names the helper never writes (exceptions swallowed in consumer callbacks → 60s timeouts), stale quiz tendency threshold.

Remaining: CI wiring (Gitea Actions, non-blocking job, Docker-enabled runner).

PR up: spikerj/spikersoft-backend#167 — full suite green (100 passed / 0 failed / 3 skipped, ~2m30s, self-contained Testcontainers). Beyond the checklist above, the validation run surfaced and fixed: - **Test auth**: config-flag selection doesn't work under minimal hosting (factory config invisible to `Program.Main`); the old process-wide env var authenticated every host and broke all 401 tests. Now a per-factory `UseTestAuthentication()` extension via `ConfigureTestServices`; bare clients keep real JWT auth. - **Decompile RPC**: `/api/decompile/*` dispatches over RabbitMQ to the decompile worker — the fixture now hosts `DecompileRpcConsumerHostedService` + `Workers.Decompile` handlers in-process against the test broker (fixed 8 tests that each burned the 60s RPC timeout). - **Suite hang root cause**: `RedisL2CacheServiceTests` had a duplicate `IClassFixture<TestEnvironmentFixture>` next to the collection fixture; its second container set published process env vars that outlived disposal, so every later host connected to dead containers (Mongo connection-refused cascade + 30s server-selection stalls). - Assorted test rot: wrong client in the decompile 401 test, `GET /api/book` vs `/api/book/all`, missing ObjectId/enum JSON converters client-side, DLQ tests consuming a queue the infrastructure never creates and asserting header names the helper never writes (exceptions swallowed in consumer callbacks → 60s timeouts), stale quiz tendency threshold. Remaining: CI wiring (Gitea Actions, non-blocking job, Docker-enabled runner).
spikerj commented 2026-07-08 22:11:10 +00:00
Author
Owner
Copy Link
Copy Source

Closing — all checklist items are resolved.

Full-suite validation run — done: 100 passed / 0 failed / 3 skipped, ~2m30s, self-contained Testcontainers (see comment above).

PR into master — done: PR #167 (merged 2026-07-07), plus follow-ups #170 (cluster fixture off port 7000), #171 (cross-platform decompile filename sanitization), #172 (Books:StagingPath temp redirect). The suite has since been extended by PRs #174–#177 (lessons/grading integration phases).

CI wiring — done via spikersoft-backend PR #184 (merged): the integration-tests job in .gitea/workflows/tests.yml was rewritten for the Testcontainers suite (restore/build + dotnet test, no compose orchestration — the old job referenced compose services deleted by this modernization). Per the decision to keep it out of the active pipeline while CI hardware is overtaxed, the job is deliberately deactivated: manual workflow_dispatch with run_integration_tests=true only, input default false. Re-activation when the new servers arrive is tracked in #461 (checklist there; also documented in both READMEs).

Notes follow-up — the redis-stack digest-pinning / redis:8 migration item is filed as #462.

Related fix that fell out of #184's CI run: 6 unit tests broken on master by the ArtStudio method additions (PRs #180/#181) were repaired in PR #185 (merged).

Closing — all checklist items are resolved. **Full-suite validation run** — done: 100 passed / 0 failed / 3 skipped, ~2m30s, self-contained Testcontainers (see comment above). **PR into master** — done: PR #167 (merged 2026-07-07), plus follow-ups #170 (cluster fixture off port 7000), #171 (cross-platform decompile filename sanitization), #172 (Books:StagingPath temp redirect). The suite has since been extended by PRs #174–#177 (lessons/grading integration phases). **CI wiring** — done via spikersoft-backend PR #184 (merged): the `integration-tests` job in `.gitea/workflows/tests.yml` was rewritten for the Testcontainers suite (restore/build + `dotnet test`, no compose orchestration — the old job referenced compose services deleted by this modernization). Per the decision to keep it out of the active pipeline while CI hardware is overtaxed, the job is **deliberately deactivated**: manual `workflow_dispatch` with `run_integration_tests=true` only, input default `false`. Re-activation when the new servers arrive is tracked in **#461** (checklist there; also documented in both READMEs). **Notes follow-up** — the redis-stack digest-pinning / redis:8 migration item is filed as **#462**. Related fix that fell out of #184's CI run: 6 unit tests broken on master by the ArtStudio method additions (PRs #180/#181) were repaired in PR #185 (merged).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#444