backend: expand integration coverage — art pipe messaging, safety RPC, GPU leases, quiz path #465

New Issue

2026-07-09T00:08:57Z

spikerj commented

2026-07-09 00:08:57 +00:00

Follow-up to #444 (suite modernization, closed). SpikerSoft.Tests.Integration is green (100 passed / 3 skipped, ~2m30s) but several recently-shipped areas have zero integration coverage. This ticket tracks expanding the suite while keeping its contract: full run green and under ~5 minutes on a dev box, no external network beyond image pulls, no real model weights/GPU, new areas reuse the existing collection/fixture patterns (TestContainers collection on TestEnvironmentFixture), and the #450 teardown ordering (hosts stop before containers) is preserved.

One PR per area, branched test/<topic> off master, referencing this ticket. Checked off as PRs merge.

Areas

Redis image pinning (#462) — pin redis/redis-stack-server to a digest in lockstep across RedisFixture.RedisImage, RedisClusterTestFixture.RedisImage, docker-compose.test.yml, and spikersoft-infrastructure/redis-cluster/docker-stack.yml; then migrate the same four locations to a pinned redis:8 (modules bundled in core; --loadmodule lines dropped), validated by the suite's Redis/RediSearch/cluster coverage. Done: backend PRs #190/#191 + infra PRs #14/#15 (all merged); also backend PR #189 (cluster fixture port-collision hardening) and a real prod bug fix (RediSearch 8.x "Index not found" wording). Prod rollout stays on #462.
Art Studio pipeline messaging (#368 resident mode) — stage dispatch honoring ArtStudio:StageModelMap (art.model.<model>.tasks vs per-stage art.asset.stage.<stage>.requests), ArtPipeStageConsumer binding resolution (resident vs stage mode) against the real broker, restart-from-stage routing. Fake stage executor (ArtPipe:UseFakeExecutor); real RabbitMQ container. Done: backend PR #192 (merged) — API-side dispatch asserted via probe queues bound to the production routing keys, consumer hosted in-process on the worker's own internal DI registration, end-to-end message → Completed run/asset in Mongo. Full suite 148 passed / 3 skipped, 4m28s.
Remote safety gate RPC — SafetyCheckRpcConsumer + RemoteArtifactSafetyGate round-trip over the RabbitMQ container (request → verdict reply → correlation id), plus timeout with fail-open/fail-closed policy and flagged-verdict propagation. Done: backend PR #193 (merged) — real client vs real responder (SafetyCheck-resident-shaped host, faked classifier only), CorrelationId isolation under concurrency, responder-side per-kind threshold + checker-error fail-closed folding, 1 s timeout fail-closed/fail-open. Full suite 154 passed / 3 skipped, 4m58s.
GPU lease protocol — SharedGpuLeaseService against a hosted GpuCoordinator (LeaseManagerService + LeaseTimeoutWatchdog): grant, queue-when-exhausted, task-complete Retain/Yield, release, resident-lease exemption from timeout reaping, MaxSkipCount head-of-line behavior. Protects the capacity work in flight (#367). Done: backend PR #194 (merged) — real client vs real coordinator over the broker, incl. #366 queue-position updates, second-scale watchdog reaping an idle per-task lease while the resident lease survives, and MaxSkipCount=3 skip-then-block-then-drain-in-order. Full suite 159 passed / 3 skipped, 4m54s.
Quiz generation message path — publish to the quiz.generation exchange and verify the worker's consume/ack/DLQ behavior with the LLM faked behind IQuizGenerationService (no GGUF loads in tests). Done: backend PR #195 (merged) — real QuizGenerationHostedService+QuizGenerationWorkerService consume from generation.process, quiz Completed in Mongo with faked questions + 100% notification + ACK; model-init failure rides the real retry TTL queue into generation.process.dlq with forensic headers; "null" payload nacked-without-requeue and the consumer keeps serving. Also documented the RetryDelays binder append pitfall. Full suite 162 passed / 3 skipped, 5m07s.
Book upload tests — revisit the three [Fact(Skip="Requires actual PDF file and AI model")] tests: convert what's testable with a tiny checked-in PDF and a faked AI service; leave truly model-bound assertions skipped with a sharper reason. Done: backend PR #196 (merged) — the skip premise was stale (test-book.pdf exists in TestAssets; upload-pdf no longer generates quizzes inline). Replaced with coverage of the deferred approval pipeline: 200 + SHA-256/metadata + null QuizId, UploadWorkflowState pending_approval in Mongo + byte-for-byte staged file, staff approval → received + upload.received published to RabbitMQ, non-staff approval → 403 (new TestStaffAuthenticationHandler). Nothing left model-bound, so the suite now has zero skips. Full suite 166 passed / 0 skipped, ~5m.
CI note (#461) — verify the expanded suite runs cleanly via the workflow's manual-dispatch invocation locally and note readiness on #461. CI defaults stay off (blocked on hardware). Done: Release-config run with the exact workflow invocation — 166 passed / 0 failed / 0 skipped, 5m19s; readiness noted on #461. No workflow changes.

Repos: spikersoft-backend (primary), spikersoft-infrastructure (redis pinning only).

Follow-up to #444 (suite modernization, closed). `SpikerSoft.Tests.Integration` is green (100 passed / 3 skipped, ~2m30s) but several recently-shipped areas have zero integration coverage. This ticket tracks expanding the suite while keeping its contract: full run green and under ~5 minutes on a dev box, no external network beyond image pulls, no real model weights/GPU, new areas reuse the existing collection/fixture patterns (`TestContainers` collection on `TestEnvironmentFixture`), and the #450 teardown ordering (hosts stop before containers) is preserved. One PR per area, branched `test/<topic>` off master, referencing this ticket. Checked off as PRs merge. ## Areas - [x] **Redis image pinning (#462)** — pin `redis/redis-stack-server` to a digest in lockstep across `RedisFixture.RedisImage`, `RedisClusterTestFixture.RedisImage`, `docker-compose.test.yml`, and `spikersoft-infrastructure/redis-cluster/docker-stack.yml`; then migrate the same four locations to a pinned `redis:8` (modules bundled in core; `--loadmodule` lines dropped), validated by the suite's Redis/RediSearch/cluster coverage. *Done: backend PRs #190/#191 + infra PRs #14/#15 (all merged); also backend PR #189 (cluster fixture port-collision hardening) and a real prod bug fix (RediSearch 8.x "Index not found" wording). Prod rollout stays on #462.* - [x] **Art Studio pipeline messaging (#368 resident mode)** — stage dispatch honoring `ArtStudio:StageModelMap` (`art.model.<model>.tasks` vs per-stage `art.asset.stage.<stage>.requests`), `ArtPipeStageConsumer` binding resolution (resident vs stage mode) against the real broker, restart-from-stage routing. Fake stage executor (`ArtPipe:UseFakeExecutor`); real RabbitMQ container. *Done: backend PR #192 (merged) — API-side dispatch asserted via probe queues bound to the production routing keys, consumer hosted in-process on the worker's own internal DI registration, end-to-end message → Completed run/asset in Mongo. Full suite 148 passed / 3 skipped, 4m28s.* - [x] **Remote safety gate RPC** — `SafetyCheckRpcConsumer` + `RemoteArtifactSafetyGate` round-trip over the RabbitMQ container (request → verdict reply → correlation id), plus timeout with fail-open/fail-closed policy and flagged-verdict propagation. *Done: backend PR #193 (merged) — real client vs real responder (SafetyCheck-resident-shaped host, faked classifier only), CorrelationId isolation under concurrency, responder-side per-kind threshold + checker-error fail-closed folding, 1 s timeout fail-closed/fail-open. Full suite 154 passed / 3 skipped, 4m58s.* - [x] **GPU lease protocol** — `SharedGpuLeaseService` against a hosted GpuCoordinator (`LeaseManagerService` + `LeaseTimeoutWatchdog`): grant, queue-when-exhausted, task-complete Retain/Yield, release, resident-lease exemption from timeout reaping, `MaxSkipCount` head-of-line behavior. Protects the capacity work in flight (#367). *Done: backend PR #194 (merged) — real client vs real coordinator over the broker, incl. #366 queue-position updates, second-scale watchdog reaping an idle per-task lease while the resident lease survives, and MaxSkipCount=3 skip-then-block-then-drain-in-order. Full suite 159 passed / 3 skipped, 4m54s.* - [x] **Quiz generation message path** — publish to the `quiz.generation` exchange and verify the worker's consume/ack/DLQ behavior with the LLM faked behind `IQuizGenerationService` (no GGUF loads in tests). *Done: backend PR #195 (merged) — real `QuizGenerationHostedService`+`QuizGenerationWorkerService` consume from `generation.process`, quiz Completed in Mongo with faked questions + 100% notification + ACK; model-init failure rides the real retry TTL queue into `generation.process.dlq` with forensic headers; `"null"` payload nacked-without-requeue and the consumer keeps serving. Also documented the RetryDelays binder append pitfall. Full suite 162 passed / 3 skipped, 5m07s.* - [x] **Book upload tests** — revisit the three `[Fact(Skip="Requires actual PDF file and AI model")]` tests: convert what's testable with a tiny checked-in PDF and a faked AI service; leave truly model-bound assertions skipped with a sharper reason. *Done: backend PR #196 (merged) — the skip premise was stale (`test-book.pdf` exists in TestAssets; `upload-pdf` no longer generates quizzes inline). Replaced with coverage of the deferred approval pipeline: 200 + SHA-256/metadata + null QuizId, `UploadWorkflowState` pending_approval in Mongo + byte-for-byte staged file, staff approval → received + `upload.received` published to RabbitMQ, non-staff approval → 403 (new `TestStaffAuthenticationHandler`). Nothing left model-bound, so the suite now has zero skips. Full suite 166 passed / 0 skipped, ~5m.* - [x] **CI note (#461)** — verify the expanded suite runs cleanly via the workflow's manual-dispatch invocation locally and note readiness on #461. CI defaults stay off (blocked on hardware). *Done: Release-config run with the exact workflow invocation — 166 passed / 0 failed / 0 skipped, 5m19s; readiness noted on #461. No workflow changes.* Repos: spikersoft-backend (primary), spikersoft-infrastructure (redis pinning only).

spikerj referenced this issue

2026-07-09 02:27:03 +00:00

ci: re-activate the integration-tests job when new CI hardware arrives #461

spikerj commented

2026-07-09 02:27:50 +00:00

All areas complete — closing.

Final state: 166 passed / 0 failed / 0 skipped, 5m19s under the CI Release invocation (was 100 / 3 skipped, ~2m30s at filing). Backend PRs #189–#196 and infra PRs #14/#15, all merged. CI readiness noted on #461; prod Redis rollout tracked on #462.

One scope note: spikersoft-infrastructure/redis-cluster/backup.yml was intentionally left untouched during the Redis image pinning/migration — it's dead config (confirmed up front), so only docker-stack.yml was updated on the infra side.

All areas complete — closing. **Final state:** 166 passed / 0 failed / 0 skipped, 5m19s under the CI Release invocation (was 100 / 3 skipped, ~2m30s at filing). Backend PRs #189–#196 and infra PRs #14/#15, all merged. CI readiness noted on #461; prod Redis rollout tracked on #462. One scope note: `spikersoft-infrastructure/redis-cluster/backup.yml` was intentionally left untouched during the Redis image pinning/migration — it's dead config (confirmed up front), so only `docker-stack.yml` was updated on the infra side.

spikerj closed this issue

2026-07-09 02:27:55 +00:00

spikerj referenced this issue

2026-07-09 03:40:25 +00:00

ci: re-activate the integration-tests job when new CI hardware arrives #461

spikerj referenced this issue

2026-07-20 17:07:51 +00:00

Registration e2e step-1 specs broken by E5 Signal Forms migration — [formcontrolname] locators no longer match (CI e2e-anonymous red on master) #737

spikerj referenced this issue

2026-07-20 17:11:07 +00:00

Child map never shows parent approval of a wishlisted state — minor-mode wishlist branch masks the approved style (profile → travel → map) #735

Sign in to join this conversation.