SpikerSoft.Tests.Integration: test-host teardown flake — non-zero exit despite all tests passing when 3+ Lessons test classes share the TestContainers collection #450

New Issue

2026-07-07T15:49:57Z

spikerj commented

2026-07-07 15:49:57 +00:00

Observed while adding Phase 4 of the Lessons integration coverage (spikersoft-backend#176), but reproduces on master today with only the Phase 1-3 Lessons test files (spikersoft-backend#174, #175) — not a regression introduced by that work.

Symptom: running dotnet test with a filter that selects 3 or more of the SpikerSoft.Tests.Integration.Lessons.* test classes together (all sharing [Collection("TestContainers")]) reports every individual test as Passed, but the overall run prints Test Run Failed. and the process exits with code 1. Running any 2 of those classes together, or any single class alone, exits 0 with Test Run Successful.

Repro (all pass individually, but the run fails):

dotnet test SpikerSoft.Tests.Integration -c Debug --filter "FullyQualifiedName~SpikerSoft.Tests.Integration.Lessons.LessonCatalogAndAttemptIntegrationTests|FullyQualifiedName~SpikerSoft.Tests.Integration.Lessons.AnonymousLessonJourneyIntegrationTests|FullyQualifiedName~SpikerSoft.Tests.Integration.Lessons.LessonGradingIntegrationTests|FullyQualifiedName~SpikerSoft.Tests.Integration.Lessons.CodeExecutionSignalRIntegrationTests"
# => Total tests: 30 / Passed: 30 / "Test Run Failed." / exit code 1

Correlated with heavy HTTP request failed after ~2700ms / SocketError noise in the logs right around the point multiple WebApplicationFactory instances (and their in-process code-execution/regrade worker IHosts from TestEnvironmentFixture) are being disposed close together — looks like a background connection (Rabbit/Redis/Mongo client, or a lingering SignalR/TestServer socket) throwing after its owning test has already reported success, which the console runner or test host then surfaces as a failed run despite zero failed assertions.

Impact: CI green/red status for this project can't be trusted at the dotnet test process-exit-code level once enough Lessons tests exist in one collection; someone has to actually read the Passed: N / Total: N summary rather than trust the exit code. Low urgency since no test outcome is wrong, but worth root-causing before this collection grows further (Phase 5 will add more classes to it).

Suggested next step: reproduce with --diag / a memory dump or DOTNET_gcServer=0 + verbose logging to catch the actual unobserved exception; likely candidates are TestEnvironmentFixture's worker IHost.StopAsync/DisposeAsync ordering, or WebApplicationFactory teardown racing a still-open RabbitMQ channel from CodeExecutionWorkerHostedService / LessonRegradeWorkerHostedService.

Observed while adding Phase 4 of the Lessons integration coverage (spikersoft-backend#176), but reproduces on `master` today with only the Phase 1-3 Lessons test files (spikersoft-backend#174, #175) — not a regression introduced by that work. **Symptom**: running `dotnet test` with a filter that selects 3 or more of the `SpikerSoft.Tests.Integration.Lessons.*` test classes together (all sharing `[Collection("TestContainers")]`) reports every individual test as `Passed`, but the overall run prints `Test Run Failed.` and the process exits with code 1. Running any 2 of those classes together, or any single class alone, exits 0 with `Test Run Successful.` Repro (all pass individually, but the run fails): ``` dotnet test SpikerSoft.Tests.Integration -c Debug --filter "FullyQualifiedName~SpikerSoft.Tests.Integration.Lessons.LessonCatalogAndAttemptIntegrationTests|FullyQualifiedName~SpikerSoft.Tests.Integration.Lessons.AnonymousLessonJourneyIntegrationTests|FullyQualifiedName~SpikerSoft.Tests.Integration.Lessons.LessonGradingIntegrationTests|FullyQualifiedName~SpikerSoft.Tests.Integration.Lessons.CodeExecutionSignalRIntegrationTests" # => Total tests: 30 / Passed: 30 / "Test Run Failed." / exit code 1 ``` Correlated with heavy `HTTP request failed after ~2700ms` / `SocketError` noise in the logs right around the point multiple `WebApplicationFactory` instances (and their in-process code-execution/regrade worker `IHost`s from `TestEnvironmentFixture`) are being disposed close together — looks like a background connection (Rabbit/Redis/Mongo client, or a lingering SignalR/TestServer socket) throwing after its owning test has already reported success, which the console runner or test host then surfaces as a failed run despite zero failed assertions. **Impact**: CI green/red status for this project can't be trusted at the `dotnet test` process-exit-code level once enough Lessons tests exist in one collection; someone has to actually read the `Passed: N / Total: N` summary rather than trust the exit code. Low urgency since no test outcome is wrong, but worth root-causing before this collection grows further (Phase 5 will add more classes to it). **Suggested next step**: reproduce with `--diag` / a memory dump or `DOTNET_gcServer=0` + verbose logging to catch the actual unobserved exception; likely candidates are `TestEnvironmentFixture`'s worker `IHost.StopAsync`/`DisposeAsync` ordering, or `WebApplicationFactory` teardown racing a still-open RabbitMQ channel from `CodeExecutionWorkerHostedService` / `LessonRegradeWorkerHostedService`.

spikerj referenced this issue

2026-07-07 16:15:28 +00:00

Integration tests: Phase 5 multi-language grading (Python/Node/C/C++) delivered #452

spikerj commented

2026-07-07 18:01:38 +00:00

Root-caused and fixed in spikersoft-backend#179 (merged to master).

Root cause — not the fixture's worker IHost ordering or a lingering worker RabbitMQ channel. The unobserved exception is an xUnit class cleanup failure on CodeExecutionSignalRIntegrationTests (the only Lessons class that registers SignalRNotificationConsumerService in its WebApplicationFactory):

Under minimal hosting, WebApplicationFactory.Dispose() stops the host twice from two threads (dotnet/aspnetcore#40271) — once from the factory's dispose thread and once from the app.Run() thread unblocked via ApplicationStopping. Diag logs show "SignalR Notification Consumer is stopping" twice at the same timestamp.
SignalRNotificationConsumerService.StopAsync wasn't idempotent, so the two concurrent calls raced CloseAsync()/Dispose() on the same RabbitMQ AutorecoveringChannel, and the loser threw ObjectDisposedException from inside AutorecoveringChannel.CloseAsync's cleanup (DeleteRecordedChannelAsync → get_ConsumerTags).
That lands during class-fixture teardown after all tests already reported Passed, so xUnit records a cleanup failure and VSTest exits 1 with Passed: N / Total: N. It's a timing race — hence needing 3+ classes' teardowns close together to reproduce reliably.

Fix — StopAsync now guards the channel/connection close with Interlocked.Exchange so only the first caller closes; every caller still awaits base.StopAsync. Nothing is suppressed; the real failure mode (double host stop) simply no longer corrupts the close sequence.

Verified on the fix branch (exit 0 for all): the 4-class repro filter from this ticket (30/30), the full SpikerSoft.Tests.Integration.Lessons namespace (42/42, previously exit 1), and the full integration suite minus the Redis-cluster classes (128 passed / 3 skipped) — those three Redis-cluster classes fail on this machine only because an unrelated compose stack is squatting on their fixed host ports 7014–7016 (pre-existing, same on unpatched master, unrelated to this fix).

Root-caused and fixed in spikersoft-backend#179 (merged to master). **Root cause** — not the fixture's worker `IHost` ordering or a lingering worker RabbitMQ channel. The unobserved exception is an xUnit **class cleanup failure** on `CodeExecutionSignalRIntegrationTests` (the only Lessons class that registers `SignalRNotificationConsumerService` in its `WebApplicationFactory`): - Under minimal hosting, `WebApplicationFactory.Dispose()` stops the host **twice from two threads** (dotnet/aspnetcore#40271) — once from the factory's dispose thread and once from the `app.Run()` thread unblocked via `ApplicationStopping`. Diag logs show "SignalR Notification Consumer is stopping" twice at the same timestamp. - `SignalRNotificationConsumerService.StopAsync` wasn't idempotent, so the two concurrent calls raced `CloseAsync()`/`Dispose()` on the same RabbitMQ `AutorecoveringChannel`, and the loser threw `ObjectDisposedException` from inside `AutorecoveringChannel.CloseAsync`'s cleanup (`DeleteRecordedChannelAsync` → `get_ConsumerTags`). - That lands during class-fixture teardown after all tests already reported Passed, so xUnit records a cleanup failure and VSTest exits 1 with `Passed: N / Total: N`. It's a timing race — hence needing 3+ classes' teardowns close together to reproduce reliably. **Fix** — `StopAsync` now guards the channel/connection close with `Interlocked.Exchange` so only the first caller closes; every caller still awaits `base.StopAsync`. Nothing is suppressed; the real failure mode (double host stop) simply no longer corrupts the close sequence. **Verified on the fix branch (exit 0 for all):** the 4-class repro filter from this ticket (30/30), the full `SpikerSoft.Tests.Integration.Lessons` namespace (42/42, previously exit 1), and the full integration suite minus the Redis-cluster classes (128 passed / 3 skipped) — those three Redis-cluster classes fail on this machine only because an unrelated compose stack is squatting on their fixed host ports 7014–7016 (pre-existing, same on unpatched master, unrelated to this fix).

spikerj closed this issue

2026-07-07 18:01:44 +00:00

spikerj referenced this issue

2026-07-09 00:08:57 +00:00

backend: expand integration coverage — art pipe messaging, safety RPC, GPU leases, quiz path #465

spikerj referenced this issue

2026-07-09 02:27:03 +00:00

ci: re-activate the integration-tests job when new CI hardware arrives #461

spikerj referenced this issue

2026-07-22 02:32:53 +00:00

Epic: Unified scheduling — recurrence engine + availability (calendar + iCal) #781

Sign in to join this conversation.