ArtPipeProcessor host has no RabbitMQ startup retry/backoff — 7s crash-loop + Seq flood whenever the broker is away #886

New Issue

2026-07-28T14:24:31Z

spikerj commented

2026-07-28 14:24:31 +00:00

Observed during the #885 broker outage: SpikerSoft.EventHandlers.ArtPipeProcessor (resident artpipe-model-safety service) attempts its RabbitMQ connection ~0s after boot, and a failure is fatal: Failed to create RabbitMQ connection → Hosting failed to start → exit 1. Swarm's on-failure restart (5s delay) turns that into a ~7s crash-loop — 3 error events per cycle, ~1,500+ Seq errors over the 2h outage, plus constant container churn on the GPU node.

Contrast: MassTransit consumers auto-recover and re-declare topology when the broker returns (per rabbitmq/docker-stack.yml deploy notes); this host's bespoke RabbitMQ.Connection bootstrap does not.

Fix: wrap broker connection creation at startup in retry-with-backoff (e.g. Polly exponential, cap ~30-60s, log Warning per attempt and Error only after N failures), or let the host start degraded and connect lazily with the client's built-in automatic recovery. Applies to whatever shares this bootstrap in SpikerSoft.EventHandlers.Infrastructure — audit the other EventHandler hosts for the same instant-fatal pattern (they were quiet during #885 only because none of them restarted).

Repro/observe: stop the broker task; watch spikersoft-artpipe-model-safety restart-loop and Seq fill with the triplet every ~7s (first occurrence today 12:17:22Z, 2026-07-28).

Observed during the #885 broker outage: `SpikerSoft.EventHandlers.ArtPipeProcessor` (resident `artpipe-model-safety` service) attempts its RabbitMQ connection ~0s after boot, and a failure is fatal: `Failed to create RabbitMQ connection` → `Hosting failed to start` → exit 1. Swarm's `on-failure` restart (5s delay) turns that into a ~7s crash-loop — 3 error events per cycle, ~1,500+ Seq errors over the 2h outage, plus constant container churn on the GPU node. Contrast: MassTransit consumers auto-recover and re-declare topology when the broker returns (per rabbitmq/docker-stack.yml deploy notes); this host's bespoke `RabbitMQ.Connection` bootstrap does not. **Fix:** wrap broker connection creation at startup in retry-with-backoff (e.g. Polly exponential, cap ~30-60s, log Warning per attempt and Error only after N failures), or let the host start degraded and connect lazily with the client's built-in automatic recovery. Applies to whatever shares this bootstrap in `SpikerSoft.EventHandlers.Infrastructure` — audit the other EventHandler hosts for the same instant-fatal pattern (they were quiet during #885 only because none of them restarted). Repro/observe: stop the broker task; watch `spikersoft-artpipe-model-safety` restart-loop and Seq fill with the triplet every ~7s (first occurrence today 12:17:22Z, 2026-07-28).

spikerj referenced this issue

2026-07-28 14:45:57 +00:00

OUTAGE: dreamstream4 down → RabbitMQ backbone unreachable since 12:17Z — consumers can't connect, redis node lost #885

spikerj commented

2026-07-29 15:21:20 +00:00

Re-verified against spikersoft-backend origin/master @ 766cb8e4 (2026-07-29) — still unfixed, and the requested audit of the other EventHandler hosts is done. Notes updated so nobody has to re-derive this:

ArtPipeProcessor, confirmed: SpikerSoft.EventHandlers.ArtPipeProcessor/Services/ArtPipeStageConsumer.cs:133 calls factory.CreateConnectionAsync(...) with no retry wrapper. ExecuteAsync awaits it at :105 inside a try whose catch (Exception ex) logs "Fatal error in ArtPipeStageConsumer" and rethrows (:112) — so a broker-down boot is fatal exactly as described.

The AutomaticRecoveryEnabled = true / NetworkRecoveryInterval / TopologyRecoveryEnabled settings at :128 are a red herring for this failure mode: the RabbitMQ client only recovers a connection that was established at least once. It does nothing for a failure on the initial CreateConnectionAsync.

Audit result — the pattern is fleet-wide, not ArtPipeProcessor-specific. All 24 SpikerSoft.EventHandlers.*/Services/*Consumer.cs hosts share it: every one creates its connection with zero retry/backoff, and every one carries the same "Fatal error in …" + throw; in ExecuteAsync. Full list:

ArtPipeStageConsumer, ArtStudioMetricsConsumer, BlogMediaMetadataConsumer, BlogMediaMoveConsumer, BlogMediaReceivedConsumer, BlogMediaScanConsumer, BlogMediaStripConsumer, BookManagementConsumer, FileMovementConsumer, GameEventsConsumer, LessonVideoMoveConsumer, LessonVideoReceivedConsumer, LessonVideoScanConsumer, LessonVideoTranscodeConsumer, CoverVariantsConsumer, MetadataExtractionConsumer, PhotographPipelineConsumer, ScheduledTaskConsumer, SecurityScanConsumer, BookCreatedConsumer, FileMovedConsumer, MetadataExtractedConsumer, ScanEventsConsumer, UploadReceivedConsumer.

The original note guessed the others "were quiet during #885 only because none of them restarted" — that is confirmed: they are equally fatal, they just happened not to be rescheduled. So the fix should land as a shared bootstrap helper in SpikerSoft.EventHandlers.Infrastructure applied to all 24, not a one-off patch to ArtPipeProcessor. The 2026-07-29 outage (#895) is the second occurrence of the same class.

Keeping this open — no implementing code exists yet.

Re-verified against `spikersoft-backend` `origin/master` @ 766cb8e4 (2026-07-29) — **still unfixed**, and the requested audit of the other EventHandler hosts is done. Notes updated so nobody has to re-derive this: **ArtPipeProcessor, confirmed:** `SpikerSoft.EventHandlers.ArtPipeProcessor/Services/ArtPipeStageConsumer.cs:133` calls `factory.CreateConnectionAsync(...)` with no retry wrapper. `ExecuteAsync` awaits it at :105 inside a `try` whose `catch (Exception ex)` logs "Fatal error in ArtPipeStageConsumer" and rethrows (:112) — so a broker-down boot is fatal exactly as described. The `AutomaticRecoveryEnabled = true` / `NetworkRecoveryInterval` / `TopologyRecoveryEnabled` settings at :128 are a red herring for this failure mode: the RabbitMQ client only recovers a connection that was established at least once. It does nothing for a failure on the *initial* `CreateConnectionAsync`. **Audit result — the pattern is fleet-wide, not ArtPipeProcessor-specific.** All **24** `SpikerSoft.EventHandlers.*/Services/*Consumer.cs` hosts share it: every one creates its connection with zero retry/backoff, and every one carries the same `"Fatal error in …"` + `throw;` in `ExecuteAsync`. Full list: ArtPipeStageConsumer, ArtStudioMetricsConsumer, BlogMediaMetadataConsumer, BlogMediaMoveConsumer, BlogMediaReceivedConsumer, BlogMediaScanConsumer, BlogMediaStripConsumer, BookManagementConsumer, FileMovementConsumer, GameEventsConsumer, LessonVideoMoveConsumer, LessonVideoReceivedConsumer, LessonVideoScanConsumer, LessonVideoTranscodeConsumer, CoverVariantsConsumer, MetadataExtractionConsumer, PhotographPipelineConsumer, ScheduledTaskConsumer, SecurityScanConsumer, BookCreatedConsumer, FileMovedConsumer, MetadataExtractedConsumer, ScanEventsConsumer, UploadReceivedConsumer. The original note guessed the others "were quiet during #885 only because none of them restarted" — that is confirmed: they are equally fatal, they just happened not to be rescheduled. So the fix should land as a **shared bootstrap helper** in `SpikerSoft.EventHandlers.Infrastructure` applied to all 24, not a one-off patch to ArtPipeProcessor. The 2026-07-29 outage (#895) is the second occurrence of the same class. Keeping this open — no implementing code exists yet.

spikerj referenced this issue

2026-07-29 18:34:31 +00:00

[Bug] SonarQube S7059: async operation inside a constructor #242

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#886