[Bug][Backend][Scheduler] ScheduledTaskPublisher does not propagate W3C trace context on publish — violates AGENTS.md non-negotiable #1; scheduled-task work is untraceable end-to-end #575

New Issue

2026-07-14T14:59:15Z

spikerj commented

2026-07-14 14:59:15 +00:00

QA Team — filed 2026-07-14 from the README audit, at spikerj's direction (decision: file, fix the propagation).

The bug: SpikerSoft.EventHandlers.Scheduler's ScheduledTaskPublisher publishes to RabbitMQ without injecting W3C trace context via ActivityHelper — it sets only its own message properties. AGENTS.md non-negotiable #1 states: "every RabbitMQ publish/consume must propagate W3C trace context via ActivityHelper". Every other publisher in the codebase complies (the audit checked them).

Impact: every downstream unit of work kicked off by a scheduled task (GeoIP updates and whatever else the Scheduler drives) starts a fresh, orphaned trace in Jaeger. There is no way to answer "what did last night's scheduled run actually do, and where did it fail" by trace — you have to reconstruct it from logs. Scheduled work is precisely the category nobody is watching in real time, so it's the category that most needs a trace.

Fix shape: start an Activity for the scheduled trigger and inject its context into the AMQP BasicProperties headers via ActivityHelper exactly as the other publishers do; the consumers already extract with ActivityHelper.StartActivityFromRabbitMQMessage, so this single change closes the loop.

Context: the Scheduler was historically built and deployed nowhere (#540, now being wired into CI) — so it is about to start running for real, which makes this the right moment to fix its telemetry before anyone depends on it.

**QA Team** — filed 2026-07-14 from the README audit, at spikerj's direction (decision: file, fix the propagation). **The bug:** `SpikerSoft.EventHandlers.Scheduler`'s `ScheduledTaskPublisher` publishes to RabbitMQ **without injecting W3C trace context via `ActivityHelper`** — it sets only its own message properties. AGENTS.md **non-negotiable #1** states: *"every RabbitMQ publish/consume must propagate W3C trace context via ActivityHelper"*. Every other publisher in the codebase complies (the audit checked them). **Impact:** every downstream unit of work kicked off by a scheduled task (GeoIP updates and whatever else the Scheduler drives) starts a **fresh, orphaned trace** in Jaeger. There is no way to answer "what did last night's scheduled run actually do, and where did it fail" by trace — you have to reconstruct it from logs. Scheduled work is precisely the category nobody is watching in real time, so it's the category that most needs a trace. **Fix shape:** start an `Activity` for the scheduled trigger and inject its context into the AMQP `BasicProperties` headers via `ActivityHelper` exactly as the other publishers do; the consumers already extract with `ActivityHelper.StartActivityFromRabbitMQMessage`, so this single change closes the loop. **Context:** the Scheduler was historically built and deployed nowhere (#540, now being wired into CI) — so it is about to start running for real, which makes this the right moment to fix its telemetry before anyone depends on it.

spikerj commented

2026-07-14 20:30:55 +00:00

Fix in backend PR #285. Diagnosis was right. Two corrections to the fix shape, both of which would have bitten:

1. "Inject via ActivityHelper" alone would have been a no-op here. InjectTraceContext reads Activity.Current and returns silently when there is none:

activity ??= Activity.Current;
if (activity == null) return;      // ← injects nothing

Every other publisher gets away with a bare InjectTraceContext(props) because it runs under an ambient trace (an API turn, a consumer handler). A scheduled task dispatched from the Scheduler's own timer has no ambient Activity at all — so the "compliant" call would have shipped no traceparent and left the code looking instrumented. Same silent-failure shape as the #307 Jaeger:EndPoint audit. The publisher now starts a real Producer span. And because an ActivitySource with no listener also yields a null Activity, TraceSources.Scheduler is registered unconditionally by both hosts on this path (the API, which publishes via the create/trigger/resume handlers, and the Scheduler worker).

2. The consumers do NOT already extract. ScheduledTaskConsumer had no ActivitySource and never called StartActivityFromRabbitMQMessage — the loop was open at both ends. Fixing only the publisher would have produced a properly-tagged message that the consumer then ignored, still rooting a fresh trace per task. It now starts a Consumer span from the headers and scopes its logs to the correlation context.

Tests: 4 xUnit, and I confirmed all 4 fail on master (stashed the production change, watched them go red, restored). They cover the timer path with no ambient Activity, the API path inheriting the request's TraceId, the loop closing (a consumer span rebuilt from the published properties lands in the publisher's trace and hangs off the producer span), and — the one that worries me most — that trace injection does not clobber the x-delay header it shares a dictionary with. Losing x-delay would silently turn every delayed task into an immediate one, which would be a much worse bug than the one being fixed.

Good call on the timing: with #540 wiring the Scheduler into CI, this lands before anything depends on its telemetry.

Leaving open for close after merge.

**Fix in backend PR #285.** Diagnosis was right. Two corrections to the fix shape, both of which would have bitten: **1. "Inject via ActivityHelper" alone would have been a no-op here.** `InjectTraceContext` reads `Activity.Current` and returns silently when there is none: ```csharp activity ??= Activity.Current; if (activity == null) return; // ← injects nothing ``` Every other publisher gets away with a bare `InjectTraceContext(props)` because it runs under an ambient trace (an API turn, a consumer handler). **A scheduled task dispatched from the Scheduler's own timer has no ambient Activity at all** — so the "compliant" call would have shipped no `traceparent` and left the code looking instrumented. Same silent-failure shape as the #307 `Jaeger:EndPoint` audit. The publisher now starts a real Producer span. And because an `ActivitySource` with no listener also yields a null Activity, `TraceSources.Scheduler` is registered unconditionally by both hosts on this path (the API, which publishes via the create/trigger/resume handlers, and the Scheduler worker). **2. The consumers do NOT already extract.** `ScheduledTaskConsumer` had no `ActivitySource` and never called `StartActivityFromRabbitMQMessage` — the loop was open at **both** ends. Fixing only the publisher would have produced a properly-tagged message that the consumer then ignored, still rooting a fresh trace per task. It now starts a Consumer span from the headers and scopes its logs to the correlation context. **Tests:** 4 xUnit, and I confirmed **all 4 fail on `master`** (stashed the production change, watched them go red, restored). They cover the timer path with no ambient Activity, the API path inheriting the request's TraceId, the loop closing (a consumer span rebuilt from the published properties lands in the publisher's trace and hangs off the producer span), and — the one that worries me most — that trace injection does **not** clobber the `x-delay` header it shares a dictionary with. Losing `x-delay` would silently turn every delayed task into an immediate one, which would be a much worse bug than the one being fixed. Good call on the timing: with #540 wiring the Scheduler into CI, this lands before anything depends on its telemetry. Leaving open for close after merge.

spikerj commented

2026-07-14 20:47:54 +00:00

Resolved in spikersoft-backend PR #285 (merged to master as f3f5166). ScheduledTaskPublisher now starts a real Producer span and injects W3C trace context; ScheduledTaskConsumer extracts it (it never did — the loop was open at both ends); and TraceSources.Scheduler is registered unconditionally by both the API and the Scheduler worker, since an unlistened ActivitySource would have made the injection a silent no-op. Closing.

Resolved in spikersoft-backend PR #285 (merged to `master` as `f3f5166`). `ScheduledTaskPublisher` now starts a real Producer span and injects W3C trace context; `ScheduledTaskConsumer` extracts it (it never did — the loop was open at both ends); and `TraceSources.Scheduler` is registered unconditionally by both the API and the Scheduler worker, since an unlistened `ActivitySource` would have made the injection a silent no-op. Closing.

spikerj closed this issue

2026-07-14 20:48:02 +00:00

spikerj referenced this issue

2026-07-14 21:04:47 +00:00

[Bug][Telemetry] Every service reports service.namespace='1.0.0' — AddService()'s 2nd positional arg is serviceNamespace, not serviceVersion. Spans also carry no node identity #588

spikerj referenced this issue

2026-07-14 21:31:22 +00:00

[Bug][Backend][Observability] #575 was fixed in ONE publisher — BookController (5 publish sites) and 3 others still ship RabbitMQ messages with no traceparent, so user-initiated flows appear in Jaeger as disconnected orphan traces #597

spikerj referenced this issue

2026-07-14 21:35:20 +00:00

[Bug][CI][Deploy] cancel-in-progress on master kills DEPLOY jobs — back-to-back merges cancel each other and fixes silently never ship #590

spikerj referenced this issue