art_pipe observability #1 (.NET): propagate W3C traceparent into the worker + per-stage child span #429

New Issue

2026-07-06T02:25:54Z

spikerj commented

2026-07-06 02:25:54 +00:00

Part of #428. First workstream — self-contained, backend-mergeable, and a no-op until #2 (art_pipe OTel) consumes it, so it can land immediately.

Scope (spikersoft-backend / SpikerSoft.EventHandlers.ArtPipeProcessor)

Inject the current trace context into every job handed to the worker, using the per-job stdin spec as the carrier (mirrors how ImageDescription.Python puts _traceparent in the message dict):
- In the spec built for the worker (SubprocessArtPipeStageExecutor ~line 61, and the resident path via ServeWorkerProcess/ResidentArtPipeStageExecutor), add ["_traceparent"] = Activity.Current?.Id (+ _tracestate when present).
- Also export TRACEPARENT in ApplyWorkerEnvironment as a belt-and-suspenders carrier for tools that read env.
- Per-job, not per-process: the resident serve worker (#368) is long-lived and serves many jobs — the traceparent must be set on each job line, not once at process spawn.
Add a per-stage child span in the executor around the actual worker call (child of the ProcessArtAssetStage consumer span), tagged with asset_id, stage, stage_run_id, job_id, model backend, and executor mode (subprocess/resident). Today the subprocess call runs inside the consumer span with no dedicated child — this gives the stage its own timed span even before art_pipe reports internal spans.

Notes / traps

Guard for Activity.Current == null (BypassGpuLease/fake-executor/dev paths) — emit the job without a traceparent rather than throwing.
Keep the span open across the async worker round-trip; close on the result contract (success/failure) with ActivityStatusCode.

Acceptance

Worker job spec + subprocess env carry a valid W3C traceparent for every stage in both executor modes.
Jaeger shows a per-stage child span under ProcessArtAssetStage.
Unit test asserts the spec/env carry Activity.Current.Id when an activity is active, and omit it cleanly when none is.

Refs #428, #368.

Part of #428. **First workstream — self-contained, backend-mergeable, and a no-op until #2 (art_pipe OTel) consumes it, so it can land immediately.** ## Scope (spikersoft-backend / SpikerSoft.EventHandlers.ArtPipeProcessor) 1. **Inject the current trace context into every job** handed to the worker, using the per-job stdin spec as the carrier (mirrors how ImageDescription.Python puts `_traceparent` in the message dict): - In the spec built for the worker (`SubprocessArtPipeStageExecutor` ~line 61, and the resident path via `ServeWorkerProcess`/`ResidentArtPipeStageExecutor`), add `["_traceparent"] = Activity.Current?.Id` (+ `_tracestate` when present). - Also export `TRACEPARENT` in `ApplyWorkerEnvironment` as a belt-and-suspenders carrier for tools that read env. - **Per-job, not per-process:** the resident serve worker (#368) is long-lived and serves many jobs — the traceparent must be set on each job line, not once at process spawn. 2. **Add a per-stage child span** in the executor around the actual worker call (child of the `ProcessArtAssetStage` consumer span), tagged with `asset_id`, `stage`, `stage_run_id`, `job_id`, model backend, and executor mode (subprocess/resident). Today the subprocess call runs inside the consumer span with no dedicated child — this gives the stage its own timed span even before art_pipe reports internal spans. ## Notes / traps - Guard for `Activity.Current == null` (BypassGpuLease/fake-executor/dev paths) — emit the job without a traceparent rather than throwing. - Keep the span open across the async worker round-trip; close on the result contract (success/failure) with `ActivityStatusCode`. ## Acceptance - Worker job spec + subprocess env carry a valid W3C `traceparent` for every stage in both executor modes. - Jaeger shows a per-stage child span under `ProcessArtAssetStage`. - Unit test asserts the spec/env carry `Activity.Current.Id` when an activity is active, and omit it cleanly when none is. Refs #428, #368.

spikerj referenced this issue

2026-07-06 02:26:53 +00:00

Epic: End-to-end trace + log correlation for the art_pipe native-microservice integration (Jaeger + Seq) #428

spikerj commented

2026-07-06 02:37:45 +00:00

Resolved in spikersoft-backend PR #109 (merged to master). Added WorkerTraceContext.Inject, stamping the active W3C traceparent/tracestate onto the per-job worker spec in both executor modes (subprocess spec + resident per-job request). Per-job payload chosen over an env var because the resident serve worker multiplexes many jobs over one process. 3/3 unit tests pass; SpikerSoft.Tests.Unit builds clean. Inert until #430 consumes the context. Closing — next: #430 (art_pipe OTel spans).

Resolved in spikersoft-backend PR #109 (merged to `master`). Added `WorkerTraceContext.Inject`, stamping the active W3C `traceparent`/`tracestate` onto the per-job worker spec in both executor modes (subprocess spec + resident per-job request). Per-job payload chosen over an env var because the resident serve worker multiplexes many jobs over one process. 3/3 unit tests pass; `SpikerSoft.Tests.Unit` builds clean. Inert until #430 consumes the context. Closing — next: #430 (art_pipe OTel spans).

spikerj closed this issue

2026-07-06 02:37:56 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#429