spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Art Studio] End-to-end observability for the new ArtPipe worker stage actions (Jaeger spans + correlated Seq logs) #457

New Issue
Closed
opened 2026-07-08 01:55:44 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-08 01:55:44 +00:00
Owner
Copy Link
Copy Source

Problem

The new Art Studio backend pipeline (CQRS submit → RabbitMQ → ArtPipeProcessor consumer → orchestrator → art_pipe subprocess) does not yet give us the same level of debuggability as the original workers. When a stage is slow or fails, we can't easily answer, from Jaeger/Seq alone:

  • which container picked up the message and ran the stage,
  • when/where it went through RabbitMQ and when a consumer took it in,
  • what happened and how long each step took (queue/GPU wait vs. the art_pipe subprocess vs. artifact persistence + safety gate),
  • and pivot across the C# ↔ Python boundary via correlation ids.

The consumer already starts a W3C-correlated ProcessArtAssetStage span (per the telemetry-correlation rule), but the orchestrator emits no per-step spans and its Seq log lines don't all carry the asset/stage/run/correlation ids, so a single failing run is hard to isolate.

Scope (this ticket — backend, no cross-repo/cross-cutting decision)

Bounded, verifiable, decision-free slice inside SpikerSoft.EventHandlers.ArtPipeProcessor:

  • P2 — per-step Jaeger spans under the existing consumer span (reusing the already-registered SpikerSoft.EventHandlers.ArtPipeProcessor ActivitySource, zero tracer-config change):
    • artpipe.gpu.lease — the GPU/queue wait bucket,
    • artpipe.stage.execute — end-to-end executing stage (tags: asset id, stage, model, action, run id), with error status set on failure paths,
    • artpipe.worker.execute — the art_pipe subprocess itself,
    • artpipe.outputs.persist — GridFS upload + output-side safety gate.
  • P3 — correlated Seq logs: push AssetId / Stage / StageRunId / CorrelationId into the Serilog LogContext around the orchestrator dispatch so every downstream log line is filterable (TraceId is already enriched from the active Activity; ServiceName/DeploymentName/MachineName already answer "which container"). Add ProcessingTimeMs to the two stage-completion log lines.

Out of scope (tracked separately — each needs a decision or another repo)

  • P1 — port the OTLP + _traceparent consumer-span pattern from image_description_service.py into art_pipe's worker.py (separate art_pipe repo, adds opentelemetry dep) — the biggest single win for true cross-language traces; the C# side already injects _traceparent via WorkerTraceContext.Inject.
  • P4 — register the dropped SpikerSoft.MessageBus / SpikerSoft.Api ActivitySources with the tracer (global span volume).
  • P5 — per-stack Deployment__Name differentiation in the infra stack files (mostly covered already by the existing DeploymentName/MachineName/ServiceName Serilog enrichers).

Part of the ProArt Art Studio epic (#346); follows the telemetry-correlation conventions (Jaeger:EndPoint, ActivityHelper, jaeger/seq-attachable overlays).

## Problem The new Art Studio backend pipeline (CQRS submit → RabbitMQ → `ArtPipeProcessor` consumer → orchestrator → art_pipe subprocess) does not yet give us the same level of debuggability as the original workers. When a stage is slow or fails, we can't easily answer, from Jaeger/Seq alone: - **which container** picked up the message and ran the stage, - **when/where** it went through RabbitMQ and when a consumer took it in, - **what happened and how long** each step took (queue/GPU wait vs. the art_pipe subprocess vs. artifact persistence + safety gate), - and pivot across the **C# ↔ Python** boundary via correlation ids. The consumer already starts a W3C-correlated `ProcessArtAssetStage` span (per the `telemetry-correlation` rule), but the orchestrator emits no per-step spans and its Seq log lines don't all carry the asset/stage/run/correlation ids, so a single failing run is hard to isolate. ## Scope (this ticket — backend, no cross-repo/cross-cutting decision) Bounded, verifiable, decision-free slice inside `SpikerSoft.EventHandlers.ArtPipeProcessor`: - **P2 — per-step Jaeger spans** under the existing consumer span (reusing the already-registered `SpikerSoft.EventHandlers.ArtPipeProcessor` ActivitySource, zero tracer-config change): - `artpipe.gpu.lease` — the GPU/queue wait bucket, - `artpipe.stage.execute` — end-to-end executing stage (tags: asset id, stage, model, action, run id), with error status set on failure paths, - `artpipe.worker.execute` — the art_pipe subprocess itself, - `artpipe.outputs.persist` — GridFS upload + output-side safety gate. - **P3 — correlated Seq logs**: push `AssetId` / `Stage` / `StageRunId` / `CorrelationId` into the Serilog `LogContext` around the orchestrator dispatch so **every** downstream log line is filterable (TraceId is already enriched from the active Activity; `ServiceName`/`DeploymentName`/`MachineName` already answer "which container"). Add `ProcessingTimeMs` to the two stage-completion log lines. ## Out of scope (tracked separately — each needs a decision or another repo) - **P1** — port the OTLP + `_traceparent` consumer-span pattern from `image_description_service.py` into `art_pipe`'s `worker.py` (separate `art_pipe` repo, adds `opentelemetry` dep) — the biggest single win for true cross-language traces; the C# side already injects `_traceparent` via `WorkerTraceContext.Inject`. - **P4** — register the dropped `SpikerSoft.MessageBus` / `SpikerSoft.Api` ActivitySources with the tracer (global span volume). - **P5** — per-stack `Deployment__Name` differentiation in the infra stack files (mostly covered already by the existing `DeploymentName`/`MachineName`/`ServiceName` Serilog enrichers). Part of the ProArt Art Studio epic (#346); follows the `telemetry-correlation` conventions (`Jaeger:EndPoint`, `ActivityHelper`, `jaeger`/`seq-attachable` overlays).
spikerj commented 2026-07-08 01:57:16 +00:00
Author
Owner
Copy Link
Copy Source

P2 + P3 (the backend, decision-free slice) opened for review as spikersoft-backend PR #182 (feat/artpipe-worker-observability-457 → master), awaiting merge.

  • Per-step Jaeger spans (artpipe.gpu.lease / stage.execute / worker.execute / outputs.persist) nested under the existing consumer span via the already-registered ActivitySource — zero tracer-config change.
  • AssetId/Stage/StageRunId/CorrelationId pushed into the Serilog LogContext so every orchestrator log line is filterable in Seq; {ProcessingTimeMs} added to the two completion logs.
  • Build clean, 209 ArtPipe unit tests pass.

P1 (art_pipe worker.py OTLP for true C#↔Python spans), P4 (MessageBus/Api ActivitySource registration), and P5 (per-stack Deployment name) remain open here — each needs a decision or the art_pipe/infrastructure repo. Will close this ticket once #182 merges and note the residual P1/P4/P5 as follow-ups.

P2 + P3 (the backend, decision-free slice) opened for review as **spikersoft-backend PR #182** (`feat/artpipe-worker-observability-457` → `master`), awaiting merge. - Per-step Jaeger spans (`artpipe.gpu.lease` / `stage.execute` / `worker.execute` / `outputs.persist`) nested under the existing consumer span via the already-registered ActivitySource — zero tracer-config change. - `AssetId`/`Stage`/`StageRunId`/`CorrelationId` pushed into the Serilog `LogContext` so every orchestrator log line is filterable in Seq; `{ProcessingTimeMs}` added to the two completion logs. - Build clean, 209 ArtPipe unit tests pass. P1 (art_pipe `worker.py` OTLP for true C#↔Python spans), P4 (MessageBus/Api ActivitySource registration), and P5 (per-stack Deployment name) remain open here — each needs a decision or the `art_pipe`/infrastructure repo. Will close this ticket once #182 merges and note the residual P1/P4/P5 as follow-ups.
spikerj commented 2026-07-08 01:59:28 +00:00
Author
Owner
Copy Link
Copy Source

Resolved by spikersoft-backend PR #182 (merged to master, HEAD 1638bf0). The P2 per-step Jaeger spans and P3 correlated Seq logs + ProcessingTimeMs are in — the new Art Studio stage worker now has span-level "how long / where / what happened" and asset/stage/run/correlation-id filtering, nesting under the existing consumer span. Build clean, 209 ArtPipe unit tests pass.

The remaining P1 (art_pipe worker.py OTLP for true C#↔Python spans), P4 (MessageBus/Api ActivitySource registration), and P5 (per-stack Deployment name) are decision-gated / cross-repo and moved to #458. Closing.

Resolved by spikersoft-backend **PR #182** (merged to `master`, HEAD `1638bf0`). The P2 per-step Jaeger spans and P3 correlated Seq logs + `ProcessingTimeMs` are in — the new Art Studio stage worker now has span-level "how long / where / what happened" and asset/stage/run/correlation-id filtering, nesting under the existing consumer span. Build clean, 209 ArtPipe unit tests pass. The remaining P1 (art_pipe `worker.py` OTLP for true C#↔Python spans), P4 (MessageBus/Api ActivitySource registration), and P5 (per-stack Deployment name) are decision-gated / cross-repo and moved to **#458**. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#457