spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

art_pipe observability #2 (Python): OpenTelemetry spans with parent-context extraction from the job spec #430

New Issue
Closed
opened 2026-07-06 02:26:09 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-06 02:26:09 +00:00
Owner
Copy Link
Copy Source

Part of #428. Depends on #1 (which supplies the _traceparent on each job).

Scope (art_pipe)

  1. Add the OTel SDK (optional dependency; degrade gracefully when absent so the stdlib-only orchestrator/serve path still runs): opentelemetry-sdk, opentelemetry-exporter-otlp-proto-grpc. Init a TracerProvider + Resource(SERVICE_NAME="art_pipe.worker") + BatchSpanProcessor(OTLPSpanExporter(endpoint=os.getenv("JAEGER_ENDPOINT","http://jaeger:4317"), insecure=True)).
  2. Extract the parent context from each job's _traceparent (+ _tracestate) using TraceContextTextMapPropagator().extract(carrier=...) and start_as_current_span(..., context=parent_context) — per job in the serve loop (worker.py / persistent_worker.py), so the resident worker parents each job correctly.
  3. Span-per-stage/model-call: wrap the dispatcher stage execution and each model backend run() (SDXL/TripoSR/TRELLIS-mac/Blender/etc.) in child spans, tagged with model name, device (cuda/mps/cpu via device.resolve), and timing. Blender subprocess renders get their own span.

Reuse the reference implementation (do not reinvent)

SpikerSoft.EventHandlers.ImageDescription.Python/image_description_service.py + Trellis3D.Python — copy the wiring and the already-solved gotchas:

  • "invalid parent span" race: set a short BatchSpanProcessor schedule_delay_millis (~1000ms) — the 5000ms default exports the Python child before the .NET parent lands, orphaning it in Jaeger.
  • traceparent may arrive as bytes → decode utf-8.
  • Metrics disabled (Jaeger OTLP = traces only).

Coordination

art_pipe is a separate checkout (Eric's pipeline). Keep OTel strictly additive and optional — no import-time hard dependency, no behavior change when JAEGER_ENDPOINT is unset. This preserves the WRAP-first posture and the stdlib-only server.py invariant.

Acceptance

  • With #1 merged and the stack up, a single Jaeger trace shows API → bus → .NET stage span → art_pipe stage + model spans, correctly parented (no orphans).
  • No regression / no new hard dependency when OTel env is absent.

Refs #428, #358 (trellis-mac), #368.

Part of #428. Depends on #1 (which supplies the `_traceparent` on each job). ## Scope (art_pipe) 1. **Add the OTel SDK** (optional dependency; degrade gracefully when absent so the stdlib-only orchestrator/serve path still runs): `opentelemetry-sdk`, `opentelemetry-exporter-otlp-proto-grpc`. Init a `TracerProvider` + `Resource(SERVICE_NAME="art_pipe.worker")` + `BatchSpanProcessor(OTLPSpanExporter(endpoint=os.getenv("JAEGER_ENDPOINT","http://jaeger:4317"), insecure=True))`. 2. **Extract the parent context** from each job's `_traceparent` (+ `_tracestate`) using `TraceContextTextMapPropagator().extract(carrier=...)` and `start_as_current_span(..., context=parent_context)` — **per job** in the serve loop (`worker.py` / `persistent_worker.py`), so the resident worker parents each job correctly. 3. **Span-per-stage/model-call**: wrap the dispatcher stage execution and each model backend `run()` (SDXL/TripoSR/TRELLIS-mac/Blender/etc.) in child spans, tagged with model name, device (cuda/mps/cpu via `device.resolve`), and timing. Blender subprocess renders get their own span. ## Reuse the reference implementation (do not reinvent) `SpikerSoft.EventHandlers.ImageDescription.Python/image_description_service.py` + `Trellis3D.Python` — copy the wiring and the **already-solved gotchas**: - **"invalid parent span" race:** set a short `BatchSpanProcessor` `schedule_delay_millis` (~1000ms) — the 5000ms default exports the Python child before the .NET parent lands, orphaning it in Jaeger. - `traceparent` may arrive as `bytes` → decode utf-8. - Metrics disabled (Jaeger OTLP = traces only). ## Coordination art_pipe is a separate checkout (Eric's pipeline). Keep OTel **strictly additive and optional** — no import-time hard dependency, no behavior change when `JAEGER_ENDPOINT` is unset. This preserves the WRAP-first posture and the stdlib-only `server.py` invariant. ## Acceptance - With #1 merged and the stack up, a single Jaeger trace shows API → bus → .NET stage span → **art_pipe stage + model spans**, correctly parented (no orphans). - No regression / no new hard dependency when OTel env is absent. Refs #428, #358 (trellis-mac), #368.
spikerj commented 2026-07-08 02:25:31 +00:00
Author
Owner
Copy Link
Copy Source

Implementing PR up for review: spikersoft-artpipe PR #7 — spikerj/spikersoft-artpipe#7

Wires the Python side to consume the _traceparent the .NET ArtPipeProcessor (WorkerTraceContext) already injects into each per-job stdin dict:

  • New artpipe.telemetry module — optional OTLP→Jaeger tracing, strict no-op unless JAEGER_ENDPOINT is set and opentelemetry is installed (keeps the stdlib-only server invariant + per-model-venv isolation; no new hard dependency).
  • worker.py (single-job and --serve): per-job CONSUMER span re-parented from its own _traceparent (so a resident worker parents each job under its own .NET stage), plus nested artpipe.model.run / artpipe.preprocess child spans. Tags model / action / backend / job_id / device.
  • Reference gotchas carried over: 1s batch-export delay (avoids the 5s-default orphan race), bytes→utf-8 traceparent decode, traces-only, force-flush on exit.

Local verification: in-memory-exporter proof shows the artpipe.job span lands in the same trace as the injected _traceparent with parent span id == the .NET span id, and artpipe.model.run nests under it — no orphans, matching the acceptance criterion. 11 new tests + 27 existing worker-protocol tests pass; ruff clean.

Leaving this ticket open until PR #7 merges. Prod activation (install the telemetry extra + set JAEGER_ENDPOINT on the worker subprocess env) is an infra follow-up, not in this PR.

Implementing PR up for review: **spikersoft-artpipe PR #7** — https://git.spikersoft.com/spikerj/spikersoft-artpipe/pulls/7 Wires the Python side to consume the `_traceparent` the .NET `ArtPipeProcessor` (`WorkerTraceContext`) already injects into each per-job stdin dict: - New `artpipe.telemetry` module — optional OTLP→Jaeger tracing, **strict no-op** unless `JAEGER_ENDPOINT` is set *and* `opentelemetry` is installed (keeps the stdlib-only server invariant + per-model-venv isolation; no new hard dependency). - `worker.py` (single-job **and** `--serve`): per-job CONSUMER span re-parented from *its own* `_traceparent` (so a resident worker parents each job under its own .NET stage), plus nested `artpipe.model.run` / `artpipe.preprocess` child spans. Tags model / action / backend / job_id / device. - Reference gotchas carried over: 1s batch-export delay (avoids the 5s-default orphan race), bytes→utf-8 traceparent decode, traces-only, force-flush on exit. **Local verification:** in-memory-exporter proof shows the `artpipe.job` span lands in the same trace as the injected `_traceparent` with parent span id == the .NET span id, and `artpipe.model.run` nests under it — **no orphans**, matching the acceptance criterion. 11 new tests + 27 existing worker-protocol tests pass; ruff clean. Leaving this ticket **open** until PR #7 merges. Prod activation (install the `telemetry` extra + set `JAEGER_ENDPOINT` on the worker subprocess env) is an infra follow-up, not in this PR.
spikerj commented 2026-07-08 04:11:51 +00:00
Author
Owner
Copy Link
Copy Source

Resolved in spikersoft-artpipe PR #7 (merged to main as 5797f86). The art_pipe worker now consumes the W3C _traceparent the .NET ArtPipeProcessor injects and emits per-job OTel spans (CONSUMER artpipe.job.* re-parented from the job's own traceparent, nested artpipe.model.run / artpipe.preprocess), so a single Jaeger trace spans API → RabbitMQ → .NET stage → art_pipe worker + model steps. Strictly additive/optional (artpipe.telemetry, no-op unless JAEGER_ENDPOINT set + OTel installed). Prod activation (install the telemetry extra + set JAEGER_ENDPOINT on the worker subprocess env) tracked as infra follow-up. Closing.

Resolved in spikersoft-artpipe **PR #7** (merged to `main` as `5797f86`). The art_pipe worker now consumes the W3C `_traceparent` the .NET `ArtPipeProcessor` injects and emits per-job OTel spans (CONSUMER `artpipe.job.*` re-parented from the job's own traceparent, nested `artpipe.model.run` / `artpipe.preprocess`), so a single Jaeger trace spans API → RabbitMQ → .NET stage → art_pipe worker + model steps. Strictly additive/optional (`artpipe.telemetry`, no-op unless `JAEGER_ENDPOINT` set + OTel installed). Prod activation (install the `telemetry` extra + set `JAEGER_ENDPOINT` on the worker subprocess env) tracked as infra follow-up. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#430