Assignees
spikerj (Joseph Spiker)
Clear assignees
spikerj (Joseph Spiker)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: spikerj/spikersoft-issues#460
Reference in New Issue
Block a user
Follow-up split from #459. With #459 the .NET side sets
JAEGER_ENDPOINTon the worker subprocess, butartpipe.telemetry.init_worker_tracing()only activates whenopentelemetryis importable in the venv that runs-m artpipe.worker. art_pipe is volume-mounted, not vendored into the ArtPipeProcessor image, and each model runs in its own bootstrapped venv — so the OTel packages must be installed into those venvs.Scope (art_pipe
venv_setup/bootstrap)Install the
telemetryextra (opentelemetry-sdk,opentelemetry-exporter-otlp-proto-grpc— already declared asartpipe[telemetry]inpyproject.toml) into the worker venvs during bootstrap.Decision needed (why this isn't auto-done)
PythonFallback/ a shared base? Per-model venvs are dependency-pinned (torch/CUDA); adding two pure-Python packages is low-risk but multiplies install work across N venvs.ARTPIPE_INSTALL_TELEMETRY) vs. always. Spans are a no-op at runtime unlessJAEGER_ENDPOINTis set, so "always install" is safe but adds bootstrap time.Note: #431 (Seq log shipping) needs nothing here — it's stdlib-only (
urllib), soSEQ_URLfrom #459 already activates it. This ticket only unblocks #430's span export.Acceptance
import opentelemetryand, withJAEGER_ENDPOINTset, exports spans to Jaeger (verified end-to-end: API → bus → .NET stage → art_pipe worker + model spans in one trace).Refs #428, #430, #459.
Status: the implementing PR merged today — spikersoft-artpipe PR #9 'Install OTel packages into model venvs behind opt-in flag' — which also answers this ticket's open decision (opt-in via flag, all venvs when enabled). Remaining before close is the acceptance step: re-bootstrap the model venvs with the flag enabled on the GPU host, then verify one end-to-end trace (API → bus → .NET stage → art_pipe worker spans in Jaeger). Ops synergy: #486 already requires a SafetyCheck venv re-bootstrap on the same host — enable the telemetry flag in that same session and both tickets' host work collapses into one pass.
Board-sweep status (2026-07-22): implementation merged (spikersoft-artpipe #9 — OTel into venvs, opt-in). REMAINING: acceptance only — re-bootstrap venvs on the GPU host + verify one end-to-end trace.
Activation shipped for the baked-image lane — spikersoft-artpipe PR #30 (merged):
Dockerfile.model-envnow setsARTPIPE_INSTALL_TELEMETRY=1so every Layer-2 venv build installs the OTel packages (the PR #9 opt-in), and the Layer-4 root venv gets the[telemetry]extra (env-less/Blender models trace through it, mirroringensure_server_venv). Runtime remains gated onJAEGER_ENDPOINT, whichSubprocessArtPipeStageExecutoralready injects into every worker subprocess fromJaeger:EndPoint— verified end-to-end in code, so spans export as soon as rebuilt images roll.Env-image rebuild is running now (
model-env-imagesrun 12051, safetycheck+prodstages set). Remaining to close:spikersoft-artpipe-processorworkflow so the tier-3 finals rebuild on the new env base and deploy — watch the #717 supersede/cancel trap; verify the finals actually reached the registry./opt/art_pipeon the GPU host): one re-bootstrap with the flag —ARTPIPE_INSTALL_TELEMETRY=1 python bootstrap.py(re-running setup upgrades healthy venvs in place, no torch-sized rebuild; can ride the same host session as #486's SafetyCheck re-bootstrap).Rollout status (2026-07-22 ~12:45): tier-2 done and registry-verified — all three OTel-carrying env images are published (
artpipe-model-env-{safetycheck,prodstages,photostack}at artpipe shad15e2ec). Tier-3 finals are NOT yet rebuilt on that base: newest finals in the registry remain0f6e9f0-0a57aeb(pre-OTel). Three dispatch attempts were consumed by environment, not code: (1) run 12130 died mid-push in the #700 Gitea outage (10:03, recurrence documented there), (2) a re-dispatch was lost in the runner queue, (3) run 12999 was mass-cancelled while queued when the coverage-wave PR force-pushed and CI cancelled queued runs — the #717 supersede pattern amplified by ~1400 coverage-wave runs flooding the queue today.Holding further dispatches until the coverage wave's CI storm subsides — a queued dispatch just gets starved or mass-cancelled again. One targeted
workflow_dispatchofspikersoft-artpipe-processor(emptyimagesinput) in a quiet window produces the finals; registry check for*-d15e2ectags is the acceptance. Host-side venv re-bootstrap + Jaeger trace acceptance unchanged.Attempt 4 (run 13851, dispatched in a verified-quiet CI window): base image built+pushed+verified, then the per-model step failed pulling the env base image —
unexpected EOF× 5 attempts, ~19 min each (full evidence on #775). So the tier-3 leg is now hard-blocked on the registry read path, not on CI contention or code; further dispatches are pointless until that changes.Two unblock options (both operator-side):
docker save git.spikersoft.com/spikerj/artpipe-model-env-prodstages:latest | ssh <tier3-runner> docker load(same forenv-safetycheck), then re-dispatch — the pull step no-ops on a cache hit and the build proceeds.Standing state: tier-2 OTel env images published + verified (
*-d15e2ec); tier-3 finals pending the above; host/opt/art_pipere-bootstrap (ARTPIPE_INSTALL_TELEMETRY=1 python bootstrap.py) and the Jaeger end-to-end trace remain the acceptance steps.Audited against artpipe
origin/main— the code shipped; the acceptance criterion has never been reachable. Staying open.Landed (PRs #9
850773b/85bd96eand #30a2f4450/d15e2ec):src/artpipe/venv_setup.py:51TELEMETRY_ENV_FLAG, opt-in logic at:345bootstrap.py:202,208pyproject.toml:17-19[telemetry]extradocker/Dockerfile.model-env:59ENV ARTPIPE_INSTALL_TELEMETRY=1src/artpipe/worker.py:350,394callinginit_worker_tracing()tests/test_venv_setup_telemetry.pyWhy no span has ever reached Jaeger: the tier-3 final images were never rebuilt on the OTel-enabled env base. The 2026-07-22 comment records the newest finals still pinned at the pre-OTel
0f6e9f0-0a57aeb, after four failed dispatch attempts — Gitea outage, queue starvation, mass-cancel, then registryunexpected EOFpull failures. That last one is #775, which is still open, so this is hard-blocked on it rather than on anything in this ticket's own scope.Remaining:
/opt/art_pipewithARTPIPE_INSTALL_TELEMETRY=1.*-d15e2ectier-3 tags, plus a single trace containing worker spans.Worth noting the dependency chain explicitly, since it isn't obvious from this ticket alone: #775 → #460 → the observability payoff. There's no point re-attempting the dispatch until the registry blob-integrity work lands.