spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

art_pipe observability #3 (Python): ship worker logs to Seq + stamp trace_id for log↔trace correlation #431

New Issue
Closed
opened 2026-07-06 02:26:16 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-06 02:26:16 +00:00
Owner
Copy Link
Copy Source

Part of #428. Complements #2 (once spans exist, close the log side).

Scope (art_pipe)

Today logging_config.py writes only a local logs/debug.log (RotatingFileHandler) with a thread-local asset_id/job_id context — those lines never reach Seq, so a Jaeger trace can't be pivoted to the worker's internal logs.

  1. Stamp trace_id / span_id into the log context (extend the existing thread-local correlation context) so every worker log line is tagged with the active trace — the minimum needed for trace↔log correlation.
  2. Ship to Seq — add an optional CLEF/GELF-style Seq handler pointed at SEQ_URL (default the compose http://seq:5341), gated on env like the OTel exporter in #2 (no-op when unset). Structured properties: asset_id, job_id, stage, stage_run_id, trace_id, model, device.
    • Alternative considered: have the .NET executor forward parsed worker stdout lines to Seq. Rejected as lossy (only surfaces what the result contract emits, not internal DEBUG). Direct shipping keeps full fidelity.

Acceptance

  • In Seq, filtering by an asset's trace_id returns both the .NET ArtPipeProcessor logs and the art_pipe worker's internal logs for that run.
  • Local debug.log behavior unchanged; Seq shipping is additive and env-gated.

Refs #428.

Part of #428. Complements #2 (once spans exist, close the log side). ## Scope (art_pipe) Today `logging_config.py` writes only a local `logs/debug.log` (RotatingFileHandler) with a thread-local `asset_id/job_id` context — those lines never reach Seq, so a Jaeger trace can't be pivoted to the worker's internal logs. 1. **Stamp `trace_id` / `span_id`** into the log context (extend the existing thread-local correlation context) so every worker log line is tagged with the active trace — the minimum needed for trace↔log correlation. 2. **Ship to Seq** — add an optional CLEF/GELF-style Seq handler pointed at `SEQ_URL` (default the compose `http://seq:5341`), gated on env like the OTel exporter in #2 (no-op when unset). Structured properties: `asset_id`, `job_id`, `stage`, `stage_run_id`, `trace_id`, `model`, `device`. - Alternative considered: have the .NET executor forward parsed worker stdout lines to Seq. Rejected as lossy (only surfaces what the result contract emits, not internal DEBUG). Direct shipping keeps full fidelity. ## Acceptance - In Seq, filtering by an asset's `trace_id` returns **both** the .NET ArtPipeProcessor logs and the art_pipe worker's internal logs for that run. - Local `debug.log` behavior unchanged; Seq shipping is additive and env-gated. Refs #428.
spikerj commented 2026-07-08 02:35:27 +00:00
Author
Owner
Copy Link
Copy Source

Implementing PR up for review: spikersoft-artpipe PR #8 — spikerj/spikersoft-artpipe#8

Delivers both acceptance items:

  1. Stamp trace_id/span_id — read per-record from the active OTel span (guarded), so a worker log line is tagged with the active trace even for a resident --serve worker. Plus a new thread-local structured-property context (asset_id/job_id/stage/stage_run_id/model/device).
  2. Ship to Seq — new artpipe.seq_logging: stdlib-only CLEF→Seq handler over urllib, env-gated on SEQ_URL (no-op when unset), non-blocking (daemon-thread batch POST, swallows network errors, drops on overflow — never crashes a model job). Chose direct shipping over .NET-forwarded stdout, exactly as the ticket's rejected-alternative note prefers (full DEBUG fidelity).

Local debug.log unchanged; Seq shipping is additive + env-gated — matches acceptance. SEQ_API_KEY honored if set.

Independent of #430 (PR #7), mergeable in either order — worker.py change is a single hunk in _configure_paths_and_debug_log, disjoint from #430's edits; logging_config.py/seq_logging.py untouched by #430. Runtime coupling only: trace_id populates when a span is active (#430); merged first, logs still ship with job_id/model correlation.

Verification: 11 new tests (incl. mocked-POST CLEF payload assertion + active-span trace-id stamping) + 27 existing worker-protocol tests pass; ruff clean.

Leaving open until PR #8 merges. Prod activation (set SEQ_URL + join seq-attachable network on the worker) is an infra follow-up, same pattern as #430's JAEGER_ENDPOINT.

Implementing PR up for review: **spikersoft-artpipe PR #8** — https://git.spikersoft.com/spikerj/spikersoft-artpipe/pulls/8 Delivers both acceptance items: 1. **Stamp `trace_id`/`span_id`** — read per-record from the active OTel span (guarded), so a worker log line is tagged with the active trace even for a resident `--serve` worker. Plus a new thread-local **structured-property** context (`asset_id`/`job_id`/`stage`/`stage_run_id`/`model`/`device`). 2. **Ship to Seq** — new `artpipe.seq_logging`: stdlib-only CLEF→Seq handler over `urllib`, **env-gated on `SEQ_URL`** (no-op when unset), non-blocking (daemon-thread batch POST, swallows network errors, drops on overflow — never crashes a model job). Chose direct shipping over .NET-forwarded stdout, exactly as the ticket's rejected-alternative note prefers (full DEBUG fidelity). **Local `debug.log` unchanged; Seq shipping is additive + env-gated** — matches acceptance. `SEQ_API_KEY` honored if set. **Independent of #430 (PR #7), mergeable in either order** — worker.py change is a single hunk in `_configure_paths_and_debug_log`, disjoint from #430's edits; `logging_config.py`/`seq_logging.py` untouched by #430. Runtime coupling only: `trace_id` populates when a span is active (#430); merged first, logs still ship with `job_id`/`model` correlation. **Verification:** 11 new tests (incl. mocked-POST CLEF payload assertion + active-span trace-id stamping) + 27 existing worker-protocol tests pass; ruff clean. Leaving **open** until PR #8 merges. Prod activation (set `SEQ_URL` + join `seq-attachable` network on the worker) is an infra follow-up, same pattern as #430's `JAEGER_ENDPOINT`.
spikerj commented 2026-07-08 04:11:54 +00:00
Author
Owner
Copy Link
Copy Source

Resolved in spikersoft-artpipe PR #8 (merged to main as 9d0650c). Worker logs now stamp trace_id/span_id from the active span plus structured props (asset_id/job_id/stage/stage_run_id/model/device) and ship to Seq via a stdlib-only CLEF handler (artpipe.seq_logging), env-gated on SEQ_URL (no-op when unset), non-blocking. Filtering Seq by trace_id now returns both the .NET ArtPipeProcessor logs and the art_pipe worker's internal logs for a run; local debug.log unchanged. Prod activation (set SEQ_URL + join seq-attachable on the worker) tracked as infra follow-up. Closing.

Resolved in spikersoft-artpipe **PR #8** (merged to `main` as `9d0650c`). Worker logs now stamp `trace_id`/`span_id` from the active span plus structured props (asset_id/job_id/stage/stage_run_id/model/device) and ship to Seq via a stdlib-only CLEF handler (`artpipe.seq_logging`), env-gated on `SEQ_URL` (no-op when unset), non-blocking. Filtering Seq by `trace_id` now returns both the .NET ArtPipeProcessor logs and the art_pipe worker's internal logs for a run; local `debug.log` unchanged. Prod activation (set `SEQ_URL` + join `seq-attachable` on the worker) tracked as infra follow-up. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#431