spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Telemetry][QA] #588 was closed but the fix never reached production — every .NET service STILL reports service.namespace=1.0.0 and no span carries node identity (verified in live Jaeger) #602

New Issue
Open
opened 2026-07-14 22:36:22 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-14 22:36:22 +00:00
Owner
Copy Link
Copy Source

QA Team — regression check on the recently-closed #588.

#588 ("Every service reports service.namespace='1.0.0'… spans also carry no node identity") was closed 2026-07-14 21:35Z. We audited live Jaeger data and the bug is still fully present in production. The fix appears to have merged but never reached the running containers — the same merge-but-never-deploy pattern as #553/#582.

Evidence (live Jaeger, http://192.168.0.105:16686, sampled 2,760 traces / 4,926 spans across all 23 reporting services)

service.namespace is still 1.0.0 on every .NET service — 15 confirmed:
DockerMonitor, NodeAgent, SecurityMonitor, SystemRemediation, GpuCoordinator, and EventHandlers ArtPipeProcessor, ArtStudioMetrics, CodeExecution, InfluxDashboard, KeycloakEvents, MetadataExtractor, Notifications, QuizGeneration, SecurityScanner, UploadCoordinator.

Node identity is still absent. No host.name on any SpikerSoft service — only jaeger and traefik carry it, and those are third-party images self-instrumenting. We censused all 56 distinct span tag keys: there is no node/host tag at span level either. service.instance.id is a 12-hex Docker container id (e.g. 54f4d6d146ef), not a node hostname — so you still cannot answer "which node emitted this span".

Why we are confident this is a deploy gap, not a partial rollout: NodeAgent (7 replicas) and CodeExecution (3 replicas) each report multiple distinct service.instance.ids, and every single replica carries the bad value. A partially-deployed fix would show a mix. There is none.

Two variants worth folding in

  • SpikerSoft API: has neither service.namespace nor service.version — only service.instance.id + telemetry.sdk.*. It is on a different instrumentation path from the workers, so whatever fix landed for the workers would not have covered it anyway.
  • SpikerSoft.GpuCoordinator: service.namespace=1.0.0 but service.version absent.

Probable mechanism

Consistent with the registry problem we are reporting on #587: the :latest tag no longer resolves (manifest unknown), so nodes keep running stale cached images. A fix can merge, build, and still never appear in a running container. Recommend not re-closing #588 until service.namespace is verified in live Jaeger, not just in the diff.

Suggested acceptance

  • Query Jaeger for any SpikerSoft service and assert service.namespace != "1.0.0".
  • Assert a node-identity tag (host.name or equivalent) is present on spans from a service with >1 replica, and that it differs across nodes.

Related: #587 (registry :latest missing), #597 (trace-context propagation), #582/#553 (closed-but-never-deployed precedent).

**QA Team** — regression check on the recently-closed #588. #588 ("Every service reports `service.namespace='1.0.0'`… spans also carry no node identity") was **closed 2026-07-14 21:35Z**. We audited live Jaeger data and the bug is **still fully present in production**. The fix appears to have merged but never reached the running containers — the same merge-but-never-deploy pattern as #553/#582. ## Evidence (live Jaeger, http://192.168.0.105:16686, sampled 2,760 traces / 4,926 spans across all 23 reporting services) **`service.namespace` is still `1.0.0` on every .NET service — 15 confirmed:** `DockerMonitor`, `NodeAgent`, `SecurityMonitor`, `SystemRemediation`, `GpuCoordinator`, and EventHandlers `ArtPipeProcessor`, `ArtStudioMetrics`, `CodeExecution`, `InfluxDashboard`, `KeycloakEvents`, `MetadataExtractor`, `Notifications`, `QuizGeneration`, `SecurityScanner`, `UploadCoordinator`. **Node identity is still absent.** No `host.name` on any SpikerSoft service — only `jaeger` and `traefik` carry it, and those are third-party images self-instrumenting. We censused all 56 distinct span tag keys: there is no node/host tag at span level either. `service.instance.id` is a 12-hex Docker **container** id (e.g. `54f4d6d146ef`), not a node hostname — so you still cannot answer "which node emitted this span". **Why we are confident this is a deploy gap, not a partial rollout:** `NodeAgent` (7 replicas) and `CodeExecution` (3 replicas) each report multiple distinct `service.instance.id`s, and **every single replica** carries the bad value. A partially-deployed fix would show a mix. There is none. ## Two variants worth folding in - **`SpikerSoft API`**: has *neither* `service.namespace` nor `service.version` — only `service.instance.id` + `telemetry.sdk.*`. It is on a different instrumentation path from the workers, so whatever fix landed for the workers would not have covered it anyway. - **`SpikerSoft.GpuCoordinator`**: `service.namespace=1.0.0` but `service.version` **absent**. ## Probable mechanism Consistent with the registry problem we are reporting on #587: the `:latest` tag no longer resolves (`manifest unknown`), so nodes keep running stale cached images. A fix can merge, build, and still never appear in a running container. **Recommend not re-closing #588 until `service.namespace` is verified in live Jaeger, not just in the diff.** ## Suggested acceptance - Query Jaeger for any SpikerSoft service and assert `service.namespace != "1.0.0"`. - Assert a node-identity tag (`host.name` or equivalent) is present on spans from a service with >1 replica, and that it differs across nodes. Related: #587 (registry `:latest` missing), #597 (trace-context propagation), #582/#553 (closed-but-never-deployed precedent).
spikerj commented 2026-07-22 03:12:09 +00:00
Author
Owner
Copy Link
Copy Source

Reconciliation check 2026-07-22 (code vs. ticket): the code side is fully merged, the prod-verification side is not done, so this stays open.

  • Backend: SpikerSoft.Common/Telemetry/SpikerSoftResource.cs (commit 7425b64c, #588) is on master — named AddService(serviceName:, serviceVersion:) so service.namespace is no longer "1.0.0", plus host.name from NODE_HOSTNAME with container-name fallback.
  • Infrastructure: PR #82 (merged) passes NODE_HOSTNAME={{.Node.Hostname}} in the .NET service stack files.

What remains is exactly what this ticket is about: confirming every .NET service in live Jaeger now reports a real namespace + node identity, which requires all services redeployed on post-#588 images. Note the open deploy-reliability tickets (#774 silent rollback, #717 superseded builds) are plausible reasons a fleet-wide redeploy silently didn't take. Close only after re-checking live Jaeger.

Reconciliation check 2026-07-22 (code vs. ticket): the **code side is fully merged**, the **prod-verification side is not done**, so this stays open. - Backend: `SpikerSoft.Common/Telemetry/SpikerSoftResource.cs` (commit 7425b64c, #588) is on master — named `AddService(serviceName:, serviceVersion:)` so `service.namespace` is no longer "1.0.0", plus `host.name` from `NODE_HOSTNAME` with container-name fallback. - Infrastructure: PR #82 (merged) passes `NODE_HOSTNAME={{.Node.Hostname}}` in the .NET service stack files. What remains is exactly what this ticket is about: confirming every .NET service in **live Jaeger** now reports a real namespace + node identity, which requires all services redeployed on post-#588 images. Note the open deploy-reliability tickets (#774 silent rollback, #717 superseded builds) are plausible reasons a fleet-wide redeploy silently didn't take. Close only after re-checking live Jaeger.
spikerj commented 2026-07-29 15:47:10 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master in both repos. The code fix is now complete and adopted fleet-wide — which was not true when this was filed. Deliberately not closing it; see the last paragraph.

Both bugs are fixed at the source. SpikerSoft.Common/Telemetry/SpikerSoftResource.cs:

  • Create() calls .AddService(serviceName: serviceName, serviceVersion: serviceVersion) with named arguments — the class comment at :14 records the original defect verbatim ("the third positional parameter is serviceNamespace, so every service reported service.namespace = "1.0.0"").
  • host.name is populated from ResolveNodeHostname() (:44), reading NODE_HOSTNAME and falling back to Environment.MachineName. service.instance.id is kept separately as the container identity, which is the right distinction.
  • One Create() is used for both the tracing and metrics resource builders (EventHandlers.Infrastructure/Extensions/ServiceCollectionExtensions.cs:150 and :220) — the doc comment notes those two had already drifted, with the metrics one carrying no deployment.environment at all.

Adoption is real, not partial. AddEventHandlerTelemetry is invoked from EventHandlerHostBuilder.cs:99, and 26 worker hosts build on EventHandlerHostBuilder — so every one inherits it rather than each having to opt in. Plus SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs and EventHandlers.GpuCoordinator/Program.cs directly.

The infrastructure half is wired too. NODE_HOSTNAME now appears in 46 stack files in spikersoft-infrastructure (of 81 docker-stack*.yml total — the remainder are non-.NET stacks that never emitted spans). When this ticket was written, SpikerSoftResourceTests.cs:100 still carried the note that "the stack files are a separate repo, so services will run without NODE_HOSTNAME for a while." That gap has closed.

Regression coverage exists for both, pinned to the observed symptoms: SpikerSoftResourceTests.cs:63 ServiceNamespace_IsNotTheVersion, :92 asserting host.name == "dreamstream4" from an interpolated NODE_HOSTNAME, and :105 asserting the machine-name fallback.

Why I'm leaving this open. This ticket exists precisely because #588 was closed on a merged PR while production still showed the broken values. Closing it again on code-completeness alone would repeat that mistake exactly. The remaining step is the one that originally failed: confirm in live Jaeger that spans now carry a real service.namespace (not 1.0.0) and a populated host.name. One person, one query — then close with the result recorded here.

Audited against `origin/master` in both repos. **The code fix is now complete and adopted fleet-wide** — which was not true when this was filed. Deliberately **not** closing it; see the last paragraph. **Both bugs are fixed at the source.** `SpikerSoft.Common/Telemetry/SpikerSoftResource.cs`: - `Create()` calls `.AddService(serviceName: serviceName, serviceVersion: serviceVersion)` with **named arguments** — the class comment at `:14` records the original defect verbatim ("the third positional parameter is `serviceNamespace`, so every service reported `service.namespace = "1.0.0"`"). - `host.name` is populated from `ResolveNodeHostname()` (`:44`), reading `NODE_HOSTNAME` and falling back to `Environment.MachineName`. `service.instance.id` is kept separately as the container identity, which is the right distinction. - One `Create()` is used for **both** the tracing and metrics resource builders (`EventHandlers.Infrastructure/Extensions/ServiceCollectionExtensions.cs:150` and `:220`) — the doc comment notes those two had already drifted, with the metrics one carrying no `deployment.environment` at all. **Adoption is real, not partial.** `AddEventHandlerTelemetry` is invoked from `EventHandlerHostBuilder.cs:99`, and **26 worker hosts** build on `EventHandlerHostBuilder` — so every one inherits it rather than each having to opt in. Plus `SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs` and `EventHandlers.GpuCoordinator/Program.cs` directly. **The infrastructure half is wired too.** `NODE_HOSTNAME` now appears in **46 stack files** in spikersoft-infrastructure (of 81 `docker-stack*.yml` total — the remainder are non-.NET stacks that never emitted spans). When this ticket was written, `SpikerSoftResourceTests.cs:100` still carried the note that "the stack files are a separate repo, so services will run without NODE_HOSTNAME for a while." That gap has closed. **Regression coverage exists for both**, pinned to the observed symptoms: `SpikerSoftResourceTests.cs:63` `ServiceNamespace_IsNotTheVersion`, `:92` asserting `host.name == "dreamstream4"` from an interpolated `NODE_HOSTNAME`, and `:105` asserting the machine-name fallback. **Why I'm leaving this open.** This ticket exists precisely because #588 was closed on a merged PR while production still showed the broken values. Closing it again on code-completeness alone would repeat that mistake exactly. The remaining step is the one that originally failed: confirm in **live Jaeger** that spans now carry a real `service.namespace` (not `1.0.0`) and a populated `host.name`. One person, one query — then close with the result recorded here.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#602