[Bug][Telemetry] Every service reports service.namespace='1.0.0' — AddService()'s 2nd positional arg is serviceNamespace, not serviceVersion. Spans also carry no node identity #588

New Issue

Closed

opened 2026-07-14 20:30:13 +00:00 by spikerj · 2 comments

spikerj commented 2026-07-14 20:30:13 +00:00

Owner

Copy Link

Copy Source

QA Team — sweep 2026-07-14 ~21:30Z. Found by reading actual span attributes in Jaeger, not by reading code.

Bug 1 — the version is being written into the namespace field

Live in Jaeger right now, on every .NET service:

serviceName: SpikerSoft.SecurityMonitor
  deployment.environment = Production
  service.instance.id    = c7bc592954fb
  service.namespace      = 1.0.0        <-- this is a VERSION, not a namespace
  service.version        = 1.0.0

The OpenTelemetry .NET signature is:

ResourceBuilder AddService(
    string serviceName,
    string? serviceNamespace = null,   // <-- 2nd POSITIONAL
    string? serviceVersion = null,
    ...)

And the shared host builder calls it with two positional args:

// SpikerSoft.EventHandlers.Infrastructure/Extensions/ServiceCollectionExtensions.cs:119 (tracing)
// ...and again at :180 (metrics)
.AddService(serviceName, serviceVersion)      // serviceVersion binds to serviceNamespace
.AddAttributes(new Dictionary<string, object>
{
    ["service.instance.id"] = Environment.MachineName,
    ["service.version"]     = serviceVersion,  // <-- why service.version still looks right
    ["deployment.environment"] = environment
})

So service.version is correct only by accident — it is re-set explicitly on the next line. service.namespace is silently populated with the version string.

Same bug, hardcoded, in SpikerSoft.EventHandlers.GpuCoordinator/Program.cs:99:

.AddService("SpikerSoft.GpuCoordinator", "1.0.0")   // -> service.namespace = "1.0.0"

Blast radius: all 24 EventHandler workers (via the shared library, on both the tracing and metrics resource builders) plus gpu-coordinator.

Separately, the API (SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs:327 and :382) calls AddService("SpikerSoft API") with no version at all, so the API reports no service.version — the inverse omission.

Fix: use named arguments. This is precisely the class of bug positional args invite.

.AddService(serviceName: serviceName, serviceVersion: serviceVersion)

Bug 2 — you cannot tell which node a span came from

["service.instance.id"] = Environment.MachineName

Inside a container, Environment.MachineName is the container ID (c7bc592954fb), not the swarm node. There is no host.name / node attribute on any span. Confirmed by querying 200 NodeAgent traces: not one carries a host identity.

That means Jaeger cannot answer "which node was this running on?" — and we have spent today on exactly that class of question:

• #553 — quiz-generation was running on SERVER's 8 GB card while its GPU lease was granted on the 4090. A host.name on the span would have made that visible immediately; instead it took a manual docker service inspect.
• #587 — services surviving only on cached images, node-by-node.
• #562 — ds1/ds4 failing while ds5 works.

For a 10-node swarm where placement is the bug surface, spans that cannot name their node are a significant hole. node-agent is mode: global, so its spans are indistinguishable across all 10 nodes today.

Fix: stamp the real host. The swarm injects {{.Node.Hostname}} if the stack file passes it as an env var, or use AddDetector(new HostDetector()) / set host.name explicitly:

["host.name"] = Environment.GetEnvironmentVariable("NODE_HOSTNAME") ?? Environment.MachineName,
["service.instance.id"] = Environment.MachineName,   // container id — fine, but not a host

Also observed while here (separate, needs node access to root-cause)

spikersoft-quiz-generation has 1 trace in the last 24 hours and 0 in the last 90 minutes, despite restarting at 20:11:56Z onto the 4090 and logging its own startup trace id:

[20:11:56 INF] 🔍 [OTEL DEBUG] Jaeger endpoint configured: http://jaeger:4317
[20:11:56 INF] 🔍 [OTEL DEBUG] Startup activity created: True, TraceId: 1dd330ebd6840beceace48ce996b0bcc

That trace id returns HTTP 404 / "trace not found" from the Jaeger API. The span was created (the ActivitySource had a listener, or StartActivity would have returned null) but never landed. The service is attached to the jaeger overlay. I could not get inside the container to test jaeger:4317 reachability from the 4090, so I am not guessing at a cause — but a service that logs a TraceId that does not exist in Jaeger is dropping spans, and it is worth someone with node access confirming whether OTLP egress from the 4090 to jaeger:4317 actually works.

Related: the audit already found that AddEventHandlerTelemetry hardcodes .AddSource("SpikerSoft.QuizGeneration.Worker") for every handler while custom per-service ActivitySources go unregistered (KeycloakEventService's spans are dropped for that reason — it shows 0 traces too). Same file, same method — worth fixing together.

**QA Team** — sweep 2026-07-14 ~21:30Z. Found by reading actual span attributes in Jaeger, not by reading code. ## Bug 1 — the version is being written into the namespace field Live in Jaeger right now, on every .NET service: ``` serviceName: SpikerSoft.SecurityMonitor deployment.environment = Production service.instance.id = c7bc592954fb service.namespace = 1.0.0 <-- this is a VERSION, not a namespace service.version = 1.0.0 ``` The OpenTelemetry .NET signature is: ```csharp ResourceBuilder AddService( string serviceName, string? serviceNamespace = null, // <-- 2nd POSITIONAL string? serviceVersion = null, ...) ``` And the shared host builder calls it with two positional args: ```csharp // SpikerSoft.EventHandlers.Infrastructure/Extensions/ServiceCollectionExtensions.cs:119 (tracing) // ...and again at :180 (metrics) .AddService(serviceName, serviceVersion) // serviceVersion binds to serviceNamespace .AddAttributes(new Dictionary<string, object> { ["service.instance.id"] = Environment.MachineName, ["service.version"] = serviceVersion, // <-- why service.version still looks right ["deployment.environment"] = environment }) ``` So `service.version` is correct **only by accident** — it is re-set explicitly on the next line. `service.namespace` is silently populated with the version string. Same bug, hardcoded, in `SpikerSoft.EventHandlers.GpuCoordinator/Program.cs:99`: ```csharp .AddService("SpikerSoft.GpuCoordinator", "1.0.0") // -> service.namespace = "1.0.0" ``` Blast radius: **all 24 EventHandler workers** (via the shared library, on both the tracing *and* metrics resource builders) plus gpu-coordinator. Separately, the API (`SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs:327` and `:382`) calls `AddService("SpikerSoft API")` with **no version at all**, so the API reports no `service.version` — the inverse omission. **Fix:** use named arguments. This is precisely the class of bug positional args invite. ```csharp .AddService(serviceName: serviceName, serviceVersion: serviceVersion) ``` ## Bug 2 — you cannot tell which node a span came from ```csharp ["service.instance.id"] = Environment.MachineName ``` Inside a container, `Environment.MachineName` is the **container ID** (`c7bc592954fb`), not the swarm node. There is no `host.name` / `node` attribute on any span. Confirmed by querying 200 NodeAgent traces: not one carries a host identity. That means **Jaeger cannot answer "which node was this running on?"** — and we have spent today on exactly that class of question: - **#553** — quiz-generation was running on SERVER's 8 GB card while its GPU lease was granted on the 4090. A `host.name` on the span would have made that visible immediately; instead it took a manual `docker service inspect`. - **#587** — services surviving only on cached images, node-by-node. - **#562** — ds1/ds4 failing while ds5 works. For a 10-node swarm where placement *is* the bug surface, spans that cannot name their node are a significant hole. `node-agent` is `mode: global`, so its spans are indistinguishable across all 10 nodes today. **Fix:** stamp the real host. The swarm injects `{{.Node.Hostname}}` if the stack file passes it as an env var, or use `AddDetector(new HostDetector())` / set `host.name` explicitly: ```csharp ["host.name"] = Environment.GetEnvironmentVariable("NODE_HOSTNAME") ?? Environment.MachineName, ["service.instance.id"] = Environment.MachineName, // container id — fine, but not a host ``` ## Also observed while here (separate, needs node access to root-cause) `spikersoft-quiz-generation` has **1 trace in the last 24 hours** and **0 in the last 90 minutes**, despite restarting at 20:11:56Z onto the 4090 and logging its own startup trace id: ``` [20:11:56 INF] 🔍 [OTEL DEBUG] Jaeger endpoint configured: http://jaeger:4317 [20:11:56 INF] 🔍 [OTEL DEBUG] Startup activity created: True, TraceId: 1dd330ebd6840beceace48ce996b0bcc ``` That trace id returns **HTTP 404 / "trace not found"** from the Jaeger API. The span was created (the ActivitySource had a listener, or `StartActivity` would have returned null) but never landed. The service *is* attached to the `jaeger` overlay. I could not get inside the container to test `jaeger:4317` reachability from the 4090, so I am not guessing at a cause — but a service that logs a TraceId that does not exist in Jaeger is dropping spans, and it is worth someone with node access confirming whether OTLP egress from the 4090 to `jaeger:4317` actually works. Related: the audit already found that `AddEventHandlerTelemetry` hardcodes `.AddSource("SpikerSoft.QuizGeneration.Worker")` for every handler while custom per-service ActivitySources go unregistered (KeycloakEventService's spans are dropped for that reason — it shows **0 traces** too). Same file, same method — worth fixing together.

spikerj commented 2026-07-14 21:04:47 +00:00

Author

Owner

Copy Link

Copy Source

Fixed across two PRs (both open, awaiting merge). All three items in this ticket are covered.

• spikersoft-backend #288 — the code fix
• spikersoft-infrastructure #82 — the companion stack-file change (item 2 is inert without it)

Item 1 — service.namespace = "1.0.0"

Confirmed exactly as reported. AddService's second positional parameter is serviceNamespace, so AddService(serviceName, serviceVersion) bound the version to the namespace; service.version only looked right because the next line re-set it by hand.

All 5 call sites now use named arguments, behind one shared SpikerSoftResource.Create() so the fleet can't drift apart again.

Two things this ticket didn't mention, found while fixing it:

• The API had the inverse bug — AddService("SpikerSoft API") with no version at all.
• The metrics resource builder had drifted furthest: it carried no deployment.environment whatsoever. Tracing and metrics now share one definition.

Rather than take the diagnosis on faith, there's a test that reproduces the footgun with the pre-fix call verbatim and asserts the version lands in service.namespace — so it's proven, and a future OTel release that changes the overload will tell us.

Item 2 — spans couldn't name their node

host.name now comes from NODE_HOSTNAME, falling back to MachineName so a not-yet-redeployed service reports something rather than nothing. service.instance.id keeps the container id — still worth having, it just isn't a host.

spikersoft-node-agent already used NODE_HOSTNAME={{.Node.Hostname}}, so that's the convention followed rather than a new one invented. Infra PR #82 adds it to the other 34 .NET stacks.

⚠️ This only takes effect once infra #82 merges and the swarm's /mnt/infrastructure checkout is pulled. Until then host.name sits on the safe fallback.

Item 3 — KeycloakEventService's ActivitySource

Confirmed, and worse than described. The source was registered by nobody — not the API's list, not the KeycloakEvents worker (which registers AddSource(serviceName) = "SpikerSoft.EventHandlers.KeycloakEvents", a different string from "SpikerSoft.KeycloakEventService").

The consequence: the worker traced its RabbitMQ consumer span but dropped every child span of the save it triggered. Traces showed a Keycloak message arriving and then nothing happening to it. Nothing threw, nothing logged.

The name now lives in TraceSources and both hosts register it.

The pattern worth naming

This is the third time this identical bug has shipped — #458 (SpikerSoft.MessageBus), #575 (SpikerSoft.Scheduler), now #588 (SpikerSoft.KeycloakEventService). It is invisible to code review every time, because the construction site reads perfectly:

private readonly ActivitySource _activitySource = new("SpikerSoft.KeycloakEventService");

The failure is somewhere else entirely: nobody called AddSource with that name. An ActivitySource with no listener returns null from StartActivity, so using var activity = ... silently no-ops and every span vanishes.

So the tests target the class, not the instance: TraceSourceRegistrationTests builds the tracer through the real DI wiring and asserts a span started from each source actually exists, with a control test pinning the mechanism (an unregistered source really does yield a null activity). Verified by removing the AddSource line — it fails only the Keycloak test and leaves the others green.

That's the guard that would have caught all three of these.

Verification

10 new tests, every one verified failing before the fix. Build clean (0 errors); Common 609, API 1288, DockerMonitor 40 — all passing. All 34 infra files verified to still parse as YAML with the {{.Node.Hostname}} placeholder intact rather than mangled by the parser.

Will close once both PRs are merged.

Fixed across two PRs (both open, awaiting merge). All **three** items in this ticket are covered. - **spikersoft-backend #288** — the code fix - **spikersoft-infrastructure #82** — the companion stack-file change (item 2 is inert without it) ## Item 1 — `service.namespace = "1.0.0"` Confirmed exactly as reported. `AddService`'s second positional parameter is `serviceNamespace`, so `AddService(serviceName, serviceVersion)` bound the version to the namespace; `service.version` only looked right because the next line re-set it by hand. All 5 call sites now use **named arguments**, behind one shared `SpikerSoftResource.Create()` so the fleet can't drift apart again. Two things this ticket didn't mention, found while fixing it: - The **API had the inverse bug** — `AddService("SpikerSoft API")` with **no version at all**. - The **metrics** resource builder had drifted furthest: it carried **no `deployment.environment` whatsoever**. Tracing and metrics now share one definition. Rather than take the diagnosis on faith, there's a test that reproduces the footgun with the pre-fix call verbatim and asserts the version lands in `service.namespace` — so it's proven, and a future OTel release that changes the overload will tell us. ## Item 2 — spans couldn't name their node `host.name` now comes from `NODE_HOSTNAME`, falling back to `MachineName` so a not-yet-redeployed service reports something rather than nothing. `service.instance.id` keeps the container id — still worth having, it just isn't a host. `spikersoft-node-agent` **already used** `NODE_HOSTNAME={{.Node.Hostname}}`, so that's the convention followed rather than a new one invented. Infra PR #82 adds it to the other 34 .NET stacks. ⚠️ **This only takes effect once infra #82 merges and the swarm's `/mnt/infrastructure` checkout is pulled.** Until then `host.name` sits on the safe fallback. ## Item 3 — `KeycloakEventService`'s ActivitySource Confirmed, and **worse than described**. The source was registered by *nobody* — not the API's list, not the KeycloakEvents worker (which registers `AddSource(serviceName)` = `"SpikerSoft.EventHandlers.KeycloakEvents"`, a different string from `"SpikerSoft.KeycloakEventService"`). The consequence: the worker traced its RabbitMQ **consumer span** but dropped every **child span of the save it triggered**. Traces showed a Keycloak message arriving and then nothing happening to it. Nothing threw, nothing logged. The name now lives in `TraceSources` and both hosts register it. ## The pattern worth naming This is the **third** time this identical bug has shipped — #458 (`SpikerSoft.MessageBus`), #575 (`SpikerSoft.Scheduler`), now #588 (`SpikerSoft.KeycloakEventService`). It is invisible to code review every time, because the construction site reads perfectly: ```csharp private readonly ActivitySource _activitySource = new("SpikerSoft.KeycloakEventService"); ``` The failure is somewhere else entirely: nobody called `AddSource` with that name. An `ActivitySource` with no listener returns **null** from `StartActivity`, so `using var activity = ...` silently no-ops and every span vanishes. So the tests target the **class, not the instance**: `TraceSourceRegistrationTests` builds the tracer through the **real DI wiring** and asserts a span started from each source actually exists, with a control test pinning the mechanism (an unregistered source really does yield a null activity). Verified by removing the `AddSource` line — it fails *only* the Keycloak test and leaves the others green. That's the guard that would have caught all three of these. ## Verification 10 new tests, every one verified failing before the fix. Build clean (0 errors); Common 609, API 1288, DockerMonitor 40 — all passing. All 34 infra files verified to still parse as YAML with the `{{.Node.Hostname}}` placeholder intact rather than mangled by the parser. Will close once both PRs are merged.

spikerj commented 2026-07-14 21:08:36 +00:00

Author

Owner

Copy Link

Copy Source

Both PRs are merged to master:

• spikersoft-backend #288 — named-arg AddService behind a shared SpikerSoftResource, host.name node identity, and TraceSources.KeycloakEvents registered by both hosts.
• spikersoft-infrastructure #82 — NODE_HOSTNAME={{.Node.Hostname}} on all 34 .NET stacks.

host.name starts reporting the real swarm node on the next redeploy once /mnt/infrastructure is pulled; until then it sits on the safe MachineName fallback.

All three items resolved. Closing.

Both PRs are merged to `master`: - **spikersoft-backend #288** — named-arg `AddService` behind a shared `SpikerSoftResource`, `host.name` node identity, and `TraceSources.KeycloakEvents` registered by both hosts. - **spikersoft-infrastructure #82** — `NODE_HOSTNAME={{.Node.Hostname}}` on all 34 .NET stacks. `host.name` starts reporting the real swarm node on the next redeploy once `/mnt/infrastructure` is pulled; until then it sits on the safe `MachineName` fallback. All three items resolved. Closing.

spikerj closed this issue 2026-07-14 21:08:39 +00:00

spikerj referenced this issue 2026-07-14 21:12:24 +00:00

[Bug][Backend][Observability] Serilog MinimumLevel is hardcoded to Debug in EventHandlerHostBuilder — EVERY appsettings log-level setting is dead config, and ~2.9M Debug events/day are shipped to Seq #594

spikerj referenced this issue 2026-07-14 21:25:08 +00:00

[Bug][CI][Deploy] 28 of 29 deploy jobs never 'git pull' /mnt/infrastructure — they deploy STALE stack files and report success #584

spikerj referenced this issue 2026-07-14 21:31:22 +00:00

[Bug][Backend][Observability] #575 was fixed in ONE publisher — BookController (5 publish sites) and 3 others still ship RabbitMQ messages with no traceparent, so user-initiated flows appear in Jaeger as disconnected orphan traces #597

spikerj referenced this issue 2026-07-14 21:35:20 +00:00

[Bug][CI][Deploy] cancel-in-progress on master kills DEPLOY jobs — back-to-back merges cancel each other and fixes silently never ship #590

spikerj referenced this issue 2026-07-14 22:28:07 +00:00

[Bug][Backend][Observability] #575 was fixed in ONE publisher — BookController (5 publish sites) and 3 others still ship RabbitMQ messages with no traceparent, so user-initiated flows appear in Jaeger as disconnected orphan traces #597

spikerj referenced this issue 2026-07-14 22:36:22 +00:00

[Bug][Telemetry][QA] #588 was closed but the fix never reached production — every .NET service STILL reports service.namespace=1.0.0 and no span carries node identity (verified in live Jaeger) #602

spikerj referenced this issue 2026-07-14 22:36:55 +00:00

[Bug][Prod][Registry] 14 images have no :latest manifest (manifest unknown / 404) — any reschedule onto a fresh node will fail to start #587

spikerj referenced this issue 2026-07-22 03:12:09 +00:00

[Bug][Telemetry][QA] #588 was closed but the fix never reached production — every .NET service STILL reports service.namespace=1.0.0 and no span carries node identity (verified in live Jaeger) #602

spikerj referenced this issue 2026-07-29 15:27:39 +00:00

Rework /employment into a routed master-detail careers page with a globe #867

spikerj referenced this issue 2026-07-29 15:47:10 +00:00

[Bug][Telemetry][QA] #588 was closed but the fix never reached production — every .NET service STILL reports service.namespace=1.0.0 and no span carries node identity (verified in live Jaeger) #602

spikerj referenced this issue 2026-07-29 17:11:56 +00:00

[QA][Perf] No measured figures exist for the non-GPU pipelines (coderunner, blog media, lesson video, decompile, OCR) — establish a benchmark/measurement pass so the READMEs can carry real numbers #570

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

No results found.

Labels

Clear labels

agentic

Actionable surfaced by the agentic architecture & security review

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

sonarqube

Code quality issue surfaced by SonarQube (bug / code smell)

wontfix

This won't be fixed

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

spikerj (Joseph Spiker)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#588