spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[QA][Perf] No measured figures exist for the non-GPU pipelines (coderunner, blog media, lesson video, decompile, OCR) — establish a benchmark/measurement pass so the READMEs can carry real numbers #570

New Issue
Open
opened 2026-07-14 14:29:10 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-14 14:29:10 +00:00
Owner
Copy Link
Copy Source

QA Team — filed 2026-07-14 from the README audit, at spikerj's direction. First I checked whether historical data could answer this without new work — it cannot:

  • Jaeger (7 days): SpikerSoft.EventHandlers.CodeExecution has 40 traces — 39 are GET /healthz (median 0.1 ms) plus one long-running CodeExecutionWorker.Start span. Zero actual code-execution traces.
  • Service logs: no execution activity in the retained window — nobody has run student code recently.
  • So the 4-replica coderunner fleet has never been measured under load, and neither have the other non-GPU workers.

By contrast the GPU pipeline is now well-characterized from QA's live observation of a real 22-page/182-image book upload (caption throughput at two image sizes, embeddings chunk rates, model provisioning time, VRAM actuals vs leases) — and those figures are in the new per-project READMEs. The non-GPU services have nothing.

Ask: a deliberate measurement pass producing figures the READMEs can quote (attributed "measured on the production swarm, "):

  1. coderunner — per-language cold/warm execution latency, throughput across the 4 replicas, container-spawn overhead, timeout/kill behavior under a runaway program.
  2. blog-media-processor / lesson-video-processor — processing time per MB (and per resolution for video), MinIO round-trip cost post-#413.
  3. decompile / OCR — per-artifact latency, memory ceilings.
  4. Ideally driven as a repeatable harness (a script that submits N jobs of known size and scrapes Jaeger/Seq for the spans) so it can be re-run after infra changes rather than being a one-off.

Bonus finding while investigating: coderunner's own spans are so sparse that its health probes outnumber its work 39:1 in Jaeger — worth confirming that execution paths actually emit spans (W3C propagation from the API's RPC through the worker) rather than being invisible to tracing. If they don't, that's a telemetry gap in its own right (AGENTS.md non-negotiable #1).

**QA Team** — filed 2026-07-14 from the README audit, at spikerj's direction. First I checked whether historical data could answer this without new work — **it cannot**: - **Jaeger (7 days):** `SpikerSoft.EventHandlers.CodeExecution` has 40 traces — **39 are `GET /healthz`** (median 0.1 ms) plus one long-running `CodeExecutionWorker.Start` span. Zero actual code-execution traces. - **Service logs:** no execution activity in the retained window — nobody has run student code recently. - So the 4-replica coderunner fleet has **never been measured under load**, and neither have the other non-GPU workers. By contrast the GPU pipeline is now well-characterized from QA's live observation of a real 22-page/182-image book upload (caption throughput at two image sizes, embeddings chunk rates, model provisioning time, VRAM actuals vs leases) — and those figures are in the new per-project READMEs. The non-GPU services have nothing. **Ask:** a deliberate measurement pass producing figures the READMEs can quote (attributed "measured on the production swarm, <date>"): 1. **coderunner** — per-language cold/warm execution latency, throughput across the 4 replicas, container-spawn overhead, timeout/kill behavior under a runaway program. 2. **blog-media-processor / lesson-video-processor** — processing time per MB (and per resolution for video), MinIO round-trip cost post-#413. 3. **decompile / OCR** — per-artifact latency, memory ceilings. 4. Ideally driven as a repeatable harness (a script that submits N jobs of known size and scrapes Jaeger/Seq for the spans) so it can be re-run after infra changes rather than being a one-off. **Bonus finding while investigating:** coderunner's own spans are so sparse that its *health probes* outnumber its *work* 39:1 in Jaeger — worth confirming that execution paths actually emit spans (W3C propagation from the API's RPC through the worker) rather than being invisible to tracing. If they don't, that's a telemetry gap in its own right (AGENTS.md non-negotiable #1).
spikerj commented 2026-07-29 17:10:44 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master. The measurement pass itself is NOT DONE (no harness, no figures) — but I chased the "bonus finding" and it's confirmed, with a precise root cause. Leading with that, because it changes the order of work.

The coderunner execution spans are structurally unexportable

Both handlers create their own ActivitySource as a field:

  • SpikerSoft.Business.CodeExecution/Domain/CodeExecution/Commands/ExecuteCode/ExecuteCodeCommandHandler.cs:25 — private readonly ActivitySource _activitySource = new("SpikerSoft.CodeExecution");, used at :32 (StartActivity("ExecuteCodeCommand"))
  • .../ExecuteLessonCode/ExecuteLessonCodeCommandHandler.cs:25 and :32 — same pattern

But "SpikerSoft.CodeExecution" is never passed to AddSource(...) anywhere. The API's tracer provider registers a specific list at SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs:340-354 — SpikerSoft.MediatR, SpikerSoft.DecompilerRouter, SpikerSoft.EmailService, TraceSources.Scheduler, TraceSources.KeycloakEvents, TraceSources.MessagePublishing, SpikerSoft.QuizGeneration, SpikerSoft.QuizGeneration.Worker — and SpikerSoft.CodeExecution is not among them. TraceSources.cs doesn't define a constant for it either.

In OpenTelemetry .NET an ActivitySource with no registered listener means StartActivity returns null — the activity is never created, so there is nothing to sample, export, or propagate. These aren't spans being dropped at the exporter; they never exist.

This explains your 39:1 observation exactly. The 39 GET /healthz traces come from auto-instrumented ASP.NET, and the single CodeExecutionWorker.Start span comes from CodeExecutionWorkerHostedService.cs:67, which uses an injected ActivitySource (:29, :48) rather than a self-constructed one — so that one is registered and does export. The two handlers that wrap actual execution are invisible.

So the conclusion isn't "nobody ran student code recently." It's that code execution has never been traceable, and running a load test today would still produce zero execution spans.

That also breaks W3C propagation end to end: with no activity on the handler side, there's no context to inject into the code.execution publish, so the worker's StartActivityFromRabbitMQMessage gets nothing to continue from and would start an orphan root — the exact failure LessonRegradeClient.cs:104 warns about in a neighbouring path.

This is an AGENTS.md non-negotiable (telemetry) violation and worth its own ticket — the fix is a one-line .AddSource("SpikerSoft.CodeExecution") (ideally promoted to a TraceSources constant, since the ad-hoc string literals at :340-342 and :353-354 are how this got missed).

The measurement pass

Still entirely outstanding: no benchmark/measurement harness exists, and the non-GPU READMEs carry no measured figures. All four asks stand.

But item 1 is now blocked on the telemetry fix. A harness that "submits N jobs of known size and scrapes Jaeger/Seq for the spans" cannot work for coderunner until those spans exist. Items 2–4 (blog media, lesson video, decompile, OCR) are worth checking for the same defect before building the harness — if their ActivitySources are also unregistered, the harness would come back empty across the board and look like a harness bug.

Suggested order: verify the ActivitySource registration for all five services, fix any that are unregistered, then build the harness against telemetry that actually emits.

Audited against `origin/master`. The measurement pass itself is **NOT DONE** (no harness, no figures) — but I chased the "bonus finding" and **it's confirmed, with a precise root cause**. Leading with that, because it changes the order of work. ## The coderunner execution spans are structurally unexportable Both handlers create their own ActivitySource as a field: - `SpikerSoft.Business.CodeExecution/Domain/CodeExecution/Commands/ExecuteCode/ExecuteCodeCommandHandler.cs:25` — `private readonly ActivitySource _activitySource = new("SpikerSoft.CodeExecution");`, used at `:32` (`StartActivity("ExecuteCodeCommand")`) - `.../ExecuteLessonCode/ExecuteLessonCodeCommandHandler.cs:25` and `:32` — same pattern **But `"SpikerSoft.CodeExecution"` is never passed to `AddSource(...)` anywhere.** The API's tracer provider registers a specific list at `SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs:340-354` — `SpikerSoft.MediatR`, `SpikerSoft.DecompilerRouter`, `SpikerSoft.EmailService`, `TraceSources.Scheduler`, `TraceSources.KeycloakEvents`, `TraceSources.MessagePublishing`, `SpikerSoft.QuizGeneration`, `SpikerSoft.QuizGeneration.Worker` — and `SpikerSoft.CodeExecution` is not among them. `TraceSources.cs` doesn't define a constant for it either. In OpenTelemetry .NET an ActivitySource with no registered listener means `StartActivity` returns **null** — the activity is never created, so there is nothing to sample, export, or propagate. These aren't spans being dropped at the exporter; they never exist. **This explains your 39:1 observation exactly.** The 39 `GET /healthz` traces come from auto-instrumented ASP.NET, and the single `CodeExecutionWorker.Start` span comes from `CodeExecutionWorkerHostedService.cs:67`, which uses an **injected** `ActivitySource` (`:29`, `:48`) rather than a self-constructed one — so that one is registered and does export. The two handlers that wrap actual execution are invisible. So the conclusion isn't "nobody ran student code recently." It's that **code execution has never been traceable**, and running a load test today would still produce zero execution spans. That also breaks W3C propagation end to end: with no activity on the handler side, there's no context to inject into the `code.execution` publish, so the worker's `StartActivityFromRabbitMQMessage` gets nothing to continue from and would start an orphan root — the exact failure `LessonRegradeClient.cs:104` warns about in a neighbouring path. This is an AGENTS.md non-negotiable (telemetry) violation and worth its own ticket — the fix is a one-line `.AddSource("SpikerSoft.CodeExecution")` (ideally promoted to a `TraceSources` constant, since the ad-hoc string literals at `:340-342` and `:353-354` are how this got missed). ## The measurement pass Still entirely outstanding: no benchmark/measurement harness exists, and the non-GPU READMEs carry no measured figures. All four asks stand. **But item 1 is now blocked on the telemetry fix.** A harness that "submits N jobs of known size and scrapes Jaeger/Seq for the spans" cannot work for coderunner until those spans exist. Items 2–4 (blog media, lesson video, decompile, OCR) are worth checking for the same defect before building the harness — if their ActivitySources are also unregistered, the harness would come back empty across the board and look like a harness bug. Suggested order: verify the ActivitySource registration for all five services, fix any that are unregistered, **then** build the harness against telemetry that actually emits.
spikerj commented 2026-07-29 17:11:56 +00:00
Author
Owner
Copy Link
Copy Source

Refining my previous comment — I enumerated every AddSource(...) registration repo-wide, and the picture is sharper (and more embarrassing) than "somebody forgot to register it".

It's a name mismatch, not an omission. The registered set does contain a CodeExecution source:

  • Registered: "SpikerSoft.EventHandlers.CodeExecution" — the worker host's source
  • Used by the handlers: "SpikerSoft.CodeExecution" — not registered anywhere

Two nearly identical names, one letter-group apart. That's why this passed review and why the service looks instrumented: grep for "CodeExecution" in the registration list and you find a hit. It's just not the one the execution path uses.

It also explains the Jaeger picture exactly. CodeExecutionWorker.Start exports because CodeExecutionWorkerHostedService uses the injected (registered) worker source. The two command handlers construct new("SpikerSoft.CodeExecution") as a field and export nothing.

The codebase already knows this failure mode by name. SpikerSoft.EventHandlers.DockerMonitor.Tests/Telemetry/TraceSourceRegistrationTests.cs:107-120 has a deliberate control test — AnUnregisteredSource_YieldsNoSpanAtAll_WhichIsHowThisBugKeepsHiding, using a source literally named "SpikerSoft.NobodyRegisteredThis" — whose doc comment reads:

"This pins the mechanism: an UNregistered source really does yield a null activity, which is precisely how #458, #575 and #588 each hid."

So this is the fourth occurrence of a documented, named bug class. The guard exists — but only inside DockerMonitor.Tests, asserting DockerMonitor's own sources. Nothing checks the property repo-wide, which is why #458, #575, #588 and now this each had to be found by hand.

Two fixes, and I'd argue for both:

  1. Register SpikerSoft.CodeExecution (or better, rename the handlers' source to the already-registered SpikerSoft.EventHandlers.CodeExecution — one string, no config change).
  2. Promote the DockerMonitor control test into a repo-wide guard: enumerate every ActivitySource name declared in non-test code, assert each appears in some AddSource(...). That converts a recurring manual discovery into a build failure. Given four occurrences, it's earned its place — and it's the durable fix, whereas registering one string just resets the counter.

Note the ad-hoc string literals are part of the problem: the API registers "SpikerSoft.MediatR", "SpikerSoft.DecompilerRouter", "SpikerSoft.EmailService", "SpikerSoft.QuizGeneration" as bare strings alongside TraceSources.* constants. Anything not going through TraceSources has no compiler help keeping the two ends in sync.

Both fixes are prerequisites for this ticket's harness — item 1 of the measurement pass cannot produce a single figure until the coderunner spans exist.

Refining my previous comment — I enumerated every `AddSource(...)` registration repo-wide, and the picture is sharper (and more embarrassing) than "somebody forgot to register it". **It's a name mismatch, not an omission.** The registered set *does* contain a CodeExecution source: - **Registered:** `"SpikerSoft.EventHandlers.CodeExecution"` — the **worker** host's source - **Used by the handlers:** `"SpikerSoft.CodeExecution"` — **not registered anywhere** Two nearly identical names, one letter-group apart. That's why this passed review and why the service *looks* instrumented: grep for "CodeExecution" in the registration list and you find a hit. It's just not the one the execution path uses. It also explains the Jaeger picture exactly. `CodeExecutionWorker.Start` exports because `CodeExecutionWorkerHostedService` uses the injected (registered) worker source. The two command handlers construct `new("SpikerSoft.CodeExecution")` as a field and export nothing. **The codebase already knows this failure mode by name.** `SpikerSoft.EventHandlers.DockerMonitor.Tests/Telemetry/TraceSourceRegistrationTests.cs:107-120` has a deliberate control test — `AnUnregisteredSource_YieldsNoSpanAtAll_WhichIsHowThisBugKeepsHiding`, using a source literally named `"SpikerSoft.NobodyRegisteredThis"` — whose doc comment reads: > *"This pins the mechanism: an UNregistered source really does yield a null activity, which is precisely how **#458, #575 and #588** each hid."* So this is the **fourth** occurrence of a documented, named bug class. The guard exists — but only inside `DockerMonitor.Tests`, asserting DockerMonitor's own sources. Nothing checks the property repo-wide, which is why #458, #575, #588 and now this each had to be found by hand. **Two fixes, and I'd argue for both:** 1. Register `SpikerSoft.CodeExecution` (or better, rename the handlers' source to the already-registered `SpikerSoft.EventHandlers.CodeExecution` — one string, no config change). 2. **Promote the DockerMonitor control test into a repo-wide guard**: enumerate every ActivitySource name declared in non-test code, assert each appears in some `AddSource(...)`. That converts a recurring manual discovery into a build failure. Given four occurrences, it's earned its place — and it's the durable fix, whereas registering one string just resets the counter. Note the ad-hoc string literals are part of the problem: the API registers `"SpikerSoft.MediatR"`, `"SpikerSoft.DecompilerRouter"`, `"SpikerSoft.EmailService"`, `"SpikerSoft.QuizGeneration"` as bare strings alongside `TraceSources.*` constants. Anything not going through `TraceSources` has no compiler help keeping the two ends in sync. Both fixes are prerequisites for this ticket's harness — item 1 of the measurement pass cannot produce a single figure until the coderunner spans exist.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#570