enhancement: Configure OpenTelemetry sampling for API/workers so telemetry stays cheap on constrained hosts #130

New Issue

2026-05-12T23:55:53Z

spikerj commented

2026-05-12 23:55:53 +00:00

Summary

Heavy OpenTelemetry exporters and always-on instrumentation can consume measurable CPU alongside request handling — especially noticeable on asymmetric CPU layouts (e.g. NVIDIA Jetson Denver + A57).

We should add explicit, configurable sampling for traces (and optionally adjust metric export intervals) so production observability stays useful without dominating the hottest cores.

Goals

Tracing: Use a configurable ratio sampler (e.g. TraceIdRatioBasedSampler) with a sane default under load; allow 100% in dev/Diagnostics scenarios.
Configuration: Wire via existing config patterns (appsettings.json / env), e.g. OpenTelemetry:Tracing:SampleRatio or similar — document defaults.
Optional: Separate sampling knobs for ASP.NET incoming vs outbound HttpClient if we need finer control later.
Metrics: Consider longer export intervals or reduced metric cardinality in constrained profiles (secondary; traces are usually the hotspot).

Proposed backend touchpoints

SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs → AddTelemetryConfiguration (where tracing + exporters are wired today).
Mirror the same pattern in worker Program.cs stacks that register OTel if we want uniform behavior cluster-wide.

Acceptance criteria

Sampling is off-by-default-transparent or has a conservative production default documented in appsettings commentary.
Operators can tune ratio without rebuilding (env / configuration).
Document how to correlate when sampling is < 100% (e.g. rely on logs + trace context propagation from sampled roots).
Smoke test: API starts with sampling 0, 1, and 0.1 without exceptions.

Labels

enhancement · backend · telemetry

## Summary Heavy OpenTelemetry exporters and always-on instrumentation can consume measurable CPU alongside request handling — especially noticeable on asymmetric CPU layouts (e.g. NVIDIA Jetson Denver + A57). We should add **explicit, configurable sampling** for traces (and optionally adjust metric export intervals) so production observability stays useful without dominating the hottest cores. ## Goals - **Tracing:** Use a configurable ratio sampler (e.g. `TraceIdRatioBasedSampler`) with a sane default under load; allow `100%` in dev/Diagnostics scenarios. - **Configuration:** Wire via existing config patterns (`appsettings.json` / env), e.g. `OpenTelemetry:Tracing:SampleRatio` or similar — document defaults. - **Optional:** Separate sampling knobs for ASP.NET incoming vs outbound `HttpClient` if we need finer control later. - **Metrics:** Consider longer export intervals or reduced metric cardinality in constrained profiles (secondary; traces are usually the hotspot). ## Proposed backend touchpoints - `SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs` → `AddTelemetryConfiguration` (where tracing + exporters are wired today). - Mirror the same pattern in worker `Program.cs` stacks that register OTel if we want uniform behavior cluster-wide. ## Acceptance criteria 1. Sampling is **off-by-default-transparent** or has a conservative production default documented in appsettings commentary. 2. Operators can tune ratio without rebuilding (env / configuration). 3. Document how to correlate when sampling is `< 100%` (e.g. rely on logs + trace context propagation from sampled roots). 4. Smoke test: API starts with sampling `0`, `1`, and `0.1` without exceptions. ## Labels `enhancement` · `backend` · `telemetry`

spikerj commented

2026-05-14 17:57:15 +00:00

Implementation Summary

Configurable OpenTelemetry trace sampling has been implemented across the API and all 19 event handler workers.

Changes

1. API — SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs

AddTelemetryConfiguration now reads OpenTelemetry:Tracing:SampleRatio from config (default 1.0).
Value is clamped to [0.0, 1.0] for safety.
A ParentBasedSampler(new TraceIdRatioBasedSampler(ratio)) is wired into the TracerProviderBuilder. This ensures:
- Root traces are sampled at the configured ratio.
- Child spans inherit the sampling decision from their parent, preserving full distributed-trace correlation.

2. Workers — SpikerSoft.EventHandlers.Infrastructure/Extensions/ServiceCollectionExtensions.cs

AddEventHandlerTelemetry follows the identical pattern: reads ratio → clamps → ParentBasedSampler.
All 19 workers that call EventHandlerHostBuilder.WithTelemetry() automatically inherit this behavior.

3. Configuration — appsettings.json (API + all workers)

Added OpenTelemetry.Tracing.SampleRatio key with default 1.0 (transparent / no change in behavior).
Operators can override via env var: OpenTelemetry__Tracing__SampleRatio=0.1

4. Unit Tests — SpikerSoft.Tests.Unit/Telemetry/TelemetrySamplingTests.cs

14 new tests covering:
- Valid ratios (0.0, 0.1, 0.5, 1.0) don't throw.
- Missing config defaults to 1.0.
- Out-of-range values (−1, 2, 100) are clamped without exceptions.
- Ratio 0.0 marks traces as not sampled (IsAllDataRequested = false).
- Ratio 1.0 records all traces (IsAllDataRequested = true).
- ParentBasedSampler honors a sampled parent even when root ratio is 0.0.
- Config value round-trips correctly.

All 14 tests pass ✅

Correlation at < 100% Sampling

Because ParentBasedSampler wraps the ratio sampler, any child span whose parent was sampled will also be sampled. This means distributed traces that start as sampled will be complete end-to-end. Un-sampled roots still propagate W3C traceparent headers, so downstream services can correlate via logs + trace context even when the trace itself is not exported.

Acceptance Criteria

Sampling is transparent by default (1.0 = sample everything, same as before).
Operators can tune ratio without rebuilding (env / appsettings).
Correlation preserved via ParentBasedSampler + W3C context propagation.
Smoke: API & workers start with 0, 1, and 0.1 without exceptions (covered by unit tests).

## Implementation Summary Configurable OpenTelemetry trace sampling has been implemented across the API and all 19 event handler workers. ### Changes **1. API — `SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs`** - `AddTelemetryConfiguration` now reads `OpenTelemetry:Tracing:SampleRatio` from config (default `1.0`). - Value is clamped to `[0.0, 1.0]` for safety. - A `ParentBasedSampler(new TraceIdRatioBasedSampler(ratio))` is wired into the `TracerProviderBuilder`. This ensures: - Root traces are sampled at the configured ratio. - Child spans inherit the sampling decision from their parent, preserving full distributed-trace correlation. **2. Workers — `SpikerSoft.EventHandlers.Infrastructure/Extensions/ServiceCollectionExtensions.cs`** - `AddEventHandlerTelemetry` follows the identical pattern: reads ratio → clamps → `ParentBasedSampler`. - All 19 workers that call `EventHandlerHostBuilder.WithTelemetry()` automatically inherit this behavior. **3. Configuration — `appsettings.json` (API + all workers)** - Added `OpenTelemetry.Tracing.SampleRatio` key with default `1.0` (transparent / no change in behavior). - Operators can override via env var: `OpenTelemetry__Tracing__SampleRatio=0.1` **4. Unit Tests — `SpikerSoft.Tests.Unit/Telemetry/TelemetrySamplingTests.cs`** - 14 new tests covering: - Valid ratios (0.0, 0.1, 0.5, 1.0) don't throw. - Missing config defaults to 1.0. - Out-of-range values (−1, 2, 100) are clamped without exceptions. - Ratio 0.0 marks traces as not sampled (`IsAllDataRequested = false`). - Ratio 1.0 records all traces (`IsAllDataRequested = true`). - `ParentBasedSampler` honors a sampled parent even when root ratio is 0.0. - Config value round-trips correctly. All 14 tests pass ✅ ### Correlation at < 100% Sampling Because `ParentBasedSampler` wraps the ratio sampler, any child span whose parent was sampled will also be sampled. This means distributed traces that start as sampled will be complete end-to-end. Un-sampled roots still propagate W3C `traceparent` headers, so downstream services can correlate via logs + trace context even when the trace itself is not exported. ### Acceptance Criteria - [x] Sampling is transparent by default (`1.0` = sample everything, same as before). - [x] Operators can tune ratio without rebuilding (env / appsettings). - [x] Correlation preserved via `ParentBasedSampler` + W3C context propagation. - [x] Smoke: API & workers start with `0`, `1`, and `0.1` without exceptions (covered by unit tests).

spikerj closed this issue

2026-05-14 17:57:19 +00:00

spikerj referenced this issue

2026-07-06 14:13:13 +00:00

[Security][Go-live] Pre-registration email verification codes generated with System.Random (predictable) #440

spikerj referenced this issue

2026-07-11 23:26:31 +00:00

[API/Angular] School-operator status surface — capability-level rollup (api/status + read-only panel), separate from the #297 admin console #377

Sign in to join this conversation.