[Spike] Stream code-execution output incrementally over the existing SignalR channel #306

New Issue

2026-07-01T15:29:40Z

spikerj commented

2026-07-01 15:29:40 +00:00

Problem

When code runs on the server path, stdout/stderr are captured to completion and returned as a single CodeExecutionResponse. For longer runs (clang compile+run, x86 emulation, slower Python/JS), the learner stares at a spinner until the whole run finishes, then sees all output at once. Incremental output would be a meaningful UX improvement.

Current state (encouraging — transport already exists)

The frontend runner already uses SignalR for execution: ClangRunnerService holds a HubConnection (libraries/platform/clang-runtime/src/lib/clang-runner.service.ts), with execution-state signals (isExecuting, currentExecutionId, lastResult).
Execution is a distributed worker flow: the API publishes a request; SpikerSoft.EventHandlers.CodeExecution/Services/CodeExecutionWorkerService.cs runs it and returns one response.
Executors run a Process with RedirectStandardOutput/Error and currently buffer to completion (ReadToEndAsync) — e.g. PythonLessonExecutor, RoslynCodeExecutor, etc.
Precedent for line-streaming already in the codebase: HostCommandRunner.StreamShellLinesAsync (used by host monitoring) streams process output line-by-line via BeginOutputReadLine.
Some runtimes already sidestep the server entirely via browser/WASM grading (ClangRuntimeService worker), which is the strongest scaling answer where feasible.

So the transport (SignalR), the async worker pipeline, and a streaming primitive all exist — what's missing is emitting incremental chunks instead of one final blob.

Spike deliverables

Confirm the end-to-end path for incremental output: executor emits chunks → worker relays over the bus → API hub pushes to the client → runner appends to an output signal. Identify every hop that currently assumes a single terminal response.
Decide the grading boundary: streamed stdout is display-only; the pass/fail assertion must still run on the complete captured output. Confirm streaming doesn't change grading correctness (assertions compare full output/locals).
Assess risk on the hot path: backpressure/chunk flooding, output caps (tie to existing max-output limits), cancellation/timeout behavior mid-stream, ordering, and interleaving stdout/stderr.
Scope for one pilot runtime (suggest Python or C# server path) behind a flag, with transparent fallback to the current single-response behavior.
Recommend whether to invest here vs. expanding WASM/browser grading for more runtimes (which removes the backend from the hot path altogether).

Why the eventual change should be non-breaking

Gate streaming behind a flag with fallback to the existing buffered response; runtimes/paths not opted in behave exactly as today.
Grading logic unchanged — streaming only adds a display channel for partial stdout; the final assertion still runs on complete output.
Reuses the existing SignalR hub + worker pipeline; no new transport.

Acceptance criteria (spike)

Written feasibility + risk assessment covering the hops above, output caps, cancellation, and grading-correctness boundary.
Recommendation: stream-output vs. expand-WASM-grading (or both), with a pilot-runtime plan behind a flag.
A throwaway POC proving incremental chunks reach the client for one runtime (does not need to be production-ready).
Follow-up implementation ticket(s) opened from the recommendation.

Priority

Medium/low — pick up when run latency (not content delivery) becomes the learner complaint. Largest scope of the current perf items and touches the execution hot path, hence spike-first.

(Local-hosting only — no CDN.)

## Problem When code runs on the **server** path, stdout/stderr are captured to completion and returned as a single `CodeExecutionResponse`. For longer runs (clang compile+run, x86 emulation, slower Python/JS), the learner stares at a spinner until the whole run finishes, then sees all output at once. Incremental output would be a meaningful UX improvement. ## Current state (encouraging — transport already exists) - The frontend runner **already uses SignalR** for execution: `ClangRunnerService` holds a `HubConnection` (`libraries/platform/clang-runtime/src/lib/clang-runner.service.ts`), with execution-state signals (`isExecuting`, `currentExecutionId`, `lastResult`). - Execution is a **distributed worker** flow: the API publishes a request; `SpikerSoft.EventHandlers.CodeExecution/Services/CodeExecutionWorkerService.cs` runs it and returns one response. - Executors run a `Process` with `RedirectStandardOutput/Error` and currently **buffer to completion** (`ReadToEndAsync`) — e.g. `PythonLessonExecutor`, `RoslynCodeExecutor`, etc. - Precedent for line-streaming already in the codebase: `HostCommandRunner.StreamShellLinesAsync` (used by host monitoring) streams process output line-by-line via `BeginOutputReadLine`. - Some runtimes already sidestep the server entirely via **browser/WASM grading** (`ClangRuntimeService` worker), which is the strongest scaling answer where feasible. So the transport (SignalR), the async worker pipeline, and a streaming primitive all exist — what's missing is emitting incremental chunks instead of one final blob. ## Spike deliverables 1. Confirm the end-to-end path for incremental output: executor emits chunks → worker relays over the bus → API hub pushes to the client → runner appends to an output signal. Identify every hop that currently assumes a single terminal response. 2. Decide the **grading boundary**: streamed stdout is display-only; the pass/fail assertion must still run on the complete captured output. Confirm streaming doesn't change grading correctness (assertions compare full output/locals). 3. Assess risk on the hot path: backpressure/chunk flooding, output caps (tie to existing max-output limits), cancellation/timeout behavior mid-stream, ordering, and interleaving stdout/stderr. 4. Scope for **one pilot runtime** (suggest Python or C# server path) behind a flag, with transparent fallback to the current single-response behavior. 5. Recommend whether to invest here vs. **expanding WASM/browser grading** for more runtimes (which removes the backend from the hot path altogether). ## Why the eventual change should be non-breaking - Gate streaming behind a flag with fallback to the existing buffered response; runtimes/paths not opted in behave exactly as today. - Grading logic unchanged — streaming only adds a display channel for partial stdout; the final assertion still runs on complete output. - Reuses the existing SignalR hub + worker pipeline; no new transport. ## Acceptance criteria (spike) - [ ] Written feasibility + risk assessment covering the hops above, output caps, cancellation, and grading-correctness boundary. - [ ] Recommendation: stream-output vs. expand-WASM-grading (or both), with a pilot-runtime plan behind a flag. - [ ] A throwaway POC proving incremental chunks reach the client for one runtime (does not need to be production-ready). - [ ] Follow-up implementation ticket(s) opened from the recommendation. ## Priority Medium/low — pick up when run **latency** (not content delivery) becomes the learner complaint. Largest scope of the current perf items and touches the execution hot path, hence spike-first. _(Local-hosting only — no CDN.)_

spikerj added the enhancement label 2026-07-01 15:29:40 +00:00

spikerj referenced this issue

2026-07-15 18:21:29 +00:00

[Security][CI] OpenBao Phase 1: Gitea Actions secrets → AppRole fetch; rotate the committed runner token (epic #543) #545

spikerj commented

2026-08-07 13:49:34 +00:00

Migrated to spikerj/spikersoft-backend#556 as part of the umbrella-tracker breakup.

Verified 2026-08-07. backend@98102023: Python/Clang lesson executors still ReadToEndAsync to completion; no streaming flag/chunk type anywhere; spike never run

Status: The whole spike: hop-by-hop feasibility, the grading-boundary decision, backpressure/caps/cancellation risk assessment, a flagged pilot runtime with a throwaway POC, the stream-vs-expand-WASM-grading recommendation, and the follow-up tickets. Routed here because every hop that must change first (executor chunk emission → worker relay → hub push) is .NET; the Angular ClangRunnerService already holds the HubConnection and only needs an output signal once the backend emits.

Closing here. Work now lives in the repo that holds the fix, so fixes #556 in a PR will auto-close it on merge. The umbrella tracker keeps cross-repo epics only.

— Opus 5 Agent

Migrated to **spikerj/spikersoft-backend#556** as part of the umbrella-tracker breakup. Verified 2026-08-07. backend@98102023: Python/Clang lesson executors still ReadToEndAsync to completion; no streaming flag/chunk type anywhere; spike never run Status: The whole spike: hop-by-hop feasibility, the grading-boundary decision, backpressure/caps/cancellation risk assessment, a flagged pilot runtime with a throwaway POC, the stream-vs-expand-WASM-grading recommendation, and the follow-up tickets. Routed here because every hop that must change first (executor chunk emission → worker relay → hub push) is .NET; the Angular `ClangRunnerService` already holds the `HubConnection` and only needs an output signal once the backend emits. Closing here. Work now lives in the repo that holds the fix, so `fixes #556` in a PR will auto-close it on merge. The umbrella tracker keeps cross-repo epics only. — Opus 5 Agent

spikerj closed this issue

2026-08-07 13:49:34 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#306