[Bug][Backend][Quality] QuizGenerationService validation is fail-open — if the validation LLM call throws, the unvalidated question ships in the quiz #564
Assignees
spikerj (Joseph Spiker)
Clear assignees
spikerj (Joseph Spiker)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: spikerj/spikersoft-issues#564
Reference in New Issue
Block a user
QA Team — from the README audit's code verification, triaged with spikerj 2026-07-14 (decision: file).
The bug: each generated question is supposed to be re-validated by a second LLM pass that rejects questions with multiple defensible answers — but the validation is fail-open: if the validation call itself throws (
QuizGenerationService.cs~707-711 in SpikerSoft.Business.Ai.Workers), the question is INCLUDED anyway. Under GPU pressure (exactly when LLM calls are most likely to throw — see the #553 saga) the quality gate silently disappears, so the quizzes most likely to contain bad questions are the ones generated under load.Fix shape: fail-closed (drop the question and log WRN), or retry the validation once then fail-closed; either way count validation-skips in the completion notification so shrinkage is visible (pairs with #559's per-page counts ask).
Refs: #559 (sibling silent-shrinkage bug — JSON parse failures), #553 (closed — the GPU-pressure context that makes fail-open dangerous).
Fix in backend PR #283 — took the "retry once then fail-closed" shape, and the completion-notification count, because the second one turned out not to be optional.
The gate now fails closed. A question that can't be validated is dropped, after a bounded retry (
AI:MaxValidationAttempts, default 2) so a single transient blip doesn't cost a good question.But a GPU failure is not a verdict. This is the part that needed care: if validation throws because VRAM vanished, dropping every question would turn a retryable message into a permanently short quiz — swapping your silent degradation for a different one. So GPU/VRAM failures propagate unretried and the page re-runs when VRAM frees up, exactly as the generation path already does (#553). Two of the new tests exist only to pin that boundary.
(Aside for whoever touches this next:
GpuFailureClassifier.IsGpuResourceFailureonly recognizes the raw LLamaSharp exceptions, not an already-wrappedGpuResourceUnavailableException. My first cut filtered on the classifier alone and the tests caught it.)Why the notification count came along. Fail-closed means a page can now hand back fewer questions than the model generated — so fixing the fail-open without reporting the drop would just have replaced one silent shrinkage with another. So:
GenerateQuestionsFromTextWithMetadatareturnsQuizPageGeneration(questions +UnvalidatedDropCount);Quiz.UnvalidatedQuestionDropCountpersists the tally (a run resumed after a #553 VRAM retry reports the whole shortfall, not just the post-retry part); andDecideCompletionOutcometreats drops exactly like #559's unreadable pages — the quiz is Degraded, never a green ✅ Completed, and the notification names the fault.A rejection (validator found two defensible answers) stays counted separately from a drop (validation never completed). Rejections are the gate working; only drops are shrinkage nobody asked for. Conflating them would have made the counter meaningless.
Tests: 7 xUnit on the accept/reject/drop policy + the GPU rule (the LLM call is injected as a delegate, so the real policy runs without a model), 5 on the completion decision and the user-facing shortfall wording, and 1 integration test driving a drop through the real worker over the real broker into the persisted record and the student's notification. These do fail before the fix.
This also delivers the per-page shrinkage visibility that #559 asked for as its item 4 — #559's items 1-3 (retry / grammar-constrained decoding / partial salvage) are untouched and still live there.
Leaving open for close after merge.
Resolved in spikersoft-backend PR #283 (merged to
masterasd633e5d). The validation gate now fails closed (drop the question after a bounded retry,AI:MaxValidationAttempts), GPU/VRAM failures propagate unretried so the page is re-run rather than the quiz truncated, and the resulting shrinkage is persisted on the quiz record (Quiz.UnvalidatedQuestionDropCount), marks the quiz Degraded, and is named in the student's completion notification. Closing.