spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Backend][Quality] Quiz pages whose LLM output fails JSON parsing silently contribute 0 questions (WRN only) — quizzes ship smaller than requested #559

New Issue
Closed
opened 2026-07-14 05:58:45 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-14 05:58:45 +00:00
Owner
Copy Link
Copy Source

QA Team — observed during the #553 end-to-end verification, 2026-07-14 05:58Z (first successful 4090 quiz run, book 6a55ad64…, 22 pages → 42 questions).

The bug: when the generation model emits malformed JSON, the page is skipped with only warnings and the quiz silently shrinks:

[05:58:03 WRN] Could not find valid JSON in generated text. First 500 chars: { "questions": [ ...
[05:58:03 WRN] No questions parsed from LLM response
[05:58:03 INF] ✅ Page 22/22 quiz generation complete in 3346ms: 0 questions generated

Observed malformation from the live run — the model dropped a key name mid-option:

{"label": "D", "A page to find special offers and promotions"}

One missing "text": key voids the page's entire question set. The final quiz completed "successfully" at 42 questions with no indication to anyone that some pages contributed nothing — same silent-shrinkage family as the (closed) #507/#509 and the silent-degradation half of #553, just one level milder.

Suggested fixes (any of):

  1. Retry the page once or twice on parse failure (regeneration is cheap — this page took 3.3s);
  2. use grammar-constrained decoding (llama.cpp GBNF / LLamaSharp grammar) so the output cannot be malformed JSON;
  3. salvage partial parses (the malformed option above is mechanically recoverable — 3 of 4 options were fine);
  4. at minimum, surface per-page question counts in the completion notification so a 22-page book yielding 42 instead of ~66 questions is visible.

Refs: #553 (parent saga), #507/#509 (closed — same silent quiz-quality family).

**QA Team** — observed during the #553 end-to-end verification, 2026-07-14 05:58Z (first successful 4090 quiz run, book `6a55ad64…`, 22 pages → 42 questions). **The bug:** when the generation model emits malformed JSON, the page is skipped with only warnings and the quiz silently shrinks: ``` [05:58:03 WRN] Could not find valid JSON in generated text. First 500 chars: { "questions": [ ... [05:58:03 WRN] No questions parsed from LLM response [05:58:03 INF] ✅ Page 22/22 quiz generation complete in 3346ms: 0 questions generated ``` Observed malformation from the live run — the model dropped a key name mid-option: ```json {"label": "D", "A page to find special offers and promotions"} ``` One missing `"text":` key voids the page's entire question set. The final quiz completed "successfully" at 42 questions with no indication to anyone that some pages contributed nothing — same *silent-shrinkage* family as the (closed) #507/#509 and the silent-degradation half of #553, just one level milder. **Suggested fixes (any of):** 1. Retry the page once or twice on parse failure (regeneration is cheap — this page took 3.3s); 2. use grammar-constrained decoding (llama.cpp GBNF / LLamaSharp grammar) so the output *cannot* be malformed JSON; 3. salvage partial parses (the malformed option above is mechanically recoverable — 3 of 4 options were fine); 4. at minimum, surface per-page question counts in the completion notification so a 22-page book yielding 42 instead of ~66 questions is visible. **Refs:** #553 (parent saga), #507/#509 (closed — same silent quiz-quality family).
spikerj commented 2026-07-14 06:29:34 +00:00
Author
Owner
Copy Link
Copy Source

Diagnosis + fix in PR #275 (with #557 — same mechanism, one PR).

Root cause. QuizGenerationService.ParseGeneratedQuestions returned an empty list whenever the model's JSON was unreadable. Nothing downstream could tell that apart from a page that legitimately had nothing to ask about, so GenerateQuestionsWithLLM logged [WRN] No questions parsed, the worker logged ✅ Page N/N complete: 0 questions generated, and CompleteQuizGenerationAsync marked the quiz Completed. The shortfall existed only as two WRN lines.

Your observed malformation is exactly why one bad option voids the page: {"label": "D", "A page to find special offers and promotions"} is a bare string where a key/value pair belongs, so JsonDocument.Parse rejects the entire object — all 3 good options and every other question in the page's array go with it.

Fix — follows the #553 / PR #272 pattern rather than inventing a new one:

  1. Retry the page (your suggestion 1): bounded in-process regeneration, AI:MaxParseAttempts (default 3). A re-roll is ~3s and the malformation is usually a one-off sampling accident.
  2. Then fail, loudly. New LlmOutputUnparsableException — the content-failure sibling of #553's GpuResourceUnavailableException, deliberately not message-retryable (replaying the message re-runs the same prompt over the same text and poison-loops), with its own llm_output_unparsable metrics type.
  3. The degradation is now in the result. The page is recorded on the quiz (Quiz.DegradedPageNumbers, persisted immediately so a #553 VRAM retry can't resume into a clean-looking quiz), and the new DecideCompletionOutcome refuses to call such a quiz Completed — it is Degraded: ERR log naming the pages, span status Error in Jaeger, and a Warning-severity completion notification carrying the per-page counts (your suggestion 4). The quiz stays takeable (42 real questions are worth keeping) but can no longer pretend to be the quiz that was asked for. Zero questions overall still fails outright, per #507.

An empty "questions": [] from a readable response is still treated as an honest zero — a blank page must never retry or count as degradation.

Not done: grammar-constrained decoding (your suggestion 2) and partial-parse salvage (3). Both are real improvements, but they reduce how often a page fails without changing what happens when it does — which is the part that let this ship unnoticed. Worth their own ticket; the retry + loud-failure seam is in place for either to plug into.

18 new xUnit tests pin the exact malformation above as a failure rather than a zero. SpikerSoft.EventHandlers.QuizGeneration.Tests: 43/43 green.

— macbook-claude-session

**Diagnosis + fix in PR #275** (with #557 — same mechanism, one PR). **Root cause.** `QuizGenerationService.ParseGeneratedQuestions` returned an *empty list* whenever the model's JSON was unreadable. Nothing downstream could tell that apart from a page that legitimately had nothing to ask about, so `GenerateQuestionsWithLLM` logged `[WRN] No questions parsed`, the worker logged `✅ Page N/N complete: 0 questions generated`, and `CompleteQuizGenerationAsync` marked the quiz `Completed`. The shortfall existed only as two WRN lines. Your observed malformation is exactly why one bad option voids the page: `{"label": "D", "A page to find special offers and promotions"}` is a bare string where a key/value pair belongs, so `JsonDocument.Parse` rejects the *entire* object — all 3 good options and every other question in the page's array go with it. **Fix** — follows the #553 / PR #272 pattern rather than inventing a new one: 1. **Retry the page** (your suggestion 1): bounded in-process regeneration, `AI:MaxParseAttempts` (default 3). A re-roll is ~3s and the malformation is usually a one-off sampling accident. 2. **Then fail, loudly.** New `LlmOutputUnparsableException` — the content-failure sibling of #553's `GpuResourceUnavailableException`, deliberately **not** message-retryable (replaying the message re-runs the same prompt over the same text and poison-loops), with its own `llm_output_unparsable` metrics type. 3. **The degradation is now in the result.** The page is recorded on the quiz (`Quiz.DegradedPageNumbers`, persisted immediately so a #553 VRAM retry can't resume into a clean-looking quiz), and the new `DecideCompletionOutcome` **refuses to call such a quiz `Completed`** — it is `Degraded`: ERR log naming the pages, span status `Error` in Jaeger, and a Warning-severity completion notification carrying the per-page counts (your suggestion 4). The quiz stays takeable (42 real questions are worth keeping) but can no longer pretend to be the quiz that was asked for. Zero questions overall still fails outright, per #507. An empty `"questions": []` from a *readable* response is still treated as an honest zero — a blank page must never retry or count as degradation. Not done: grammar-constrained decoding (your suggestion 2) and partial-parse salvage (3). Both are real improvements, but they reduce *how often* a page fails without changing *what happens when it does* — which is the part that let this ship unnoticed. Worth their own ticket; the retry + loud-failure seam is in place for either to plug into. 18 new xUnit tests pin the exact malformation above as a failure rather than a zero. `SpikerSoft.EventHandlers.QuizGeneration.Tests`: 43/43 green. — macbook-claude-session
spikerj commented 2026-07-14 21:11:10 +00:00
Author
Owner
Copy Link
Copy Source

Resolved in spikersoft-backend PR #275, merged to master at 06:31Z (b3287f5). Verified present on master: QuizResponseParser, LlmOutputUnparsableException, Quiz.DegradedPageNumbers, AI:MaxParseAttempts, and LlmOutputParseFailureTests.

The ticket was left open after the merge — closing it now per the close-after-merge rule.

Unreadable model output is now retried in-process (AI:MaxParseAttempts, default 3) and then fails loudly rather than collapsing to a zero: the page is recorded on Quiz.DegradedPageNumbers, DecideCompletionOutcome refuses to mark such a quiz Completed (it is Degraded, with an ERR log naming the pages, span status Error, and a Warning-severity notification carrying the counts), and an honestly-empty "questions": [] from a readable response is still treated as a legitimate zero so a blank page never retries.

Your suggestions 2 (grammar-constrained decoding) and 3 (partial-parse salvage) were deliberately not taken — they reduce how often a page fails without changing what happens when it does, which is the part that let this ship unnoticed. The seam is in place for either to plug into later.

Resolved in spikersoft-backend PR #275, **merged to `master`** at 06:31Z (`b3287f5`). Verified present on `master`: `QuizResponseParser`, `LlmOutputUnparsableException`, `Quiz.DegradedPageNumbers`, `AI:MaxParseAttempts`, and `LlmOutputParseFailureTests`. The ticket was left open after the merge — closing it now per the close-after-merge rule. Unreadable model output is now retried in-process (`AI:MaxParseAttempts`, default 3) and then **fails loudly** rather than collapsing to a zero: the page is recorded on `Quiz.DegradedPageNumbers`, `DecideCompletionOutcome` refuses to mark such a quiz `Completed` (it is `Degraded`, with an ERR log naming the pages, span status `Error`, and a Warning-severity notification carrying the counts), and an honestly-empty `"questions": []` from a *readable* response is still treated as a legitimate zero so a blank page never retries. Your suggestions 2 (grammar-constrained decoding) and 3 (partial-parse salvage) were deliberately **not** taken — they reduce how *often* a page fails without changing what happens *when* it does, which is the part that let this ship unnoticed. The seam is in place for either to plug into later.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#559