[Bug][Backend] BookAnalysisService (ExtractChapters / AnalyzeBookWithLLM) silently ships degraded book data on GPU/model-load failure — same class as quiz-gen #553, different pipeline #557
Assignees
spikerj (Joseph Spiker)
Clear assignees
spikerj (Joseph Spiker)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: spikerj/spikersoft-issues#557
Reference in New Issue
Block a user
QA Team — filed 2026-07-14 ~04:10Z at the request of the #553 fix work (see the PR #272 comment there), so the sibling bug doesn't get lost when #553 closes.
The bug: in the book-upload flow,
BookAnalysisService.ExtractChaptersandAnalyzeBookWithLLMcatch GPU/model-load failures (theLoadWeightsFailedException/ CUDA-alloc class proven live in #553) and silently fall back to degraded output instead of failing loudly or retrying. Result: books uploaded during GPU pressure get default/degraded chapter structure and analysis with no error surfaced anywhere — the same "silently ships bad data" pattern as #553 (quiz subjects) and the closed #507/#509 (quiz data quality).Why it matters tonight specifically: #553 documented 70 silent failures in 3h under image-description's VRAM squeeze. The quiz-gen side is now fixed (PR #272: honest lease size, unload-on-idle, PrefetchCount=1, loud failure) — but any book uploaded during a similar squeeze still gets silently degraded chapters/analysis until this service gets the same treatment.
Fix shape (mirror PR #272): route the model load through an honest gpu-coordinator lease; on failure, nack+retry or dead-letter with an ERR/alert — never proceed with defaults silently; unload on idle so the lease stays honest.
Verification path: tonight's live upload (book
6a55ad649b1d8650d3fccb5c, 22 pages / 182 images) is still mid-pipeline — if its chapter analysis runs while the 4090 is saturated by captioning, current code will hit exactly this fallback; QA is watching its logs as the natural repro.Refs: #553 (sibling fix, PR #272), #507/#509 (closed — same silent-degradation class in quiz data).
Diagnosis + fix in PR #275 (with #559 — same mechanism, one PR).
Root cause — three separate holes, all in
BookAnalysisService:ExtractChaptersWithLLMhad a blanketcatch (Exception) { return new List<Chapter>(); }. It swallowed the #553 CUDA-alloc /LoadWeightsFailedExceptionand returned empty. The caller could not distinguish that from "this book has no chapters", so it answered withCreateChaptersFromPageRanges— an invented every-20-pages structure — behind a Warning-severity notification. Exactly the fallback you predicted.ExtractChaptersnever calledInitializeModelAsync.ExtractChaptersWithLLMopened withif (_model == null) return new List<Chapter>();, so on any path where nothing else had warmed the model, chapter extraction always produced the synthetic structure — no GPU pressure required.AnalyzeBookWithLLMrethrew the raw LLama exception, whichRabbitMQRetryHelper.IsTransientErrorclassifies as non-transient. So it wasn't silent, but it was wrong: the message was dead-lettered instead of waiting for VRAM, and the failures never reached thegpu_resourcesmetric added in #553.Fix — mirrors PR #272 exactly:
ExtractChaptersandAnalyzeBookWithLLM→ classified by the existingGpuFailureClassifier,LogCritical, span statusError+ exception recorded, and wrapped inGpuResourceUnavailableExceptionso the message is retried (transient) rather than degraded or dead-lettered. NeverCreateChaptersFromPageRangeson an infrastructure failure.AI:MaxParseAttempts, default 3), then the newLlmOutputUnparsableException. A synthetic structure is never substituted for output we could not read.chapters.synthetic=true/chapters.source=page_range_fallback, so an invented structure is distinguishable from a real one in Jaeger.ChapterResponseParserreturns the failure reason as part of the result, and no longer invents"Chapter"titles /startPage: 1defaults for garbage entries — so garbage in can't look like chapters out. Unit-tested without a GPU.One thing you'll want to know:
ExtractChaptersandAnalyzeBookWithLLMcurrently have no production caller.BookManagementServicesetsChapters = [], and the only other references are the interface, the test mocks and the integration-test fake. So the degraded-chapters scenario can't actually be firing today — the chapter path looks dormant/orphaned, and tonight's book wouldn't have hit it. The service-level bugs are real and now fixed (and the quiz worker does use the same class'sExtractSubjectsFromPage, which #553 already hardened), but the lease-routing half of the ask is not wired, because there is no host calling these two methods to route through the coordinator. Worth a decision: re-wire chapter extraction into the book pipeline, or retire it. Happy to do either as a follow-up.SpikerSoft.EventHandlers.QuizGeneration.Tests: 43/43 green (18 new).— macbook-claude-session
Status check after PR #275 (merged to
masterat 06:31Z,b3287f5).The silent-degradation half is fixed and shipped. Verified on
master: the blanketcatch (Exception) { return new List<Chapter>(); }is gone,ExtractChaptersnow actually callsInitializeModelAsync(it never did — so on any path where nothing else had warmed the model it always produced the synthetic every-20-pages structure), GPU/model-load failures raiseGpuResourceUnavailableException(retryable, per the #553 contract) instead of being swallowed, and unreadable output raisesLlmOutputUnparsableExceptionrather than substituting invented chapters. A genuinely chapterless book still gets its page-range fallback, but the span is taggedchapters.synthetic=trueso Jaeger can tell an invented structure from a real one.The lease-routing half cannot be done as written, because this ticket's premise doesn't hold.
This ticket says "books uploaded during GPU pressure get degraded chapter structure and analysis". That is not happening, and never has:
ExtractChaptersandAnalyzeBookWithLLMhave no production caller at all.BookManagementService.cs:84hardcodesChapters = [], nothing writes that field afterwards, andGET /books/{id}/chaptershas been returning200 OK []for every book since it was written. The degradation you set out to stop was never reachable — no book has ever entered that code path.So the service-level bug was real and worth fixing (it was one DI registration away from being live, and hardened GPU-touching code sitting inert is exactly the #573 trap), but there is no host to route a gpu-coordinator lease through until something actually calls it.
I've filed #593 with the full evidence chain. It carries the decision this now depends on: wire chapter extraction into the book pipeline (an
IRemoteCommandhop into an AI worker taking an honest GPU lease — naive wiring would reproduce #553 on the upload path), or retire it (and drop the endpoint rather than leave an API that confidently lies).I deliberately didn't guess between those — (a) is a real feature build, (b) deletes a capability someone may want. Leaving this open and blocked on #593 rather than closing it as "done", since its stated goal isn't achieved. Happy to take either direction.
QA verification — closing. Fix confirmed in current master (HEAD bc55a9ff).
BookAnalysisServiceno longer ships degraded data on GPU/model-load failure: bothAnalyzeBookWithLLM(BookAnalysisService.cs:124-146) andExtractChapters(lines 239-261) now catch the GPU-failure class viaGpuFailureClassifier.IsGpuResourceFailure,LogCritical, andthrow new GpuResourceUnavailableException(...)— explicitly REFUSING to fall back to a synthetic page-range structure. Unparsable LLM output likewise throwsLlmOutputUnparsableExceptionrather than returning empty. Landed in PR #275 (commit 0153a44d, verified ancestor of master). Test-covered byLlmOutputParseFailureTestsandGpuLoadFailureRoutingTests. (Note: this is the code-behavior fix the ticket asked for; whether ExtractChapters is wired into the upload pipeline is the separate, still-open #593.)