[Bug][Prod][GPU] image-description fails every page — idle-unload races in-flight inference, batch_decode on a None processor #614

New Issue

2026-07-16T07:09:06Z

spikerj commented

2026-07-16 07:09:06 +00:00

Summary

spikersoft-image-description (Qwen3-VL Python worker) fails every page with 'NoneType' object has no attribute 'batch_decode' whenever a request arrives after the 120s idle-unload. The idle-unload timer races with in-flight inference: it nulls the model/processor while a request is being served (or before the handler reloads), so self.processor.batch_decode(...) hits None.

Live evidence (4090 container, 2026-07-16)

06:55:46  Idle timeout (120s) reached -- unloading model and releasing GPU lease
06:56:51  ERROR Error generating description after 207212ms: 'NoneType' object has no attribute 'batch_decode'
          Traceback ... output_text = self.processor.batch_decode(
06:57:01  Acquiring GPU lease ... Model not loaded, warming up ... Vision model loaded successfully
06:59:46  Idle timeout (120s) reached -- unloading model ...
07:00:59  ERROR Error generating description after 228508ms: 'NoneType' object has no attribute 'batch_decode'

The cycle repeats: load -> idle-unload (processor=None) -> next request calls batch_decode on None -> fail -> reload AFTER the failure -> idle-unload -> fail ... Requests take ~3.5 min each and never succeed.

Impact

Blocks book ingestion at the image-description stage — a real user book (upload bd69711f, book 6a587f4b, 50 pages) is stuck here and never reaches embeddings/quiz generation. NOTE: model fetch is fine — the weights self-provision correctly from the ai-models MinIO bucket (MODEL_SOURCE=s3); this is purely a load/unload lifecycle bug.

Likely fix

Guard the idle-unload against in-flight inference (hold the model lease / a lock for the duration of a request), and/or ensure the request handler reloads the model BEFORE calling batch_decode (reload-then-infer, and retry the current request after a reload instead of only the next one). Also null-guard self.processor before use.

## Summary `spikersoft-image-description` (Qwen3-VL Python worker) fails every page with **`'NoneType' object has no attribute 'batch_decode'`** whenever a request arrives after the 120s idle-unload. The idle-unload timer races with in-flight inference: it nulls the model/processor while a request is being served (or before the handler reloads), so `self.processor.batch_decode(...)` hits `None`. ## Live evidence (4090 container, 2026-07-16) ``` 06:55:46 Idle timeout (120s) reached -- unloading model and releasing GPU lease 06:56:51 ERROR Error generating description after 207212ms: 'NoneType' object has no attribute 'batch_decode' Traceback ... output_text = self.processor.batch_decode( 06:57:01 Acquiring GPU lease ... Model not loaded, warming up ... Vision model loaded successfully 06:59:46 Idle timeout (120s) reached -- unloading model ... 07:00:59 ERROR Error generating description after 228508ms: 'NoneType' object has no attribute 'batch_decode' ``` The cycle repeats: load -> idle-unload (processor=None) -> next request calls batch_decode on None -> fail -> reload AFTER the failure -> idle-unload -> fail ... Requests take ~3.5 min each and never succeed. ## Impact Blocks book ingestion at the image-description stage — a real user book (upload bd69711f, book 6a587f4b, 50 pages) is stuck here and never reaches embeddings/quiz generation. NOTE: model fetch is fine — the weights self-provision correctly from the `ai-models` MinIO bucket (MODEL_SOURCE=s3); this is purely a load/unload lifecycle bug. ## Likely fix Guard the idle-unload against in-flight inference (hold the model lease / a lock for the duration of a request), and/or ensure the request handler reloads the model BEFORE calling batch_decode (reload-then-infer, and retry the current request after a reload instead of only the next one). Also null-guard `self.processor` before use.

spikerj commented

2026-07-16 07:12:56 +00:00

Correction on severity: it is intermittent, not total. When pages flow steadily the model stays loaded and descriptions succeed (~1-2 min each). The failure drops only the first request after each 120s idle-unload — one lost page per idle gap — then the handler reloads and continues. Observed live: book 6a587f4b progressed through ~11+ pages with occasional single-page batch_decode drops, not a full stall. Fix still stands: guard idle-unload against in-flight/next inference and reload-before-decode.

Correction on severity: it is **intermittent, not total**. When pages flow steadily the model stays loaded and descriptions succeed (~1-2 min each). The failure drops only the **first request after each 120s idle-unload** — one lost page per idle gap — then the handler reloads and continues. Observed live: book 6a587f4b progressed through ~11+ pages with occasional single-page batch_decode drops, not a full stall. Fix still stands: guard idle-unload against in-flight/next inference and reload-before-decode.

spikerj commented

2026-07-22 02:46:02 +00:00

Verified complete (ticket-hygiene sweep): backend PR #324 merged — image_description_service.py now guards the model/processor with locking against the idle-unload race (20 lock references on master) and ships a dedicated tests/test_unload_race.py regression suite. 1:1 with the reported NoneType batch_decode failure. Closing.

spikerj closed this issue

2026-07-22 02:46:03 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#614