spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Prod][GPU] image-description permanently dead-letters captions on null gpu_client — AttributeError misclassified as permanent (data loss) #506

New Issue
Closed
opened 2026-07-12 17:35:43 +00:00 by spikerj · 4 comments
spikerj commented 2026-07-12 17:35:43 +00:00
Owner
Copy Link
Copy Source

Summary

The image-description worker (Qwen3-VL captioner) permanently dead-letters image messages when its GPU-lease client is momentarily None — e.g. a message arrives during the startup window before _gpu_client is initialized, or the GpuCoordinator is transiently unreachable at init. The failure surfaces as:

ERROR - GPU lease / model loading failed: 'NoneType' object has no attribute 'acquire'
Traceback ... image_description_service.py:810 in _ensure_gpu_lease
    self._gpu_lease = self._gpu_client.acquire()
AttributeError: 'NoneType' object has no attribute 'acquire'
CRITICAL - Message rejected (permanent failure): 'NoneType' object has no attribute 'acquire'

Net effect: a transient condition causes permanent loss of the caption — the image is nack'd requeue=False to the DLQ and never retried, so those book pages are silently left un-captioned.

Confirmed casualties (2026-07-12 07:33:01, node 4090, container 6ff80baa36b5): images 6a53371529759f8302ced93d and 6a53371529759f8302ced93c were dead-lettered this way.

Root cause — two coupled defects

1. Startup ordering race. The RabbitMQ consumer begins delivering before _gpu_client is assigned. From the 07:33 logs, the order is:

07:33:01,393  Starting to consume from queue: image.description.requested
07:33:01,394  Received message for image: 6a53371529759f8302ced93d
07:33:01,395  GPU lease / model loading failed: 'NoneType' ... acquire   <-- _gpu_client still None
07:33:01,395  Image Description EventHandler started successfully         <-- init finishes AFTER

self._gpu_client = create_gpu_lease_client(conn_params) (image_description_service.py:551) runs in the async startup path, but consumption is already live, so the first message(s) hit acquire() on a None client. (A transiently-unreachable GpuCoordinator — e.g. during a SERVER flap, see #503 — could produce the same None client outside the race window.)

2. AttributeError is misclassified as a permanent failure.

# image_description_service.py:1004
_PERMANENT_FAILURES = (ValueError, KeyError, FileNotFoundError, AttributeError)

_handle_message_failure (:1006) treats anything in that tuple as permanent → basic_nack(requeue=False) straight to DLQ, no retry. A 'NoneType' … acquire AttributeError is a transient/programming-state error, never a legitimate "permanent business failure" like record-not-found — but it's bucketed with them and the message is lost.

Contrast: the TimeoutError path (:804) correctly does basic_nack(requeue=True).

Fix options

  1. Reclassify — remove AttributeError from _PERMANENT_FAILURES (keep ValueError/KeyError/FileNotFoundError for genuine bad-payload/not-found). An unexpected AttributeError should retry, not dead-letter.
  2. Guard the null client — in _ensure_gpu_lease, if self._gpu_client is None, raise a dedicated retryable error (or basic_nack(requeue=True) directly) so a not-yet-ready/temporarily-unreachable coordinator is retried, not permanently dropped.
  3. Fix startup ordering — don't register the consumer / don't begin delivering until _gpu_client is initialized (await create_gpu_lease_client before channel.basic_consume), closing the race window entirely.
  4. Recover the lost images — the two dead-lettered image IDs (and any others in the DLQ from this window) should be re-driven so those pages get captioned.

Recommend (1)+(3) as the durable fix; (2) as defense-in-depth; (4) to recover current data.

Impact

Image-bearing book pages whose caption request lands during a worker restart or a coordinator blip are permanently left without descriptions, with no error surfaced to the user (the book otherwise completes). Silent, partial data loss on the captioning path.

Related

  • #502 — image-description worker, HF gated model (different failure, same service).
  • #500 — gpu-coordinator lease budget (the client this bug fails to use).
  • #503 — SERVER flaps that can make the coordinator transiently unreachable → the non-race path into this bug.
  • Likely surfaced by the recent DLQ/retry refactor that introduced _PERMANENT_FAILURES classification (see git log on image_description_service.py — sonarqube batches / "backend changes for breaking files out").

Fix must land with tests (per repo policy)

  • xunit/pytest: a message processed while _gpu_client is None must be requeued/retried, not DLQ'd; AttributeError must not be treated as permanent.
  • Startup test: consumer does not receive deliveries until _gpu_client is non-None.
## Summary The `image-description` worker (Qwen3-VL captioner) **permanently dead-letters image messages** when its GPU-lease client is momentarily `None` — e.g. a message arrives during the startup window before `_gpu_client` is initialized, or the GpuCoordinator is transiently unreachable at init. The failure surfaces as: ``` ERROR - GPU lease / model loading failed: 'NoneType' object has no attribute 'acquire' Traceback ... image_description_service.py:810 in _ensure_gpu_lease self._gpu_lease = self._gpu_client.acquire() AttributeError: 'NoneType' object has no attribute 'acquire' CRITICAL - Message rejected (permanent failure): 'NoneType' object has no attribute 'acquire' ``` Net effect: a **transient** condition causes **permanent loss** of the caption — the image is nack'd `requeue=False` to the DLQ and never retried, so those book pages are silently left un-captioned. **Confirmed casualties (2026-07-12 07:33:01, node 4090, container 6ff80baa36b5):** images `6a53371529759f8302ced93d` and `6a53371529759f8302ced93c` were dead-lettered this way. ## Root cause — two coupled defects **1. Startup ordering race.** The RabbitMQ consumer begins delivering before `_gpu_client` is assigned. From the 07:33 logs, the order is: ``` 07:33:01,393 Starting to consume from queue: image.description.requested 07:33:01,394 Received message for image: 6a53371529759f8302ced93d 07:33:01,395 GPU lease / model loading failed: 'NoneType' ... acquire <-- _gpu_client still None 07:33:01,395 Image Description EventHandler started successfully <-- init finishes AFTER ``` `self._gpu_client = create_gpu_lease_client(conn_params)` (`image_description_service.py:551`) runs in the async startup path, but consumption is already live, so the first message(s) hit `acquire()` on a `None` client. (A transiently-unreachable GpuCoordinator — e.g. during a SERVER flap, see #503 — could produce the same `None` client outside the race window.) **2. `AttributeError` is misclassified as a permanent failure.** ```python # image_description_service.py:1004 _PERMANENT_FAILURES = (ValueError, KeyError, FileNotFoundError, AttributeError) ``` `_handle_message_failure` (`:1006`) treats anything in that tuple as permanent → `basic_nack(requeue=False)` straight to DLQ, no retry. A `'NoneType' … acquire` AttributeError is a transient/programming-state error, **never** a legitimate "permanent business failure" like record-not-found — but it's bucketed with them and the message is lost. Contrast: the `TimeoutError` path (`:804`) correctly does `basic_nack(requeue=True)`. ## Fix options 1. **Reclassify** — remove `AttributeError` from `_PERMANENT_FAILURES` (keep `ValueError`/`KeyError`/`FileNotFoundError` for genuine bad-payload/not-found). An unexpected `AttributeError` should retry, not dead-letter. 2. **Guard the null client** — in `_ensure_gpu_lease`, if `self._gpu_client is None`, raise a dedicated *retryable* error (or `basic_nack(requeue=True)` directly) so a not-yet-ready/temporarily-unreachable coordinator is retried, not permanently dropped. 3. **Fix startup ordering** — don't register the consumer / don't begin delivering until `_gpu_client` is initialized (await `create_gpu_lease_client` before `channel.basic_consume`), closing the race window entirely. 4. **Recover the lost images** — the two dead-lettered image IDs (and any others in the DLQ from this window) should be re-driven so those pages get captioned. Recommend (1)+(3) as the durable fix; (2) as defense-in-depth; (4) to recover current data. ## Impact Image-bearing book pages whose caption request lands during a worker restart or a coordinator blip are permanently left without descriptions, with no error surfaced to the user (the book otherwise completes). Silent, partial data loss on the captioning path. ## Related - #502 — image-description worker, HF gated model (different failure, same service). - #500 — gpu-coordinator lease budget (the client this bug fails to use). - #503 — SERVER flaps that can make the coordinator transiently unreachable → the non-race path into this bug. - Likely surfaced by the recent DLQ/retry refactor that introduced `_PERMANENT_FAILURES` classification (see `git log` on `image_description_service.py` — sonarqube batches / "backend changes for breaking files out"). ## Fix must land with tests (per repo policy) - xunit/pytest: a message processed while `_gpu_client is None` must be **requeued/retried**, not DLQ'd; `AttributeError` must not be treated as permanent. - Startup test: consumer does not receive deliveries until `_gpu_client` is non-None.
spikerj commented 2026-07-12 17:39:44 +00:00
Author
Owner
Copy Link
Copy Source

Fix up in spikersoft-backend PR #229, implementing exactly the recommended combination — (1)+(3) durable, (2) defense-in-depth:

  1. AttributeError is no longer a permanent failure. Classification extracted to a torch-free message_failure.py (is_permanent_failure()), so it's unit-testable without the transformers stack; ValueError/KeyError/FileNotFoundError remain DLQ-permanent.
  2. _ensure_gpu_lease now raises a dedicated retryable GpuClientNotReadyError when the client is None — the coordinator-unreachable path (#503) retries instead of dead-lettering.
  3. Startup reordered: create_gpu_lease_client runs before _init_rabbitmq() starts the consumer — the delivery-before-init race is structurally gone (client construction is lazy, no connection until first acquire, so no new startup dependency).
  4. Recovery: the two 07:33 casualties (6a53371529759f8302ced93d/…93c) belonged to books deleted later that morning, and current books' caption requests were re-published by the staleness retrier — no DLQ shoveling required.

Tests per the ticket's requirement: 6 pytest cases (classification matrix incl. the exact 'NoneType…acquire' repro + Timeout/generic-retry; source-order tripwires for the ordering and the None guard — static because importing the service module requires torch). CI now runs the suite before every image build. 6/6 green locally.

Merging auto-deploys the worker. Will close after the next caption run confirms no 'Message rejected (permanent failure)' on transient conditions.

Excellent diagnosis by QA — the log-ordering evidence (consume at :393, init-complete at :395) made this a read-and-fix.

Fix up in **spikersoft-backend PR #229**, implementing exactly the recommended combination — (1)+(3) durable, (2) defense-in-depth: 1. `AttributeError` is no longer a permanent failure. Classification extracted to a torch-free `message_failure.py` (`is_permanent_failure()`), so it's unit-testable without the transformers stack; `ValueError`/`KeyError`/`FileNotFoundError` remain DLQ-permanent. 2. `_ensure_gpu_lease` now raises a dedicated retryable `GpuClientNotReadyError` when the client is None — the coordinator-unreachable path (#503) retries instead of dead-lettering. 3. Startup reordered: `create_gpu_lease_client` runs **before** `_init_rabbitmq()` starts the consumer — the delivery-before-init race is structurally gone (client construction is lazy, no connection until first acquire, so no new startup dependency). 4. Recovery: the two 07:33 casualties (6a53371529759f8302ced93d/…93c) belonged to books deleted later that morning, and current books' caption requests were re-published by the staleness retrier — no DLQ shoveling required. Tests per the ticket's requirement: 6 pytest cases (classification matrix incl. the exact 'NoneType…acquire' repro + Timeout/generic-retry; source-order tripwires for the ordering and the None guard — static because importing the service module requires torch). **CI now runs the suite before every image build.** 6/6 green locally. Merging auto-deploys the worker. Will close after the next caption run confirms no 'Message rejected (permanent failure)' on transient conditions. Excellent diagnosis by QA — the log-ordering evidence (consume at :393, init-complete at :395) made this a read-and-fix.
spikerj commented 2026-07-12 17:42:54 +00:00
Author
Owner
Copy Link
Copy Source

⚠️ The fix for this is currently deployed but BROKEN — filed #508. The pushed image-description :latest (UpdatedAt 17:40 UTC) imports from message_failure import GpuClientNotReadyError, is_permanent_failure (image_description_service.py:33) but the new message_failure.py module isn't in the image (Dockerfile COPYs files individually, no COPY message_failure.py; module also not in origin/master). Result: ModuleNotFoundError → crash-loop → image-description fully 0/1. Good news: GpuClientNotReadyError + is_permanent_failure are exactly the retryable-error + reclassification this ticket recommended. Needs the module committed to master + added to the Dockerfile COPY, then rebuild. See #508.

⚠️ The fix for this is currently deployed but BROKEN — filed #508. The pushed image-description :latest (UpdatedAt 17:40 UTC) imports `from message_failure import GpuClientNotReadyError, is_permanent_failure` (image_description_service.py:33) but the new `message_failure.py` module isn't in the image (Dockerfile COPYs files individually, no `COPY message_failure.py`; module also not in origin/master). Result: `ModuleNotFoundError` → crash-loop → image-description fully 0/1. Good news: GpuClientNotReadyError + is_permanent_failure are exactly the retryable-error + reclassification this ticket recommended. Needs the module committed to master + added to the Dockerfile COPY, then rebuild. See #508.
spikerj commented 2026-07-12 17:53:09 +00:00
Author
Owner
Copy Link
Copy Source

✅ Fixed & verified — closing. Landed on origin/master:

  • New torch-free message_failure.py: GpuClientNotReadyError (transient/retryable), PERMANENT_FAILURES=(ValueError,KeyError,FileNotFoundError) with AttributeError removed (defect 2), is_permanent_failure().
  • image_description_service.py: null-client guard :817 if self._gpu_client is None: raise GpuClientNotReadyError(...); classifier wired at :1198 is_permanent=is_permanent_failure(exception); GPU-lease client now created before the consumer starts (defect 1, startup race).
  • Tests: tests/test_message_failure.py — I ran all 6 against master sources, 6 passed / 0 failed (AttributeError-not-permanent, GpuClientNotReadyError-retryable, genuine permanents still DLQ, generic/timeout retry, client-before-consumer ordering tripwire, null-client guard).
  • Running service is 1/1 and reached 'Starting to consume from queue: image.description.requested'.
    NOTE: deploy durability is a separate open issue — see #508 (master Dockerfile still doesn't COPY message_failure.py, so a clean rebuild would regress). Closing the logic bug here; tracking the packaging gap there.
✅ Fixed & verified — closing. Landed on origin/master: - New torch-free `message_failure.py`: `GpuClientNotReadyError` (transient/retryable), `PERMANENT_FAILURES=(ValueError,KeyError,FileNotFoundError)` with **AttributeError removed** (defect 2), `is_permanent_failure()`. - `image_description_service.py`: null-client guard `:817 if self._gpu_client is None: raise GpuClientNotReadyError(...)`; classifier wired at `:1198 is_permanent=is_permanent_failure(exception)`; GPU-lease client now created before the consumer starts (defect 1, startup race). - Tests: `tests/test_message_failure.py` — I ran all 6 against master sources, **6 passed / 0 failed** (AttributeError-not-permanent, GpuClientNotReadyError-retryable, genuine permanents still DLQ, generic/timeout retry, client-before-consumer ordering tripwire, null-client guard). - Running service is 1/1 and reached 'Starting to consume from queue: image.description.requested'. NOTE: deploy *durability* is a separate open issue — see #508 (master Dockerfile still doesn't COPY message_failure.py, so a clean rebuild would regress). Closing the logic bug here; tracking the packaging gap there.
spikerj commented 2026-07-12 18:22:55 +00:00
Author
Owner
Copy Link
Copy Source

Fix (#229) deployed and behaving — closing. The worker has been through multiple restart cycles since (deploys + a MinIO interruption mid-model-sync) with clean lease release/re-request each time and zero new 'Message rejected (permanent failure)' events; captions complete (110 in Mongo). The regression suite (AttributeError retryable, GpuClientNotReadyError, startup ordering tripwires) runs in CI before every image build.

Fix (#229) deployed and behaving — closing. The worker has been through multiple restart cycles since (deploys + a MinIO interruption mid-model-sync) with clean lease release/re-request each time and **zero new 'Message rejected (permanent failure)' events**; captions complete (110 in Mongo). The regression suite (AttributeError retryable, GpuClientNotReadyError, startup ordering tripwires) runs in CI before every image build.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#506