spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Prod][GPU][CRITICAL] image-description crash-loops (0/1) — ModuleNotFoundError: 'message_failure' — phantom deploy of the #506 fix, Dockerfile missing COPY #508

New Issue
Closed
opened 2026-07-12 17:42:46 +00:00 by spikerj · 4 comments
spikerj commented 2026-07-12 17:42:46 +00:00
Owner
Copy Link
Copy Source

Summary

The image-description service is down and crash-looping (0/1 on 4090) as of ~17:40 UTC 2026-07-12. The freshly-pushed :latest image fails at import time:

File "/app/image_description_service.py", line 33, in <module>
    from message_failure import GpuClientNotReadyError, is_permanent_failure
ModuleNotFoundError: No module named 'message_failure'

Task history: Failed "task: non-zero exit (1)" repeating every ~13s. Service UpdatedAt: 2026-07-12 17:40:01, image git.spikersoft.com/spikerj/spikersoft-image-description:latest@sha256:e4d1f36e....

Root cause — phantom deploy of the #506 fix

This is the in-progress fix for #506 (null-_gpu_client handling): image_description_service.py was changed to from message_failure import GpuClientNotReadyError, is_permanent_failure, introducing a new message_failure.py module. But the pushed image doesn't contain that module, for two compounding reasons:

  1. Dockerfile doesn't copy it. SpikerSoft.EventHandlers.ImageDescription.Python/Dockerfile copies files individually:

    COPY requirements.txt .
COPY image_description_service.py .
COPY gpu_lease.py .

    There is no COPY message_failure.py . and no blanket COPY . . / COPY *.py .. So even if the file existed in the build context, it wouldn't land in the image.

  2. The module isn't in origin/master (nor in the working tree). The running image was built from state that isn't on master — a phantom deploy (same class as commit 36a7457c "repair the phantom deploy"). The edited image_description_service.py (with the line-33 import) got baked in, but message_failure.py did not.

Fix

  1. Add the module to the image: COPY message_failure.py . in the Dockerfile (or switch to COPY *.py . so future sibling modules are picked up automatically).
  2. Commit message_failure.py to master so the build is reproducible from source (not a phantom/local-only build).
  3. Rebuild + redeploy image-description. Verify the container gets past import and reaches Starting to consume from queue: image.description.requested.

Impact

Image-description is fully offline — zero captioning. Any image-bearing book uploaded now will stall at / skip the ImageDescriptions stage. This is a hard outage of the captioner, strictly worse than the intermittent #506 data-loss it was meant to fix. Note: this compounds with #506 itself — until the fix lands correctly, image-bearing pages are not captioned at all.

Verification / tests (per repo policy)

  • Build test / CI check that the image imports cleanly (python -c "import image_description_service" in the built image) so a missing-module import can't ship again.
  • The #506 pytest coverage (GpuClientNotReadyError treated as retryable; startup ordering) should ride in on the same fix once the module is actually packaged.

Related

  • #506 — the null-_gpu_client retry-classification bug this deploy was fixing (introduces message_failure.py / GpuClientNotReadyError / is_permanent_failure).
  • 36a7457c "fix(image-description): S3 image fetch + repair the phantom deploy (#403, #493)" — prior phantom-deploy incident on this exact service; recurrence here.
  • #403 — image-description deploy lineage.
## Summary The `image-description` service is **down and crash-looping** (0/1 on 4090) as of ~17:40 UTC 2026-07-12. The freshly-pushed `:latest` image fails at import time: ``` File "/app/image_description_service.py", line 33, in <module> from message_failure import GpuClientNotReadyError, is_permanent_failure ModuleNotFoundError: No module named 'message_failure' ``` Task history: `Failed "task: non-zero exit (1)"` repeating every ~13s. Service `UpdatedAt: 2026-07-12 17:40:01`, image `git.spikersoft.com/spikerj/spikersoft-image-description:latest@sha256:e4d1f36e...`. ## Root cause — phantom deploy of the #506 fix This is the in-progress fix for **#506** (null-`_gpu_client` handling): `image_description_service.py` was changed to `from message_failure import GpuClientNotReadyError, is_permanent_failure`, introducing a new `message_failure.py` module. But the pushed image doesn't contain that module, for two compounding reasons: 1. **Dockerfile doesn't copy it.** `SpikerSoft.EventHandlers.ImageDescription.Python/Dockerfile` copies files *individually*: ``` COPY requirements.txt . COPY image_description_service.py . COPY gpu_lease.py . ``` There is **no `COPY message_failure.py .`** and no blanket `COPY . .` / `COPY *.py .`. So even if the file existed in the build context, it wouldn't land in the image. 2. **The module isn't in `origin/master`** (nor in the working tree). The running image was built from state that isn't on master — a phantom deploy (same class as commit `36a7457c` "repair the phantom deploy"). The edited `image_description_service.py` (with the line-33 import) got baked in, but `message_failure.py` did not. ## Fix 1. Add the module to the image: `COPY message_failure.py .` in the Dockerfile (or switch to `COPY *.py .` so future sibling modules are picked up automatically). 2. Commit `message_failure.py` to `master` so the build is reproducible from source (not a phantom/local-only build). 3. Rebuild + redeploy `image-description`. Verify the container gets past import and reaches `Starting to consume from queue: image.description.requested`. ## Impact Image-description is **fully offline** — zero captioning. Any image-bearing book uploaded now will stall at / skip the ImageDescriptions stage. This is a hard outage of the captioner, strictly worse than the intermittent #506 data-loss it was meant to fix. Note: this compounds with #506 itself — until the fix lands correctly, image-bearing pages are not captioned at all. ## Verification / tests (per repo policy) - Build test / CI check that the image imports cleanly (`python -c "import image_description_service"` in the built image) so a missing-module import can't ship again. - The #506 pytest coverage (GpuClientNotReadyError treated as retryable; startup ordering) should ride in on the same fix once the module is actually packaged. ## Related - #506 — the null-`_gpu_client` retry-classification bug this deploy was fixing (introduces `message_failure.py` / `GpuClientNotReadyError` / `is_permanent_failure`). - `36a7457c` "fix(image-description): S3 image fetch + repair the phantom deploy (#403, #493)" — prior phantom-deploy incident on this exact service; recurrence here. - #403 — image-description deploy lineage.
spikerj commented 2026-07-12 17:48:05 +00:00
Author
Owner
Copy Link
Copy Source

Fix up in spikersoft-backend PR #230, per this ticket's fix list:

  1. COPY *.py ./ in the Dockerfile (individual COPYs were the trap — a new sibling module could be imported by code yet absent from the image).
  2. Already done — message_failure.py landed on master with PR #229; the crash-looping image predates it (the phantom-build half of this ticket resolved itself once #229 merged, leaving only the COPY gap).
  3. New CI gate: post-build/pre-push smoke test runs python3 -c 'import image_description_service' inside the built image — the full import chain (torch, transformers, gpu_lease, message_failure) must succeed or the image never ships. 'Bootable, not just buildable.'

Merging #230 auto-builds + deploys. Verification: task stops flapping, container reaches 'Starting to consume from queue: image.description.requested', then the queued caption backlog drains (with the #506 race fixes active — same image). Will close on that evidence.

Fix up in **spikersoft-backend PR #230**, per this ticket's fix list: 1. `COPY *.py ./` in the Dockerfile (individual COPYs were the trap — a new sibling module could be imported by code yet absent from the image). 2. Already done — `message_failure.py` landed on master with PR #229; the crash-looping image predates it (the phantom-build half of this ticket resolved itself once #229 merged, leaving only the COPY gap). 3. **New CI gate**: post-build/pre-push smoke test runs `python3 -c 'import image_description_service'` inside the built image — the full import chain (torch, transformers, gpu_lease, message_failure) must succeed or the image never ships. 'Bootable, not just buildable.' Merging #230 auto-builds + deploys. Verification: task stops flapping, container reaches 'Starting to consume from queue: image.description.requested', then the queued caption backlog drains (with the #506 race fixes active — same image). Will close on that evidence.
spikerj commented 2026-07-12 17:53:10 +00:00
Author
Owner
Copy Link
Copy Source

Immediate outage is RESOLVED — image-description is back to 1/1, imports cleanly, consuming (message_failure.py committed to master + image rebuilt). BUT keep this open: the durable root cause is NOT fixed. origin/master Dockerfile still COPYs files individually —

COPY requirements.txt .
COPY image_description_service.py .
COPY gpu_lease.py .

— there is still no COPY message_failure.py (nor COPY *.py .). The currently-running image only works because it was built with the module present; a clean rebuild from master will again omit it and regress to ModuleNotFoundError crash-loop. Please add COPY message_failure.py . (or switch to COPY *.py .) to the committed Dockerfile + a CI import smoke-test (python -c 'import image_description_service' on the built image). The underlying #506 logic fix + tests have landed and #506 is closed.

Immediate outage is RESOLVED — image-description is back to 1/1, imports cleanly, consuming (message_failure.py committed to master + image rebuilt). BUT keep this open: the **durable root cause is NOT fixed**. origin/master `Dockerfile` still COPYs files individually — ``` COPY requirements.txt . COPY image_description_service.py . COPY gpu_lease.py . ``` — there is still **no `COPY message_failure.py`** (nor `COPY *.py .`). The currently-running image only works because it was built with the module present; a clean rebuild from master will again omit it and regress to `ModuleNotFoundError` crash-loop. Please add `COPY message_failure.py .` (or switch to `COPY *.py .`) to the committed Dockerfile + a CI import smoke-test (`python -c 'import image_description_service'` on the built image). The underlying #506 logic fix + tests have landed and #506 is closed.
spikerj commented 2026-07-12 18:01:33 +00:00
Author
Owner
Copy Link
Copy Source

✅ Fixed & verified — closing. Commit a4e9cd7b (fix(image-description): COPY *.py into the image + import smoke test in CI (#508)) landed the durable fix on master:

  • Dockerfile: individual COPYs replaced with COPY *.py ./ (line 30) + a #508 comment — a new sibling module can't be silently left out again. Confirmed all 3 imported modules (image_description_service.py, gpu_lease.py, message_failure.py) are in the build context and captured by the glob; service imports gpu_lease + message_failure at lines 32-33, both present.
  • Associated test (CI import smoke-test): .gitea/workflows/spikersoft-image-description.yml now runs, post-build/pre-push, docker run --entrypoint python3 …image-description:latest -c "import image_description_service" — this exact check would have caught the ModuleNotFoundError before push. Right test for a packaging bug.
  • Runtime: image-description stable at 1/1, imports clean, consuming from image.description.requested.
    Durable packaging gap closed; #506 (the logic fix) already closed with its pytest suite.
✅ Fixed & verified — closing. Commit `a4e9cd7b` (fix(image-description): COPY *.py into the image + import smoke test in CI (#508)) landed the durable fix on master: - **Dockerfile:** individual COPYs replaced with `COPY *.py ./` (line 30) + a #508 comment — a new sibling module can't be silently left out again. Confirmed all 3 imported modules (image_description_service.py, gpu_lease.py, message_failure.py) are in the build context and captured by the glob; service imports `gpu_lease` + `message_failure` at lines 32-33, both present. - **Associated test (CI import smoke-test):** `.gitea/workflows/spikersoft-image-description.yml` now runs, post-build/pre-push, `docker run --entrypoint python3 …image-description:latest -c "import image_description_service"` — this exact check would have caught the ModuleNotFoundError before push. Right test for a packaging bug. - Runtime: image-description stable at 1/1, imports clean, consuming from image.description.requested. Durable packaging gap closed; #506 (the logic fix) already closed with its pytest suite.
spikerj commented 2026-07-12 18:22:55 +00:00
Author
Owner
Copy Link
Copy Source

Verified and closing. Since #230 (COPY *.py + the image import smoke gate) the worker boots, consumes, leases, and captions — 110 completed captions in Mongo. The CI gate now runs python3 -c 'import image_description_service' inside every built image pre-push, so an import-time missing module can never ship again.

Verified and closing. Since #230 (COPY *.py + the image import smoke gate) the worker boots, consumes, leases, and captions — **110 completed captions** in Mongo. The CI gate now runs `python3 -c 'import image_description_service'` inside every built image pre-push, so an import-time missing module can never ship again.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#508