spikerj commented on issue spikerj/spikersoft-issues#500 2026-07-12 18:22:54 +00:00
[Bug][Infra][GPU] image-description lease starves — gpu-coordinator single-lane budget still on SERVER (8192MB) vs 16000MB needed; enable 4090 lane

Verified in production and closing. Since infra #40 deployed (SERVER lane parked, 4090 lane enabled, budget 24576 MB): the 16000 MB ImageDescription lease grants immediately (17:48:15,…

spikerj commented on issue spikerj/spikersoft-issues#502 2026-07-12 18:22:54 +00:00
[Bug][Prod][GPU] image-description crash-loops on gated HuggingFace model — no HF token for Qwen3-VL-8B (401 GatedRepoError)

Superseded by the MinIO model architecture and verified working — closing. The gated-repo problem was eliminated rather than tokened-around: the worker now syncs Qwen/Qwen3-VL-8B-Instruct…

spikerj closed issue spikerj/spikersoft-issues#500 2026-07-12 18:22:54 +00:00
[Bug][Infra][GPU] image-description lease starves — gpu-coordinator single-lane budget still on SERVER (8192MB) vs 16000MB needed; enable 4090 lane
spikerj commented on issue spikerj/spikersoft-issues#509 2026-07-12 18:21:10 +00:00
[Bug][Backend][Data] BookPages inflation — quiz saw 6 pages on a 3-page book; one book linked 570 pages (staleness-retry double-processing suspected)

Root-caused WITH production data and fixed — PR #234. Forensics (read-only via mongo-router): the bad book holds pages [1,1,2,2,3,3] under TWO ISBNs (both uploads of the same PDF); the…

spikerj commented on issue spikerj/spikersoft-issues#507 2026-07-12 18:13:17 +00:00
[Bug][Backend] Zero-question quizzes are marked Completed and takeable — starting one insta-expires at 0/0

Second code layer landed: PR #233 — the true mechanism of the zero-question quiz was the fixed pagesToSkip=3 making a 3-page book start at page 4 (its whole content skipped). New `ComputePag…

spikerj opened issue spikerj/spikersoft-issues#509 2026-07-12 18:13:05 +00:00
[Bug][Backend][Data] BookPages inflation — quiz saw 6 pages on a 3-page book; one book linked 570 pages (staleness-retry double-processing suspected)
spikerj commented on issue spikerj/spikersoft-issues#486 2026-07-12 18:07:53 +00:00
[Bug][Prod] SafetyCheck art-pipe resident worker missing 'torch' → ArtPipeProcessor host crash-loops (24x)

Reliability half fixed in spikersoft-backend PR #232 — both triage suspects confirmed and addressed: (a) SafetyCheckRpcConsumer now retries transport wiring with backoff instead of rethrowing…

spikerj closed issue spikerj/spikersoft-issues#508 2026-07-12 18:01:33 +00:00
[Bug][Prod][GPU][CRITICAL] image-description crash-loops (0/1) — ModuleNotFoundError: 'message_failure' — phantom deploy of the #506 fix, Dockerfile missing COPY
spikerj commented on issue spikerj/spikersoft-issues#508 2026-07-12 18:01:33 +00:00
[Bug][Prod][GPU][CRITICAL] image-description crash-loops (0/1) — ModuleNotFoundError: 'message_failure' — phantom deploy of the #506 fix, Dockerfile missing COPY

✅ Fixed & verified — closing. Commit a4e9cd7b (fix(image-description): COPY *.py into the image + import smoke test in CI (#508)) landed the durable fix on master:

  • Dockerfile: individual…
spikerj commented on issue spikerj/spikersoft-issues#504 2026-07-12 17:59:23 +00:00
[Bug][Infra] MinIO doesn't recover after a transient /mnt/fusionio drive blip — stuck '0 drives provided', 'mc ready local' healthcheck misses it, needs manual restart

Fix up in spikersoft-infrastructure PR #42: drive-aware healthcheck. One correction to this ticket's option 1: curl was REMOVED from MinIO images in 2023, so the probe is mc-based — `mc ready…

spikerj commented on issue spikerj/spikersoft-issues#507 2026-07-12 17:56:43 +00:00
[Bug][Backend] Zero-question quizzes are marked Completed and takeable — starting one insta-expires at 0/0

Fix up in spikersoft-backend PR #231 (worker Failed-state + StartQuizAttempt guard + 400 mapping + 3 xUnit tests — details on the PR). The page-count-mismatch investigation (pages 4–6 'not…

spikerj commented on issue spikerj/spikersoft-issues#508 2026-07-12 17:53:10 +00:00
[Bug][Prod][GPU][CRITICAL] image-description crash-loops (0/1) — ModuleNotFoundError: 'message_failure' — phantom deploy of the #506 fix, Dockerfile missing COPY

Immediate outage is RESOLVED — image-description is back to 1/1, imports cleanly, consuming (message_failure.py committed to master + image rebuilt). BUT keep this open: the **durable root cause…

spikerj closed issue spikerj/spikersoft-issues#506 2026-07-12 17:53:10 +00:00
[Bug][Prod][GPU] image-description permanently dead-letters captions on null gpu_client — AttributeError misclassified as permanent (data loss)
spikerj commented on issue spikerj/spikersoft-issues#506 2026-07-12 17:53:09 +00:00
[Bug][Prod][GPU] image-description permanently dead-letters captions on null gpu_client — AttributeError misclassified as permanent (data loss)

✅ Fixed & verified — closing. Landed on origin/master:

  • New torch-free message_failure.py: GpuClientNotReadyError (transient/retryable), `PERMANENT_FAILURES=(ValueError,KeyError,FileNotFound…
spikerj commented on issue spikerj/spikersoft-issues#508 2026-07-12 17:48:05 +00:00
[Bug][Prod][GPU][CRITICAL] image-description crash-loops (0/1) — ModuleNotFoundError: 'message_failure' — phantom deploy of the #506 fix, Dockerfile missing COPY

Fix up in spikersoft-backend PR #230, per this ticket's fix list:

  1. COPY *.py ./ in the Dockerfile (individual COPYs were the trap — a new sibling module could be imported by code yet…
spikerj commented on issue spikerj/spikersoft-issues#506 2026-07-12 17:42:54 +00:00
[Bug][Prod][GPU] image-description permanently dead-letters captions on null gpu_client — AttributeError misclassified as permanent (data loss)

⚠️ The fix for this is currently deployed but BROKEN — filed #508. The pushed image-description :latest (UpdatedAt 17:40 UTC) imports `from message_failure import GpuClientNotReadyError,…

spikerj opened issue spikerj/spikersoft-issues#508 2026-07-12 17:42:46 +00:00
[Bug][Prod][GPU][CRITICAL] image-description crash-loops (0/1) — ModuleNotFoundError: 'message_failure' — phantom deploy of the #506 fix, Dockerfile missing COPY
spikerj commented on issue spikerj/spikersoft-issues#506 2026-07-12 17:39:45 +00:00
[Bug][Prod][GPU] image-description permanently dead-letters captions on null gpu_client — AttributeError misclassified as permanent (data loss)

Fix up in spikersoft-backend PR #229, implementing exactly the recommended combination — (1)+(3) durable, (2) defense-in-depth:

  1. AttributeError is no longer a permanent failure.…
spikerj opened issue spikerj/spikersoft-issues#507 2026-07-12 17:36:13 +00:00
[Bug][Backend] Zero-question quizzes are marked Completed and takeable — starting one insta-expires at 0/0
spikerj commented on issue spikerj/spikersoft-issues#502 2026-07-12 17:35:55 +00:00
[Bug][Prod][GPU] image-description crash-loops on gated HuggingFace model — no HF token for Qwen3-VL-8B (401 GatedRepoError)

Related: filed #506 — a separate image-description defect surfaced while investigating this service. When the GPU-lease client is momentarily None (startup race before _gpu_client is set, or…