spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra][GPU] image-description lease starves — gpu-coordinator single-lane budget still on SERVER (8192MB) vs 16000MB needed; enable 4090 lane #500

New Issue
Closed
opened 2026-07-12 07:00:00 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-12 07:00:00 +00:00
Owner
Copy Link
Copy Source

Summary

Book uploads that contain images hang at Stage=ImageDescriptions. The image-description worker (Qwen3-VL-8B) is now deployed and healthy, but its GPU-lease request starves forever because the gpu-coordinator budget is still parked on SERVER (8192 MB) while the model needs 16000 MB. The request sits at queue position 1/1 indefinitely and the GPU stays idle. This is the last blocker in the #403 chain.

Evidence (2026-07-12, SERVER swarm)

image-description (running on the 4090 node after the placement fix) logs:

06:41:42  Received message for image 6a53370d29759f8302ced7bd
06:41:42  Acquiring GPU lease before model inference...
06:41:42  Requesting GPU lease for ImageDescription (qwen3-vl-8b, 16000 MB VRAM)...

gpu-coordinator then loops forever:

06:48:04–06:50:05  Queue status for ImageDescription (31bba53f-…): position 1/1   (every 5s)

nvidia-smi on the 4090 the whole time: 36 MiB used / 24012 MiB free — the model never loads.

Root cause

spikersoft-gpu-coordinator/docker-stack.yml env (current state on master):

GpuCoordinator__TotalGpuVramMB=8192            # legacy key
GpuCoordinator__Nodes__0__NodeId=SERVER   TotalVramMB=8192   Enabled=true
GpuCoordinator__Nodes__1__NodeId=4090     TotalVramMB=24576  Enabled=false   ← disabled

Coordinator admission budget = sum of TotalVramMB over Enabled nodes — confirmed in source:

  • SpikerSoft.EventHandlers.GpuCoordinator/Services/VramBudgetTracker.cs:196 → TotalVramMB => EnabledNodes.Sum(n => n.TotalVramMB)
  • …VramBudgetTracker.cs:210 → EnabledNodes => _nodes.Where(n => n.Enabled)
  • …/Program.cs:139 → the global GpuCoordinator__TotalGpuVramMB is the legacy key, ignored once Nodes__* are declared.

So only SERVER's 8192 MB is available → a 16000 MB request can never be granted → queues forever.

Fix

The artpipe-gpu swarm label was already moved to the 4090 node (done manually). Per the single-lane invariant (stack comment lines 75–83; docs/4090-burst-node-runbook.md "Session START" step 2): exactly ONE Nodes entry may be Enabled=true, and it must be the label holder. So flip the budget to the 4090:

In spikersoft-gpu-coordinator/docker-stack.yml:

- GpuCoordinator__Nodes__0__Enabled=true     # SERVER
+ GpuCoordinator__Nodes__0__Enabled=false
- GpuCoordinator__Nodes__1__Enabled=false    # 4090
+ GpuCoordinator__Nodes__1__Enabled=true

Budget then = 24576 MB → the 16000 MB lease grants, Qwen3-VL loads on the 24 GB card, captions flow. GpuCoordinator__TotalGpuVramMB needs no change (legacy/ignored). Do not enable both nodes — with two enabled, a worker on the 4090 can be granted against SERVER's budget and CUDA-OOM (until #367 U2 lands).

Deploy (coordinator is pinned to SERVER; deploy from any manager after syncing /mnt/infrastructure):

cd /mnt/infrastructure/spikersoft-gpu-coordinator
sudo docker stack deploy --with-registry-auth -c docker-stack.yml spikersoft-gpu-coordinator

Leases are in-memory; the image-description worker re-requests on reconnect. The currently-queued request (31bba53f) will RPC-timeout/DLQ — re-upload a test book after redeploy.

Revert (mandatory before the 4090 leaves the swarm)

Per runbook "Session END": flip back (Nodes__0 Enabled=true, Nodes__1 Enabled=false), move the artpipe-gpu label back to SERVER, redeploy. Leaving the budget on a departed node stalls every lease request.

Starting point already pushed

A branch with exactly this change is on the remote: fix/gpu-coordinator-4090-burst (commit 83916a4) — open a PR from it, or discard and redo. (I couldn't push straight to master: it had advanced with unrelated MinIO/runner PRs and a root-owned minio/docker-stack.yml in the working checkout blocked a local rebase.)

Related

  • #403 — image-description infra drift / missing stack (now deployed; this is the next-step blocker)
  • #493 — MinIO ebooks migration (metadata-extractor / file-movement env work landed today)
  • #367 U2 — node-aware lease routing (would remove the single-lane invariant)
## Summary Book uploads that contain images hang at `Stage=ImageDescriptions`. The `image-description` worker (Qwen3-VL-8B) is now deployed and healthy, but its GPU-lease request **starves forever** because the `gpu-coordinator` budget is still parked on SERVER (8192 MB) while the model needs **16000 MB**. The request sits at queue `position 1/1` indefinitely and the GPU stays idle. This is the last blocker in the #403 chain. ## Evidence (2026-07-12, SERVER swarm) `image-description` (running on the `4090` node after the placement fix) logs: ``` 06:41:42 Received message for image 6a53370d29759f8302ced7bd 06:41:42 Acquiring GPU lease before model inference... 06:41:42 Requesting GPU lease for ImageDescription (qwen3-vl-8b, 16000 MB VRAM)... ``` `gpu-coordinator` then loops forever: ``` 06:48:04–06:50:05 Queue status for ImageDescription (31bba53f-…): position 1/1 (every 5s) ``` `nvidia-smi` on the 4090 the whole time: **36 MiB used / 24012 MiB free** — the model never loads. ## Root cause `spikersoft-gpu-coordinator/docker-stack.yml` env (current state on master): ``` GpuCoordinator__TotalGpuVramMB=8192 # legacy key GpuCoordinator__Nodes__0__NodeId=SERVER TotalVramMB=8192 Enabled=true GpuCoordinator__Nodes__1__NodeId=4090 TotalVramMB=24576 Enabled=false ← disabled ``` Coordinator admission budget = **sum of `TotalVramMB` over `Enabled` nodes** — confirmed in source: - `SpikerSoft.EventHandlers.GpuCoordinator/Services/VramBudgetTracker.cs:196` → `TotalVramMB => EnabledNodes.Sum(n => n.TotalVramMB)` - `…VramBudgetTracker.cs:210` → `EnabledNodes => _nodes.Where(n => n.Enabled)` - `…/Program.cs:139` → the global `GpuCoordinator__TotalGpuVramMB` is the **legacy** key, ignored once `Nodes__*` are declared. So only SERVER's 8192 MB is available → a 16000 MB request can never be granted → queues forever. ## Fix The `artpipe-gpu` swarm label was **already moved to the `4090` node** (done manually). Per the **single-lane invariant** (stack comment lines 75–83; `docs/4090-burst-node-runbook.md` "Session START" step 2): exactly ONE `Nodes` entry may be `Enabled=true`, and it must be the label holder. So flip the budget to the 4090: In `spikersoft-gpu-coordinator/docker-stack.yml`: ```diff - GpuCoordinator__Nodes__0__Enabled=true # SERVER + GpuCoordinator__Nodes__0__Enabled=false - GpuCoordinator__Nodes__1__Enabled=false # 4090 + GpuCoordinator__Nodes__1__Enabled=true ``` Budget then = 24576 MB → the 16000 MB lease grants, Qwen3-VL loads on the 24 GB card, captions flow. `GpuCoordinator__TotalGpuVramMB` needs no change (legacy/ignored). Do **not** enable both nodes — with two enabled, a worker on the 4090 can be granted against SERVER's budget and CUDA-OOM (until #367 U2 lands). Deploy (coordinator is pinned to SERVER; deploy from any manager after syncing `/mnt/infrastructure`): ``` cd /mnt/infrastructure/spikersoft-gpu-coordinator sudo docker stack deploy --with-registry-auth -c docker-stack.yml spikersoft-gpu-coordinator ``` Leases are in-memory; the image-description worker re-requests on reconnect. The currently-queued request (`31bba53f`) will RPC-timeout/DLQ — re-upload a test book after redeploy. ## Revert (mandatory before the 4090 leaves the swarm) Per runbook "Session END": flip back (`Nodes__0 Enabled=true`, `Nodes__1 Enabled=false`), move the `artpipe-gpu` label back to SERVER, redeploy. Leaving the budget on a departed node stalls every lease request. ## Starting point already pushed A branch with exactly this change is on the remote: **`fix/gpu-coordinator-4090-burst`** (commit `83916a4`) — open a PR from it, or discard and redo. (I couldn't push straight to `master`: it had advanced with unrelated MinIO/runner PRs and a root-owned `minio/docker-stack.yml` in the working checkout blocked a local rebase.) ## Related - #403 — image-description infra drift / missing stack (now deployed; this is the next-step blocker) - #493 — MinIO ebooks migration (metadata-extractor / file-movement env work landed today) - #367 U2 — node-aware lease routing (would remove the single-lane invariant)
spikerj commented 2026-07-12 07:02:25 +00:00
Author
Owner
Copy Link
Copy Source

PR opened from the pre-pushed branch: spikersoft-infrastructure #40 (fix/gpu-coordinator-4090-burst, commit 83916a4 — used as-is; it merges cleanly, the coordinator stack was untouched by tonight's other merges).

Sequence to unblock:

  1. Merge #40.
  2. Pull /mnt/infrastructure, then: cd /mnt/infrastructure/spikersoft-gpu-coordinator && sudo docker stack deploy --with-registry-auth -c docker-stack.yml spikersoft-gpu-coordinator (env-only — no image rebuild; coordinator restarts with budget 24576).
  3. Re-upload a test book (the starved request 31bba53f will have RPC-timed-out/DLQ'd).

Verify: coordinator logs Granted lease … to ImageDescription (qwen3-vl-8b, 16000 MB) on node 4090; nvidia-smi on the 4090 jumps to ~16 GB; book advances past Stage=ImageDescriptions → embeddings → quiz (which would also finally verify #497 end-to-end — first complete pipeline run).

Sanity note on the flip: with the lane on the 4090, SERVER-pinned embeddings (1536 MB) + quiz (5000 MB) leases are granted against the 4090 ledger while physically loading on SERVER's 8 GB card — numerically safe tonight (6536 MB < SERVER's ~7680 usable, and 16000+6536=22536 < 24576 budget), and it's the documented single-lane tradeoff until #367 U2's per-node routing.

Revert reminder (Session END, before the 4090 leaves): flip both Enabled flags back + move the artpipe-gpu label to SERVER + redeploy — a departed budget node stalls every lease. I'll close this once a lease grant on node 4090 is observed.

PR opened from the pre-pushed branch: **spikersoft-infrastructure #40** (`fix/gpu-coordinator-4090-burst`, commit 83916a4 — used as-is; it merges cleanly, the coordinator stack was untouched by tonight's other merges). Sequence to unblock: 1. Merge #40. 2. Pull `/mnt/infrastructure`, then: `cd /mnt/infrastructure/spikersoft-gpu-coordinator && sudo docker stack deploy --with-registry-auth -c docker-stack.yml spikersoft-gpu-coordinator` (env-only — no image rebuild; coordinator restarts with budget 24576). 3. Re-upload a test book (the starved request 31bba53f will have RPC-timed-out/DLQ'd). Verify: coordinator logs `Granted lease … to ImageDescription (qwen3-vl-8b, 16000 MB) on node 4090`; nvidia-smi on the 4090 jumps to ~16 GB; book advances past Stage=ImageDescriptions → embeddings → quiz (which would also finally verify #497 end-to-end — first complete pipeline run). Sanity note on the flip: with the lane on the 4090, SERVER-pinned embeddings (1536 MB) + quiz (5000 MB) leases are granted against the 4090 ledger while physically loading on SERVER's 8 GB card — numerically safe tonight (6536 MB < SERVER's ~7680 usable, and 16000+6536=22536 < 24576 budget), and it's the documented single-lane tradeoff until #367 U2's per-node routing. **Revert reminder (Session END, before the 4090 leaves):** flip both Enabled flags back + move the artpipe-gpu label to SERVER + redeploy — a departed budget node stalls every lease. I'll close this once a lease grant on node 4090 is observed.
spikerj commented 2026-07-12 07:19:24 +00:00
Author
Owner
Copy Link
Copy Source

Update (2026-07-12 ~07:18): the coordinator fix appears live — the image-description worker now gets its GPU lease (no more position 1/1 starvation) and proceeds to load the model. The chain advanced to the next blocker: the Qwen3-VL model repo is gated on HuggingFace and no HF token is configured, so from_pretrained 401s and the worker crash-loops. Filed as #502. This ticket's lease change can be verified/closed once #502 is resolved and captions flow end-to-end.

Update (2026-07-12 ~07:18): the coordinator fix appears **live** — the image-description worker now gets its GPU lease (no more `position 1/1` starvation) and proceeds to load the model. The chain advanced to the next blocker: the Qwen3-VL model repo is **gated on HuggingFace** and no HF token is configured, so `from_pretrained` 401s and the worker crash-loops. Filed as #502. This ticket's lease change can be verified/closed once #502 is resolved and captions flow end-to-end.
spikerj commented 2026-07-12 18:22:54 +00:00
Author
Owner
Copy Link
Copy Source

Verified in production and closing. Since infra #40 deployed (SERVER lane parked, 4090 lane enabled, budget 24576 MB): the 16000 MB ImageDescription lease grants immediately (17:48:15, 17:54:58 UTC observed), plus Embeddings (1536) and QuizGeneration (5000) leases granting/releasing cleanly against the same budget. Downstream proof: 110 book-page images now carry completed captions in Mongo — the starved stage is fully unblocked. Revert reminder stands: run the Session END flip (both Enabled flags + label back to SERVER + redeploy) before the 4090 leaves the swarm.

Verified in production and closing. Since infra #40 deployed (SERVER lane parked, 4090 lane enabled, budget 24576 MB): the 16000 MB ImageDescription lease **grants immediately** (17:48:15, 17:54:58 UTC observed), plus Embeddings (1536) and QuizGeneration (5000) leases granting/releasing cleanly against the same budget. Downstream proof: **110 book-page images now carry completed captions** in Mongo — the starved stage is fully unblocked. Revert reminder stands: run the Session END flip (both Enabled flags + label back to SERVER + redeploy) before the 4090 leaves the swarm.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#500