GPU coordinator queue deadlocked after #467 node-routing deploy — Embeddings pinned to disabled SERVER lane wedges all GPU tenants #822

New Issue

2026-07-24T00:57:51Z

spikerj commented

2026-07-24 00:57:51 +00:00

Summary

Since the PR #467 ("worker-side node routing", #367 U2) coordinator redeploy at 2026-07-23 20:09/20:54 UTC, the shared-GPU coordinator queue is deadlocked. Embeddings can never acquire a lease, and because the anti-starvation guard escalates the stuck Embeddings request to "blocking queue", all GPU tenants behind it (ArtPipe stages, ImageDescription, QuizGen) are blocked too. Still wedged as of 00:55 UTC 2026-07-24.

Root cause

PR #467 made every lease request carry RequiredNodeId = NODE_HOSTNAME (the swarm node the worker runs on), so leases now grant against the requester's own node.

spikersoft-embeddings is pinned by swarm placement to node SERVER (constraints=[node.hostname == SERVER]).

In the coordinator's node config, the SERVER lane is DISABLED:

GpuCoordinator__Nodes__0__NodeId=SERVER  Enabled=false  TotalVramMB=8192   (RTX 3070 Ti)
GpuCoordinator__Nodes__1__NodeId=4090    Enabled=true   TotalVramMB=24576

Therefore every Embeddings request is now pinned to a disabled lane and can never be satisfied. It sits at the head of the queue forever; after MaxSkipCount=5 the coordinator flips it to "blocking queue", wedging grantable requests from other nodes too.

Before #467, requests carried no node identity, so Embeddings-on-SERVER got a (paper) grant against the single enabled 4090 budget and worked. #467 shipped the routing half of the two-lane epic, but the config still runs single-lane (4090 only) while Embeddings physically lives on the disabled SERVER lane.

Log evidence (node SERVER, service spikersoft-gpu-coordinator)

00:06:49 Granted lease ... ImageDescription (qwen3-vl-8b, 20000 MB) on node 4090. Allocated 21024/24064
00:34:25 Lease ...released by ImageDescription. Available: 23040 MB
00:34:25 Lease request from Embeddings (nomic-embed-text-v2-moe, 1536 MB)      <- 23040 MB free, still not granted
00:44:32 Lease request from Embeddings (nomic-embed-text-v2-moe, 1536 MB)      <- 2nd; client 600s timeout fires here
00:51:10 Granted lease ... ArtPipeProcessor (artpipe-sdxllightning, 6000 MB) on node 4090   <- jumps ahead of Embeddings
00:51:19 Pending request 67416d4d (Embeddings, 1536 MB) hit MaxSkipCount=5, blocking queue
00:52:23..00:55:03 Queue status: Embeddings 67416d4d position 1/N (STUCK), ArtPipe + more Embeddings queued behind it

Client-side (SpikerSoft.EventHandlers.Embeddings): System.TimeoutException: GPU lease acquisition timed out after 600s for Embeddings at SharedGpuLeaseService.AcquireAsync line 235.

Impact

Embeddings: 100% down — unroutable; will re-poison the queue ~10 min after any coordinator restart.
Entire shared-GPU queue deadlocked — ArtPipe stages, ImageDescription, QuizGeneration blocked behind the poisoned head. Art generation / image description effectively stalled.
Permanent until fixed; a bare coordinator restart only clears it transiently.

Fixes

Immediate mitigation (pick one):

(A, matches epic intent) Enable the SERVER lane so Embeddings' SERVER-pinned requests route to the real 3070 Ti: set GpuCoordinator__Nodes__0__Enabled=true (verify the 3070 Ti is present/healthy first), then redeploy coordinator. This is the "two enabled lanes" state #467 was built for.
(B, safest quick) Repin embeddings to the enabled lane: change placement to node.hostname == 4090 (or node.labels.artpipe-gpu == true), then restart the coordinator to flush the poisoned queue.

Coordinator hardening (should fix regardless — latent bug):

Fail-fast any lease request whose RequiredNodeId is a disabled/unknown node — return an error to the requester immediately instead of queuing it forever.
The MaxSkipCount "blocking queue" anti-starvation guard must not let an unsatisfiable head-of-line request block requests destined for other nodes. Per-node queues, not one global queue.

Correlation

Caused by PR #467 (spikersoft-backend), merged 2026-07-23 19:39, deployed 20:09/20:54. Parent epic #367 (U2). No matching pre-existing ticket.

## Summary Since the **PR #467** ("worker-side node routing", #367 U2) coordinator redeploy at **2026-07-23 20:09/20:54 UTC**, the shared-GPU coordinator queue is **deadlocked**. Embeddings can never acquire a lease, and because the anti-starvation guard escalates the stuck Embeddings request to "blocking queue", **all** GPU tenants behind it (ArtPipe stages, ImageDescription, QuizGen) are blocked too. Still wedged as of 00:55 UTC 2026-07-24. ## Root cause PR #467 made every lease request carry `RequiredNodeId = NODE_HOSTNAME` (the swarm node the worker runs on), so leases now grant *against the requester's own node*. - `spikersoft-embeddings` is pinned by swarm placement to node **SERVER** (`constraints=[node.hostname == SERVER]`). - In the coordinator's node config, the **SERVER lane is DISABLED**: ``` GpuCoordinator__Nodes__0__NodeId=SERVER Enabled=false TotalVramMB=8192 (RTX 3070 Ti) GpuCoordinator__Nodes__1__NodeId=4090 Enabled=true TotalVramMB=24576 ``` - Therefore every Embeddings request is now pinned to a **disabled lane** and can **never** be satisfied. It sits at the head of the queue forever; after `MaxSkipCount=5` the coordinator flips it to "blocking queue", wedging grantable requests from other nodes too. Before #467, requests carried no node identity, so Embeddings-on-SERVER got a (paper) grant against the single enabled 4090 budget and worked. #467 shipped the routing half of the two-lane epic, but the config still runs **single-lane (4090 only)** while Embeddings physically lives on the disabled SERVER lane. ## Log evidence (node SERVER, service spikersoft-gpu-coordinator) ``` 00:06:49 Granted lease ... ImageDescription (qwen3-vl-8b, 20000 MB) on node 4090. Allocated 21024/24064 00:34:25 Lease ...released by ImageDescription. Available: 23040 MB 00:34:25 Lease request from Embeddings (nomic-embed-text-v2-moe, 1536 MB) <- 23040 MB free, still not granted 00:44:32 Lease request from Embeddings (nomic-embed-text-v2-moe, 1536 MB) <- 2nd; client 600s timeout fires here 00:51:10 Granted lease ... ArtPipeProcessor (artpipe-sdxllightning, 6000 MB) on node 4090 <- jumps ahead of Embeddings 00:51:19 Pending request 67416d4d (Embeddings, 1536 MB) hit MaxSkipCount=5, blocking queue 00:52:23..00:55:03 Queue status: Embeddings 67416d4d position 1/N (STUCK), ArtPipe + more Embeddings queued behind it ``` Client-side (SpikerSoft.EventHandlers.Embeddings): `System.TimeoutException: GPU lease acquisition timed out after 600s for Embeddings` at `SharedGpuLeaseService.AcquireAsync` line 235. ## Impact - **Embeddings: 100% down** — unroutable; will re-poison the queue ~10 min after any coordinator restart. - **Entire shared-GPU queue deadlocked** — ArtPipe stages, ImageDescription, QuizGeneration blocked behind the poisoned head. Art generation / image description effectively stalled. - Permanent until fixed; a bare coordinator restart only clears it transiently. ## Fixes **Immediate mitigation (pick one):** - (A, matches epic intent) Enable the SERVER lane so Embeddings' SERVER-pinned requests route to the real 3070 Ti: set `GpuCoordinator__Nodes__0__Enabled=true` (verify the 3070 Ti is present/healthy first), then redeploy coordinator. This is the "two enabled lanes" state #467 was built for. - (B, safest quick) Repin embeddings to the enabled lane: change placement to `node.hostname == 4090` (or `node.labels.artpipe-gpu == true`), then restart the coordinator to flush the poisoned queue. **Coordinator hardening (should fix regardless — latent bug):** - Fail-fast any lease request whose `RequiredNodeId` is a disabled/unknown node — return an error to the requester immediately instead of queuing it forever. - The `MaxSkipCount` "blocking queue" anti-starvation guard must not let an unsatisfiable head-of-line request block requests destined for *other* nodes. Per-node queues, not one global queue. ## Correlation Caused by **PR #467** (spikersoft-backend), merged 2026-07-23 19:39, deployed 20:09/20:54. Parent epic **#367** (U2). No matching pre-existing ticket.

spikerj commented

2026-07-24 01:01:32 +00:00

Verification update — enabling the SERVER lane is currently blocked by a node-level driver fault. One-shot nvidia-smi on node SERVER fails:

failed to initialize NVML: Driver/library version mismatch

SERVER's RTX 3070 Ti can't initialize (kernel nvidia module vs userspace libnvidia-ml.so mismatch — needs a node reboot or nvidia module reload). This is likely why Nodes__0 (SERVER) was Enabled=false to begin with. Fix option A (enable SERVER lane) requires fixing the driver on SERVER first, otherwise embeddings would route there and CUDA-fail instead of timing out.

Recommend option B for immediate restore: repin embeddings to the healthy 4090 lane (node.hostname==4090 / label artpipe-gpu==true) + restart the coordinator to flush the poisoned queue. Revisit the two-lane (SERVER) goal once SERVER's driver is fixed.

Verification update — **enabling the SERVER lane is currently blocked by a node-level driver fault.** One-shot `nvidia-smi` on node SERVER fails: ``` failed to initialize NVML: Driver/library version mismatch ``` SERVER's RTX 3070 Ti can't initialize (kernel nvidia module vs userspace libnvidia-ml.so mismatch — needs a node reboot or `nvidia` module reload). This is likely why Nodes__0 (SERVER) was Enabled=false to begin with. **Fix option A (enable SERVER lane) requires fixing the driver on SERVER first**, otherwise embeddings would route there and CUDA-fail instead of timing out. Recommend **option B for immediate restore**: repin embeddings to the healthy 4090 lane (`node.hostname==4090` / label `artpipe-gpu==true`) + restart the coordinator to flush the poisoned queue. Revisit the two-lane (SERVER) goal once SERVER's driver is fixed.

spikerj commented

2026-07-24 01:29:13 +00:00

Hardening PR opened: #468 — fix(gpu-coordinator): reject unsatisfiable lease requests + per-node MaxSkipCount block.

Two layers: (1) VramBudgetTracker.CanEverFit classifies permanently-unsatisfiable requests (disabled/unknown node, backend with no enabled match, larger than any card) and rejects them at intake via the existing GpuLeaseRejectMessage path (#698) so the client fails fast + re-requests with backoff instead of queuing forever; (2) a node-pinned request hitting MaxSkipCount now blocks only its own node's pool, not the whole queue (unrouted requests keep the global block). Transient conditions (full or briefly-offline enabled node) still queue honestly. Full GpuCoordinator.Tests: 107 passed.

Hardening PR opened: **#468** — `fix(gpu-coordinator): reject unsatisfiable lease requests + per-node MaxSkipCount block`. Two layers: (1) `VramBudgetTracker.CanEverFit` classifies permanently-unsatisfiable requests (disabled/unknown node, backend with no enabled match, larger than any card) and rejects them at intake via the existing `GpuLeaseRejectMessage` path (#698) so the client fails fast + re-requests with backoff instead of queuing forever; (2) a node-pinned request hitting `MaxSkipCount` now blocks only its own node's pool, not the whole queue (unrouted requests keep the global block). Transient conditions (full or briefly-offline enabled node) still queue honestly. Full GpuCoordinator.Tests: 107 passed.

spikerj commented

2026-07-29 15:47:29 +00:00

Resolved in spikersoft-backend PR #468 (81fb6111) plus committed infra config. Verified against origin/master:

Hardening (the deadlock itself)

GpuCoordinator/Services/VramBudgetTracker.cs:465 — CanEverFit(requiredMB, requiredBackend, requiredNodeId).
GpuCoordinator/Services/LeaseManagerService.cs:317-324 — permanently-unsatisfiable requests are now rejected at intake rather than queueing forever.
LeaseManagerService.cs:748-751 — the MaxSkipCount block is now per-node scoped (blocking {Scope}) instead of global, which is the specific mechanism that let one pinned tenant wedge every other GPU tenant.
Tests: LeaseManagerServiceTests.cs:780-799 MaxSkipCount_NodePinnedBlock_DoesNotWedgeOtherNodes, plus VramBudgetTrackerTests.cs:695-701.

Mitigation A is committed, not just described: spikersoft-infrastructure/spikersoft-gpu-coordinator/docker-stack.yml:95-101 sets Nodes__0__NodeId=SERVER / Enabled=true alongside the 4090 lane. Embeddings stays SERVER-pinned deliberately (spikersoft-embeddings/docker-stack.yml:71-74), now documented as a lane assignment rather than an accident.

Two operator prerequisites are not verifiable from git and are worth confirming before you rely on both lanes being live:

The NVML Driver/library version mismatch on node SERVER — comment 12640 called this a hard prerequisite for option A.
The stack file's own ROLLOUT GUARD at docker-stack.yml:90-94: "only enable BOTH lanes when every shared-mode GPU tenant runs an image with #467's routing."

Closing on the code + config fix; the two items above are deploy-state checks rather than outstanding work.

Resolved in spikersoft-backend PR #468 (`81fb6111`) plus committed infra config. Verified against `origin/master`: **Hardening (the deadlock itself)** - `GpuCoordinator/Services/VramBudgetTracker.cs:465` — `CanEverFit(requiredMB, requiredBackend, requiredNodeId)`. - `GpuCoordinator/Services/LeaseManagerService.cs:317-324` — permanently-unsatisfiable requests are now rejected at intake rather than queueing forever. - `LeaseManagerService.cs:748-751` — the `MaxSkipCount` block is now **per-node scoped** (`blocking {Scope}`) instead of global, which is the specific mechanism that let one pinned tenant wedge every other GPU tenant. - Tests: `LeaseManagerServiceTests.cs:780-799` `MaxSkipCount_NodePinnedBlock_DoesNotWedgeOtherNodes`, plus `VramBudgetTrackerTests.cs:695-701`. **Mitigation A is committed, not just described:** `spikersoft-infrastructure/spikersoft-gpu-coordinator/docker-stack.yml:95-101` sets `Nodes__0__NodeId=SERVER` / `Enabled=true` alongside the 4090 lane. Embeddings stays SERVER-pinned deliberately (`spikersoft-embeddings/docker-stack.yml:71-74`), now documented as a lane assignment rather than an accident. Two operator prerequisites are **not verifiable from git** and are worth confirming before you rely on both lanes being live: 1. The NVML `Driver/library version mismatch` on node SERVER — comment 12640 called this a hard prerequisite for option A. 2. The stack file's own ROLLOUT GUARD at `docker-stack.yml:90-94`: *"only enable BOTH lanes when every shared-mode GPU tenant runs an image with #467's routing."* Closing on the code + config fix; the two items above are deploy-state checks rather than outstanding work.

spikerj closed this issue

2026-07-29 15:47:33 +00:00

spikerj referenced this issue

2026-07-29 16:59:30 +00:00

[GpuCoordinator][Bug] Lease grant races physical VRAM eviction — quiz-generation grant-fail-release cycles to DLQ while ImageDescription yields correctly #704

spikerj referenced this issue

2026-07-29 18:28:33 +00:00

[ArtStudio/Infra] Unified elastic GPU capacity — one coordinator model that scales from a single 24 GB card to multi-node, and runs identically on the M5 dev Mac #367

Sign in to join this conversation.