[GpuCoordinator][Bug] Lease grant races physical VRAM eviction — quiz-generation grant-fail-release cycles to DLQ while ImageDescription yields correctly #704

New Issue

2026-07-18T23:56:34Z

spikerj commented

2026-07-18 23:56:34 +00:00

Observed 2026-07-18 ~23:03-23:05Z (godot book 6a5ad504's quiz, right after its embeddings completed): quiz-generation exhausted its 3 message retries with GpuResourceUnavailableException in QuizGenerationService.InitializeForPresetAsync and DLQ'd the request — while the coordinator's #560 churn logic was working exactly as designed.

The race:

ImageDescription holds 20 GB (draining the 954-caption backlog). QuizGeneration requests 5 GB.
Coordinator detects the starvation and tells ImageDescription to YIELD (logged 5x, 23:03-23:04 — churn pressure, correct behavior).
ImageDescription releases → the coordinator's LEDGER frees instantly → QuizGeneration is granted immediately ("Allocated: 5000/24064 MB. Active leases: 1", 23:05:22).
But qwen's 20 GB is still PHYSICALLY resident (model eviction takes seconds; the release message races the actual CUDA free). Quiz's llama.cpp init OOMs, it releases 1 second later with zero work ("grant-fail-release cycling, 3 zero-work releases" per the coordinator's own churn tracker), the cycle repeats, and after 3 message retries the quiz request dead-letters.
generation.process.dlq now holds 6 messages — the 5 older ones today are almost certainly earlier laps of the same cycle for other books.

Fix directions (design choice needed):

Client side (cheapest): InitializeForPresetAsync should retry the CUDA allocation with short backoff (e.g. 3x10s) WITHIN the lease before giving up — the physical eviction completes within seconds, so the first retry would virtually always succeed.
Coordinator side (durable): don't re-grant freed VRAM until physical free is confirmed — the #367 U5 heartbeat (GpuNodeHeartbeatMessage.FreeVramMB + UseHeartbeatCapacity, currently OFF) is the natural seam: gate grants on reported free VRAM covering the request.
Protocol side: define release semantics as "sent AFTER physical eviction" and fix the Python/. NET workers to comply (today release ≈ 'started evicting').

Ops follow-up once the image-description backlog finishes draining (card goes idle): replay generation.process.dlq (6 msgs) and embeddings.process.dlq (2 msgs from the 07:02Z incident) — with a quiet card the quiz generations will succeed. Shovel plugin is not enabled on the broker; replay needs either enabling rabbitmq_shovel or a small authenticated republish job.

Related: #560 (churn logic — working as designed here), #367 U5 (heartbeat capacity), #698 (today's lease-protocol hardening).

**Observed 2026-07-18 ~23:03-23:05Z** (godot book 6a5ad504's quiz, right after its embeddings completed): quiz-generation exhausted its 3 message retries with `GpuResourceUnavailableException` in `QuizGenerationService.InitializeForPresetAsync` and DLQ'd the request — while the coordinator's #560 churn logic was working exactly as designed. **The race:** 1. ImageDescription holds 20 GB (draining the 954-caption backlog). QuizGeneration requests 5 GB. 2. Coordinator detects the starvation and tells ImageDescription to YIELD (logged 5x, 23:03-23:04 — churn pressure, correct behavior). 3. ImageDescription releases → the coordinator's LEDGER frees instantly → QuizGeneration is granted immediately ("Allocated: 5000/24064 MB. Active leases: 1", 23:05:22). 4. But qwen's 20 GB is still PHYSICALLY resident (model eviction takes seconds; the release message races the actual CUDA free). Quiz's llama.cpp init OOMs, it releases 1 second later with zero work ("grant-fail-release cycling, 3 zero-work releases" per the coordinator's own churn tracker), the cycle repeats, and after 3 message retries the quiz request dead-letters. 5. `generation.process.dlq` now holds 6 messages — the 5 older ones today are almost certainly earlier laps of the same cycle for other books. **Fix directions (design choice needed):** - Client side (cheapest): `InitializeForPresetAsync` should retry the CUDA allocation with short backoff (e.g. 3x10s) WITHIN the lease before giving up — the physical eviction completes within seconds, so the first retry would virtually always succeed. - Coordinator side (durable): don't re-grant freed VRAM until physical free is confirmed — the #367 U5 heartbeat (`GpuNodeHeartbeatMessage.FreeVramMB` + `UseHeartbeatCapacity`, currently OFF) is the natural seam: gate grants on reported free VRAM covering the request. - Protocol side: define release semantics as "sent AFTER physical eviction" and fix the Python/. NET workers to comply (today release ≈ 'started evicting'). **Ops follow-up once the image-description backlog finishes draining (card goes idle):** replay `generation.process.dlq` (6 msgs) and `embeddings.process.dlq` (2 msgs from the 07:02Z incident) — with a quiet card the quiz generations will succeed. Shovel plugin is not enabled on the broker; replay needs either enabling rabbitmq_shovel or a small authenticated republish job. Related: #560 (churn logic — working as designed here), #367 U5 (heartbeat capacity), #698 (today's lease-protocol hardening).

spikerj commented

2026-07-19 00:31:42 +00:00

Smoking-gun evidence (2026-07-19 ~00:20Z): a concept stage (SDXL Lightning) OOM'd at .to(device) — "Tried to allocate 26.00 MiB. GPU 0 ... 24.06 MiB is free". Mapped the traceback's PIDs to live containers on the 4090:

pid 3490290 = artpipe-model-safety, 850 MiB physical, books 1024 MB — honest.
pid 3600307 = image-description (qwen, draining the backlog), 16.9 GiB physical, books 20000 MB — honest.
pid 3684206 = quiz-generation (dotnet), 4.4 GiB physical, books NOTHING — it released its lease during the 23:05Z grant-fail-release cycle ("VRAM freed: 5000 MB. Available: 24064") but the failed InitializeForPresetAsync left the partially-loaded LLamaSharp model resident. A leaked, unbooked squatter ever since.

Ledger said ~3 GB free → concept lease granted → physical card had 24 MiB → OOM. So this ticket is really THREE defects:

Grant races physical eviction (original report).
Release does not imply freed — quiz-generation released its lease while still holding 4.4 GiB (protocol contract: GpuLeaseReleaseMessage is defined as "when it has unloaded its model").
Failure-path leak — the OOM'd init keeps its partial CUDA allocation forever; it must dispose the LLama weights/context in a finally (and only then release the lease).

Remediation now: force-restart quiz-generation to reclaim the leak. Fix priority upgrade suggested: (3) is a one-liner-class fix in QuizGenerationService and removes the standing hazard; (1)+(2) as originally scoped.

Smoking-gun evidence (2026-07-19 ~00:20Z): a concept stage (SDXL Lightning) OOM'd at `.to(device)` — "Tried to allocate 26.00 MiB. GPU 0 ... 24.06 MiB is free". Mapped the traceback's PIDs to live containers on the 4090: - pid 3490290 = artpipe-model-safety, 850 MiB physical, books 1024 MB — honest. - pid 3600307 = image-description (qwen, draining the backlog), 16.9 GiB physical, books 20000 MB — honest. - pid 3684206 = **quiz-generation (dotnet), 4.4 GiB physical, books NOTHING** — it released its lease during the 23:05Z grant-fail-release cycle ("VRAM freed: 5000 MB. Available: 24064") but the failed `InitializeForPresetAsync` left the partially-loaded LLamaSharp model resident. A leaked, unbooked squatter ever since. Ledger said ~3 GB free → concept lease granted → physical card had 24 MiB → OOM. So this ticket is really THREE defects: 1. Grant races physical eviction (original report). 2. **Release does not imply freed** — quiz-generation released its lease while still holding 4.4 GiB (protocol contract: GpuLeaseReleaseMessage is defined as "when it has unloaded its model"). 3. **Failure-path leak** — the OOM'd init keeps its partial CUDA allocation forever; it must dispose the LLama weights/context in a finally (and only then release the lease). Remediation now: force-restart quiz-generation to reclaim the leak. Fix priority upgrade suggested: (3) is a one-liner-class fix in QuizGenerationService and removes the standing hazard; (1)+(2) as originally scoped.

spikerj commented

2026-07-29 16:59:30 +00:00

Audited at file level against origin/master. NOT DONE — and #822's work does not cover this, they're different axes.

Defect (c), the failure-path CUDA leak — confirmed present, and I can point at the exact window. SpikerSoft.Business.Ai.Workers/Services/QuizGenerationService.cs:179-198:

LLamaWeights weights;
LLamaContext ctx;
try {
    weights = await LLamaWeights.LoadFromFileAsync(parameters);
    ctx     = weights.CreateContext(parameters);
}
catch (Exception ex) when (GpuFailureClassifier.IsGpuResourceFailure(ex)) {
    logger.LogCritical(...);
    throw new GpuResourceUnavailableException(...);
}

If LoadFromFileAsync succeeds and CreateContext throws, weights already holds VRAM and is never disposed — the catch rethrows without touching it, and the outer finally (:205-208) only releases _modelLock. That's the leak, and it's the worst possible moment for it: CreateContext is exactly where you run out of VRAM, so the failure that triggers a retry also strands the allocation that made the retry necessary.

There's a second, quieter path: the filter when (GpuFailureClassifier.IsGpuResourceFailure(ex)) means a non-GPU exception from CreateContext propagates uncaught — also leaking weights, and without even the #553 log line.

Fix is small: try { ctx = weights.CreateContext(parameters); } catch { weights.Dispose(); throw; }, or hoist both into a finally that disposes when SetLoadedModelState was never reached.

Defect (a) — UseHeartbeatCapacity is still off, both by CLR default (GpuSchedulingConfig.cs:314, bool with no initializer) and explicitly at EventHandlers.GpuCoordinator/appsettings.json:34 ("UseHeartbeatCapacity": false).

But its own precondition has changed, and that's worth flagging. The doc comment at GpuSchedulingConfig.cs:310-313 says flipping it is "inert until either a node goes offline or a second GPU node is enabled." #822 enabled the SERVER lane (spikersoft-gpu-coordinator/docker-stack.yml:95-101, Nodes__0__NodeId=SERVER / Enabled=true) alongside the 4090. So a second GPU node is now enabled — flipping this flag is no longer inert, and whatever reasoning kept it off may no longer apply. Worth a deliberate decision rather than leaving it at a default written for a single-node world.

On the #822 question specifically: its work is a genuinely different axis. VramBudgetTracker.CanEverFit (:465) is an admission test — could this request ever fit on this node — and the LeaseManagerService changes (:317-324 intake rejection, :748-751 per-node MaxSkipCount scoping) are about queue fairness. None of that addresses a grant racing physical eviction, which is about the coordinator's accounting diverging from actual VRAM state. So #822 closing does not close this.

Defect (b), release-does-not-imply-freed: I did not verify it. Flagging that rather than implying coverage — it needs the release path in LeaseManagerService read against what actually frees VRAM on the worker side.

Remaining: (c) the disposal fix (small, and I'd do it first — it's a real VRAM leak on the retry path); (b) verification; (a) a decision on the flag now that its precondition holds; plus the DLQ replay, which is ops and not gradeable from git.

Audited at file level against `origin/master`. **NOT DONE — and #822's work does not cover this, they're different axes.** **Defect (c), the failure-path CUDA leak — confirmed present, and I can point at the exact window.** `SpikerSoft.Business.Ai.Workers/Services/QuizGenerationService.cs:179-198`: ```csharp LLamaWeights weights; LLamaContext ctx; try { weights = await LLamaWeights.LoadFromFileAsync(parameters); ctx = weights.CreateContext(parameters); } catch (Exception ex) when (GpuFailureClassifier.IsGpuResourceFailure(ex)) { logger.LogCritical(...); throw new GpuResourceUnavailableException(...); } ``` If `LoadFromFileAsync` **succeeds** and `CreateContext` **throws**, `weights` already holds VRAM and is **never disposed** — the catch rethrows without touching it, and the outer `finally` (`:205-208`) only releases `_modelLock`. That's the leak, and it's the worst possible moment for it: `CreateContext` is exactly where you run out of VRAM, so the failure that triggers a retry also strands the allocation that made the retry necessary. There's a second, quieter path: the filter `when (GpuFailureClassifier.IsGpuResourceFailure(ex))` means a **non**-GPU exception from `CreateContext` propagates uncaught — also leaking `weights`, and without even the #553 log line. Fix is small: `try { ctx = weights.CreateContext(parameters); } catch { weights.Dispose(); throw; }`, or hoist both into a `finally` that disposes when `SetLoadedModelState` was never reached. **Defect (a) — `UseHeartbeatCapacity` is still off**, both by CLR default (`GpuSchedulingConfig.cs:314`, `bool` with no initializer) and explicitly at `EventHandlers.GpuCoordinator/appsettings.json:34` (`"UseHeartbeatCapacity": false`). **But its own precondition has changed, and that's worth flagging.** The doc comment at `GpuSchedulingConfig.cs:310-313` says flipping it is *"inert until either a node goes offline or a second GPU node is enabled."* **#822 enabled the SERVER lane** (`spikersoft-gpu-coordinator/docker-stack.yml:95-101`, `Nodes__0__NodeId=SERVER` / `Enabled=true`) alongside the 4090. So a second GPU node **is** now enabled — flipping this flag is no longer inert, and whatever reasoning kept it off may no longer apply. Worth a deliberate decision rather than leaving it at a default written for a single-node world. **On the #822 question specifically:** its work is a genuinely different axis. `VramBudgetTracker.CanEverFit` (`:465`) is an **admission** test — *could this request ever fit on this node* — and the `LeaseManagerService` changes (`:317-324` intake rejection, `:748-751` per-node `MaxSkipCount` scoping) are about queue fairness. None of that addresses a grant racing **physical** eviction, which is about the coordinator's accounting diverging from actual VRAM state. So #822 closing does not close this. **Defect (b), release-does-not-imply-freed: I did not verify it.** Flagging that rather than implying coverage — it needs the release path in `LeaseManagerService` read against what actually frees VRAM on the worker side. **Remaining:** (c) the disposal fix (small, and I'd do it first — it's a real VRAM leak on the retry path); (b) verification; (a) a decision on the flag now that its precondition holds; plus the DLQ replay, which is ops and not gradeable from git.

spikerj referenced this issue

2026-07-29 18:28:33 +00:00

[ArtStudio/Infra] Unified elastic GPU capacity — one coordinator model that scales from a single 24 GB card to multi-node, and runs identically on the M5 dev Mac #367

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#704