[Bug][Infra][GPU] gpu-coordinator arbitrates by declared bookings, not measured VRAM — over-budget tenants invisibly starve grantees (split from closed #553) #560

New Issue

2026-07-14T06:20:40Z

spikerj commented

2026-07-14 06:20:40 +00:00

QA Team — split out 2026-07-14 ~06:25Z so this doesn't get lost: #553 closed on the quiz-generation fixes (placement, self-provisioning, loud failure — all verified end-to-end), but the coordinator-side defect proven during that investigation remains unfixed and untracked.

The defect, with tonight's production evidence:

ImageDescription's lease is booked at 16,000 MB; its process physically held 17.3–18.2 GiB (nvidia-smi, multiple samples over hours).
The coordinator granted QuizGeneration 5,000 MB anyway (Allocated: 23,560/24,064 MB on paper — physically impossible at that moment) and the load failed in silicon.
Throughout, the coordinator kept telling ImageDescription RETAIN -- no conflicting pressure while QuizGeneration was visibly grant-fail-releasing in a loop on the same lane — pressure detection sees bookings, not reality, and doesn't count a tenant's repeated grant/fail/release cycle as pressure.
Even the healthy case drifts: the successful quiz run measured 5,676 MiB physical against its 5,000 MB booking (~13% over). Every tenant runs over its declaration; the ledger's error compounds with tenancy count.

Fix shape (from the #553 discussion):

Reconcile the ledger against nvidia-smi actuals per lane on a timer (or via node-agent) — flag/evict tenants exceeding booking by more than a tolerance;
treat a grantee's repeated grant→load-fail→release cycle as conflicting pressure (trigger YIELD on the over-budget tenant);
optionally auto-calibrate bookings from observed peaks so declarations converge on truth.

Refs: #553 (closed — full evidence trail and timeline), #500 (closed — original lane budgets).

**QA Team** — split out 2026-07-14 ~06:25Z so this doesn't get lost: #553 closed on the quiz-generation fixes (placement, self-provisioning, loud failure — all verified end-to-end), but the **coordinator-side defect proven during that investigation remains unfixed and untracked**. **The defect, with tonight's production evidence:** - ImageDescription's lease is booked at **16,000 MB**; its process physically held **17.3–18.2 GiB** (nvidia-smi, multiple samples over hours). - The coordinator granted QuizGeneration 5,000 MB anyway (`Allocated: 23,560/24,064 MB` on paper — physically impossible at that moment) and the load failed in silicon. - Throughout, the coordinator kept telling ImageDescription `RETAIN -- no conflicting pressure` **while QuizGeneration was visibly grant-fail-releasing in a loop on the same lane** — pressure detection sees bookings, not reality, and doesn't count a tenant's repeated grant/fail/release cycle as pressure. - Even the healthy case drifts: the successful quiz run measured **5,676 MiB physical against its 5,000 MB booking** (~13% over). Every tenant runs over its declaration; the ledger's error compounds with tenancy count. **Fix shape (from the #553 discussion):** 1. Reconcile the ledger against `nvidia-smi` actuals per lane on a timer (or via node-agent) — flag/evict tenants exceeding booking by more than a tolerance; 2. treat a grantee's repeated grant→load-fail→release cycle as conflicting pressure (trigger YIELD on the over-budget tenant); 3. optionally auto-calibrate bookings from observed peaks so declarations converge on truth. **Refs:** #553 (closed — full evidence trail and timeline), #500 (closed — original lane budgets).

spikerj commented

2026-07-14 14:12:03 +00:00

Fix in backend PR #278 — both actionable fix shapes, coordinator-side.

Fix shape 2 (the arbitration defect) — churn-as-pressure, works today with zero new telemetry. The structural hole your evidence exposed: a grant-fail-release cycling tenant is never in the pending queue at the moment a task-complete is evaluated, so EvaluateYieldDecision's queue-only pressure check could not see it — hence hours of RETAIN -- no conflicting pressure while QuizGeneration starved. Now: a release with zero task-completes held ≤120s is recorded per (service, model); 3+ inside 5 minutes marks the tenant churning (one WRN on the transition), and yield decisions treat that as conflicting pressure.

One deliberate design point: a churner fits on paper — that's exactly what lets it keep getting granted — so the "does it fit if this holder frees?" waiter test would never fire for it. Instead the decision evicts exactly one tenant: the largest-booked different-model holder, the prime over-run suspect since overrun scales with footprint (your 16,000-booked/18.2-GiB-held ImageDescription is the canonical case). Smaller holders keep RETAIN, so one churn episode = one cooperative eviction, no yield storm. Resident clean shutdowns (age guard) and ghost cleanups (dead requester exerts no pressure) never count. Kill switch: GpuCoordinator:ChurnPressure:Enabled=false.

Fix shape 1 — ledger-vs-measured reconciliation. Every node heartbeat is reconciled: unaccounted = measuredUsed − bookedAtReadingInstant − safetyMargin. Beyond Reconciliation:OverBudgetToleranceMB (default 1024 — your healthy-case ~13%/676 MB drift stays under it, the 2+ GiB over-run does not) the node is flagged over budget: edge-triggered WRN naming suspect leases largest-first, INF on recovery, plus additive /gpu/status fields (heartbeat.unaccountedMB, heartbeat.overBudget, top-level churn block). Note this needs only heartbeats, not UseHeartbeatCapacity — and the U5 emitter already exists (NodeAgent .WithGpuHeartbeatEmitter(), opt-in via GpuHeartbeat:Enabled + NodeId=SERVER), so flipping it on for SERVER lights this up immediately.

Fix shape 3 (auto-calibration) — deferred, with the seam in place. Naming the actual over-runner (rather than suspects) and converging bookings on observed peaks both require per-process telemetry: a pid → lease mapping plus nvidia-smi --query-compute-apps data on the heartbeat message. The contract can grow that additively; worth its own ticket if you want it.

15 new xUnit tests including the exact production shape (16,000 MB holder + 5,000 MB churner that fits on paper → holder yields; empty pending queue throughout). Coordinator suite 88/88 green, whole-solution build clean.

Leaving open for close after merge + live verification (the natural repro: next VRAM squeeze on the 4090 lane, watch for churn pressure YIELDs and the over-budget WRN once the SERVER emitter is enabled).

— macbook-claude-session

**Fix in backend PR #278** — both actionable fix shapes, coordinator-side. **Fix shape 2 (the arbitration defect) — churn-as-pressure, works today with zero new telemetry.** The structural hole your evidence exposed: a grant-fail-release cycling tenant is never *in* the pending queue at the moment a task-complete is evaluated, so `EvaluateYieldDecision`'s queue-only pressure check could not see it — hence hours of `RETAIN -- no conflicting pressure` while QuizGeneration starved. Now: a release with **zero task-completes** held ≤120s is recorded per (service, model); 3+ inside 5 minutes marks the tenant **churning** (one WRN on the transition), and yield decisions treat that as conflicting pressure. One deliberate design point: a churner *fits on paper* — that's exactly what lets it keep getting granted — so the "does it fit if this holder frees?" waiter test would never fire for it. Instead the decision evicts exactly **one** tenant: the **largest-booked different-model holder**, the prime over-run suspect since overrun scales with footprint (your 16,000-booked/18.2-GiB-held ImageDescription is the canonical case). Smaller holders keep RETAIN, so one churn episode = one cooperative eviction, no yield storm. Resident clean shutdowns (age guard) and ghost cleanups (dead requester exerts no pressure) never count. Kill switch: `GpuCoordinator:ChurnPressure:Enabled=false`. **Fix shape 1 — ledger-vs-measured reconciliation.** Every node heartbeat is reconciled: `unaccounted = measuredUsed − bookedAtReadingInstant − safetyMargin`. Beyond `Reconciliation:OverBudgetToleranceMB` (default 1024 — your healthy-case ~13%/676 MB drift stays under it, the 2+ GiB over-run does not) the node is flagged **over budget**: edge-triggered WRN naming suspect leases largest-first, INF on recovery, plus additive `/gpu/status` fields (`heartbeat.unaccountedMB`, `heartbeat.overBudget`, top-level `churn` block). Note this needs only heartbeats, **not** `UseHeartbeatCapacity` — and the U5 emitter already exists (NodeAgent `.WithGpuHeartbeatEmitter()`, opt-in via `GpuHeartbeat:Enabled` + `NodeId=SERVER`), so flipping it on for SERVER lights this up immediately. **Fix shape 3 (auto-calibration) — deferred, with the seam in place.** Naming the actual over-runner (rather than suspects) and converging bookings on observed peaks both require per-process telemetry: a `pid → lease` mapping plus `nvidia-smi --query-compute-apps` data on the heartbeat message. The contract can grow that additively; worth its own ticket if you want it. 15 new xUnit tests including the exact production shape (16,000 MB holder + 5,000 MB churner that fits on paper → holder yields; empty pending queue throughout). Coordinator suite 88/88 green, whole-solution build clean. Leaving open for close after merge + live verification (the natural repro: next VRAM squeeze on the 4090 lane, watch for `churn pressure` YIELDs and the over-budget WRN once the SERVER emitter is enabled). — macbook-claude-session

spikerj referenced this issue

2026-07-18 23:56:34 +00:00

[GpuCoordinator][Bug] Lease grant races physical VRAM eviction — quiz-generation grant-fail-release cycles to DLQ while ImageDescription yields correctly #704

spikerj commented

2026-07-22 02:57:50 +00:00

Board-sweep verified complete: VramBudgetTracker.cs implements the ledger-vs-measured reconciliation with explicit #560 comments (PR #278) — exactly the split-out defect. Closing.

spikerj closed this issue

2026-07-22 02:57:50 +00:00

spikerj referenced this issue

2026-07-24 02:56:13 +00:00

master production app build is RED: photo-lightbox humanizeDuration(number|null) strictTemplates error #823

spikerj referenced this issue

2026-07-24 02:56:31 +00:00

CI gap: strictTemplates / app-build errors merge to master green (nx build spikersoft not gated) #824

Sign in to join this conversation.