[ArtStudio/Infra] Unified elastic GPU capacity — one coordinator model that scales from a single 24 GB card to multi-node, and runs identically on the M5 dev Mac #367

New Issue

2026-07-04T22:45:09Z

spikerj commented

2026-07-04 22:45:09 +00:00

Epic #346 adjacent; touches every GPU tenant (ArtPipe stage services, Embeddings, QuizGeneration, Ocr, Decompile, LessonVideoProcessor). Motivation (Joey): hardware availability varies day to day — sometimes we can scale GPU compute out, today it's one 24 GB SERVER card in a development capacity, and on the M5 Pro Max (128 GB unified) we want to run the entire stack to debug and develop workflows. The system must degrade and scale without code changes — only capacity changes.

Current reality (three arbitration layers that don't know about each other)

GpuCoordinator: single replica, in-memory ledger, hard-coded TotalGpuVramMB=24576, implicitly one host (SERVER). No capacity discovery, no multi-node concept, Jetson GPUs unschedulable.
art_pipe's own admission gate (added in #357): dispatcher checks model vram_mb against free CUDA VRAM or unified memory — a second, independent gate that can disagree with the coordinator's ledger (double-gating on prod, sole gate on Mac).
ArtPipe:BypassGpuLease on the Mac — a third mode where the coordinator doesn't exist at all, so dev exercises different code paths than prod.

Direction

Capacity as data, not code: GPU hosts register capacity with the coordinator (node id, total/free VRAM or unified memory, backend cuda|mps, capabilities) — via config at minimum, node-agent/nvidia-smi heartbeats ideally (both stacks already exist: spikersoft-node-agent, nvidia-smi monitors). Adding/removing a GPU node = capacity appears/disappears; leases keep working.
Placement-aware leases: requests carry vram + backend requirement; coordinator grants against a specific node's budget; per-node stage services consume only their node's grants (Swarm placement stays the deployment mechanism — this is arbitration, not orchestration).
One gate: art_pipe's internal admission becomes advisory/telemetry (or is driven BY the lease grant) so there is exactly one arbiter; document which layer owns OOM protection.
Same code on the Mac: instead of bypass, the dev profile runs the real coordinator locally with auto-detected unified-memory capacity (art_pipe's hardware detect already reports it; mac-dev-up.sh can launch it) — dev/prod parity for the entire lease path. Keep bypass only as an escape hatch.
Honest degradation: when capacity is zero/saturated, requests queue with position events (#366, in flight) rather than failing — the student experience is "waiting in line", never an error, whether we have one card or ten.
Ops visibility: /gpu/status grows per-node capacity/queue; a line in the README's swarm docs on how to enroll a new GPU node.

Non-goals (this ticket)

Autoscaling service replicas from queue depth; Kubernetes; preemption. Single-replica in-memory ledger stays (documented) until capacity registration forces the Redis question.

Suggested phasing

U1 capacity registry + per-node budgets (config-first) → U2 placement-aware leases + per-node stage services → U3 Mac profile runs the real coordinator (retire bypass from the default dev path; wire into mac-dev-up) → U4 art_pipe gate demoted to telemetry → U5 heartbeat-driven capacity (node-agent/nvidia-smi).

Related: #366 (queue-position emit — lands first, this builds on it), #348 (artpipe-modeling service), #354 (model×VRAM matrix feeds per-node budgets), #357 (unified-memory admission), #359 (mac-dev-up).

Epic #346 adjacent; touches every GPU tenant (ArtPipe stage services, Embeddings, QuizGeneration, Ocr, Decompile, LessonVideoProcessor). Motivation (Joey): hardware availability varies day to day — sometimes we can scale GPU compute out, today it's one 24 GB SERVER card in a development capacity, and on the M5 Pro Max (128 GB unified) we want to run the entire stack to debug and develop workflows. The system must degrade and scale **without code changes** — only capacity changes. ## Current reality (three arbitration layers that don't know about each other) 1. **GpuCoordinator**: single replica, in-memory ledger, **hard-coded `TotalGpuVramMB=24576`**, implicitly one host (SERVER). No capacity discovery, no multi-node concept, Jetson GPUs unschedulable. 2. **art_pipe's own admission gate** (added in #357): dispatcher checks model `vram_mb` against free CUDA VRAM or unified memory — a second, independent gate that can disagree with the coordinator's ledger (double-gating on prod, sole gate on Mac). 3. **`ArtPipe:BypassGpuLease`** on the Mac — a third mode where the coordinator doesn't exist at all, so dev exercises different code paths than prod. ## Direction - **Capacity as data, not code**: GPU hosts register capacity with the coordinator (node id, total/free VRAM or unified memory, backend cuda|mps, capabilities) — via config at minimum, node-agent/nvidia-smi heartbeats ideally (both stacks already exist: `spikersoft-node-agent`, `nvidia-smi` monitors). Adding/removing a GPU node = capacity appears/disappears; leases keep working. - **Placement-aware leases**: requests carry vram + backend requirement; coordinator grants against a specific node's budget; per-node stage services consume only their node's grants (Swarm placement stays the deployment mechanism — this is arbitration, not orchestration). - **One gate**: art_pipe's internal admission becomes advisory/telemetry (or is driven BY the lease grant) so there is exactly one arbiter; document which layer owns OOM protection. - **Same code on the Mac**: instead of bypass, the dev profile runs the real coordinator locally with auto-detected unified-memory capacity (art_pipe's hardware detect already reports it; `mac-dev-up.sh` can launch it) — dev/prod parity for the entire lease path. Keep bypass only as an escape hatch. - **Honest degradation**: when capacity is zero/saturated, requests queue with position events (#366, in flight) rather than failing — the student experience is "waiting in line", never an error, whether we have one card or ten. - **Ops visibility**: `/gpu/status` grows per-node capacity/queue; a line in the README's swarm docs on how to enroll a new GPU node. ## Non-goals (this ticket) Autoscaling service replicas from queue depth; Kubernetes; preemption. Single-replica in-memory ledger stays (documented) until capacity registration forces the Redis question. ## Suggested phasing U1 capacity registry + per-node budgets (config-first) → U2 placement-aware leases + per-node stage services → U3 Mac profile runs the real coordinator (retire bypass from the default dev path; wire into mac-dev-up) → U4 art_pipe gate demoted to telemetry → U5 heartbeat-driven capacity (node-agent/nvidia-smi). Related: #366 (queue-position emit — lands first, this builds on it), #348 (artpipe-modeling service), #354 (model×VRAM matrix feeds per-node budgets), #357 (unified-memory admission), #359 (mac-dev-up).

spikerj added the enhancement label 2026-07-04 22:45:09 +00:00

spikerj referenced this issue

2026-07-04 23:22:04 +00:00

[Epic] ProArt 3D Art Studio — student asset pipeline (concept → model → texture → rig → animate → game-ready GLB) #346

spikerj commented

2026-07-04 23:30:42 +00:00

U1 merged (backend PR #74): GpuCoordinator:Nodes capacity registry with per-node ledgers, legacy-config synthesis pinned bit-identical, optional RequiredBackend on leases, NodeId on grants, per-node /gpu/status, coordinator README with the node-enrollment procedure. 446/446 filtered tests. Next: U2 placement-aware routing + per-node stage services (seam documented at VramBudgetTracker.TryAllocate), then U3 mac-dev-up runs the real coordinator.

U1 merged (backend PR #74): `GpuCoordinator:Nodes` capacity registry with per-node ledgers, legacy-config synthesis pinned bit-identical, optional `RequiredBackend` on leases, `NodeId` on grants, per-node `/gpu/status`, coordinator README with the node-enrollment procedure. 446/446 filtered tests. Next: U2 placement-aware routing + per-node stage services (seam documented at `VramBudgetTracker.TryAllocate`), then U3 mac-dev-up runs the real coordinator.

spikerj referenced this issue

2026-07-04 23:48:25 +00:00

[ArtStudio] Model-resident workers — one always-loaded model per service instance, task queues per model, coordinator-arbitrated, scale by replicas #368

spikerj referenced this issue

2026-07-05 01:59:21 +00:00

[Epic] ProArt 3D Art Studio — student asset pipeline (concept → model → texture → rig → animate → game-ready GLB) #346

spikerj commented

2026-07-05 07:58:49 +00:00

U2 (coordinator-side placement-aware routing) merged in spikersoft-backend PR #89 — best-fit/bin-packing in VramBudgetTracker.TryAllocate (the U1-documented seam), behind GpuCoordinator:PlacementAwareRouting (default false). Inert by default (proven byte-identical to pre-U2 first-fit) and additionally inert on the current single-SERVER-node registry even when on — it only changes placement once a 2nd GPU node is enabled in GpuCoordinator:Nodes. /gpu/status + RabbitMQ wire shapes untouched. 58/58 GpuCoordinator tests (8 new).

Enable (deploy-time, yours): set PlacementAwareRouting=true in the spikersoft-infrastructure coordinator stack + restart (single-replica in-memory — restart, don't scale).

#367 remaining: U2 worker-side per-node stage services · U3 (Mac runs the real coordinator via mac-dev-up — dev tooling) · U4 (art_pipe gate → telemetry) · U5 (heartbeat-driven capacity). Ticket stays open.

**U2 (coordinator-side placement-aware routing) merged** in spikersoft-backend PR #89 — best-fit/bin-packing in `VramBudgetTracker.TryAllocate` (the U1-documented seam), behind **`GpuCoordinator:PlacementAwareRouting` (default false)**. Inert by default (proven byte-identical to pre-U2 first-fit) and additionally inert on the current single-`SERVER`-node registry even when on — it only changes placement once a 2nd GPU node is enabled in `GpuCoordinator:Nodes`. `/gpu/status` + RabbitMQ wire shapes untouched. 58/58 GpuCoordinator tests (8 new). **Enable (deploy-time, yours):** set `PlacementAwareRouting=true` in the `spikersoft-infrastructure` coordinator stack + restart (single-replica in-memory — restart, don't scale). **#367 remaining:** U2 worker-side per-node stage services · U3 (Mac runs the real coordinator via mac-dev-up — dev tooling) · U4 (art_pipe gate → telemetry) · U5 (heartbeat-driven capacity). Ticket stays open.

spikerj commented

2026-07-05 08:22:01 +00:00

U5 (heartbeat-driven capacity) coordinator-side merged in spikersoft-backend PR #90. New gpu.node.heartbeat exchange/queue + GpuNodeHeartbeatMessage; VramBudgetTracker.ApplyHeartbeat maintains a per-node live view; /gpu/status gains an additive heartbeat block. Behavior behind GpuCoordinator:UseHeartbeatCapacity (default false) — OFF is byte-identical to today (proven); ON excludes stale/offline nodes and caps grants by reported free VRAM (can only reduce below config, never raise — U1 invariant preserved). 72/72 GpuCoordinator tests (14 new).

Node-side emitter deferred (cross-cutting: nvidia-smi publisher on prod nodes + Mac unified-memory emitter for U3) — the contract + exchange are in place for it. With no emitter, the flag must stay false (README notes this). Nothing to enable today.

#367 remaining: node-side heartbeat emitter · U2 worker-side per-node stage services · U3 (Mac runs the real coordinator) · U4 (art_pipe gate → telemetry). Ticket stays open.

**U5 (heartbeat-driven capacity) coordinator-side merged** in spikersoft-backend PR #90. New `gpu.node.heartbeat` exchange/queue + `GpuNodeHeartbeatMessage`; `VramBudgetTracker.ApplyHeartbeat` maintains a per-node live view; `/gpu/status` gains an additive `heartbeat` block. Behavior behind **`GpuCoordinator:UseHeartbeatCapacity` (default false)** — OFF is byte-identical to today (proven); ON excludes stale/offline nodes and caps grants by reported free VRAM (can only reduce below config, never raise — U1 invariant preserved). 72/72 GpuCoordinator tests (14 new). **Node-side emitter deferred** (cross-cutting: nvidia-smi publisher on prod nodes + Mac unified-memory emitter for U3) — the contract + exchange are in place for it. With no emitter, the flag must stay false (README notes this). Nothing to enable today. **#367 remaining:** node-side heartbeat emitter · U2 worker-side per-node stage services · U3 (Mac runs the real coordinator) · U4 (art_pipe gate → telemetry). Ticket stays open.

spikerj commented

2026-07-05 09:55:49 +00:00

Node-side heartbeat emitter merged in spikersoft-backend PR #96 — completes U5 into a functional loop (nodes report VRAM → coordinator consumes). Opt-in per node (GpuHeartbeat:Enabled, default false → registers nothing). IGpuVramProbe with NvidiaSmiVramProbe (pure ParseNvidiaSmiCsv, 16-case matrix) + StaticVramProbe (Mac dev box via config); GpuHeartbeatEmitterService publishes GpuNodeHeartbeatMessage to gpu.node.heartbeat with W3C trace context. Refactored the coordinator's heartbeat constants to a shared GpuHeartbeatTopology (producer+consumer one source; GpuCoordinator 72/72 unchanged). Wired disabled into NodeAgent (one emitter per node). 30 new tests, 104 filtered green.

Enable (per node, yours): NodeAgent GpuHeartbeat:Enabled=true + NodeId + Probe (nvidia-smi | static), then coordinator UseHeartbeatCapacity=true.

#367 status: U1 (registry) ✅ · U2 (placement routing) ✅ · U5 (heartbeat capacity: coordinator ✅ + emitter ✅) — the heartbeat loop is now end-to-end. Remaining: U3 (Mac runs the real coordinator via mac-dev-up — dev tooling) · U4 (art_pipe gate → telemetry) · real Apple-Silicon Metal VRAM probe (static covers Mac for now). Ticket stays open.

**Node-side heartbeat emitter merged** in spikersoft-backend PR #96 — completes U5 into a functional loop (nodes report VRAM → coordinator consumes). Opt-in per node (`GpuHeartbeat:Enabled`, default false → registers nothing). `IGpuVramProbe` with `NvidiaSmiVramProbe` (pure `ParseNvidiaSmiCsv`, 16-case matrix) + `StaticVramProbe` (Mac dev box via config); `GpuHeartbeatEmitterService` publishes `GpuNodeHeartbeatMessage` to `gpu.node.heartbeat` with W3C trace context. Refactored the coordinator's heartbeat constants to a shared `GpuHeartbeatTopology` (producer+consumer one source; GpuCoordinator 72/72 unchanged). Wired disabled into NodeAgent (one emitter per node). 30 new tests, 104 filtered green. **Enable (per node, yours):** NodeAgent `GpuHeartbeat:Enabled=true` + `NodeId` + `Probe` (nvidia-smi | static), then coordinator `UseHeartbeatCapacity=true`. **#367 status:** U1 (registry) ✅ · U2 (placement routing) ✅ · U5 (heartbeat capacity: coordinator ✅ + emitter ✅) — the heartbeat loop is now end-to-end. **Remaining:** U3 (Mac runs the real coordinator via mac-dev-up — dev tooling) · U4 (art_pipe gate → telemetry) · real Apple-Silicon Metal VRAM probe (static covers Mac for now). Ticket stays open.

spikerj commented

2026-07-05 11:58:45 +00:00

U4 (art_pipe gate → telemetry) — blocked, needs a design decision; not a clean flag-gated backend change. Investigated against primary source:

The admission gate U4 targets is Dispatcher._acquire_gpu (art_pipe/src/artpipe/dispatcher.py:522-579) in Eric's art_pipe repo, not spikersoft-backend. It is load-bearing OOM protection, not a redundant check: it reserves vram_mb × 1.4 (allocator/CUDA-context/fragmentation headroom), caps total reservation at 0.85 × total VRAM, reads live free VRAM via nvidia-smi (unified memory on Apple Silicon), and blocks on a condition variable — governing intra-dispatcher concurrency across worker subprocesses.

Backend has no lever over it: ArtPipeStageOrchestrator only requests a GpuCoordinator lease (or bypasses on Mac); SubprocessArtPipeStageExecutor passes the Python worker no VRAM/admission env. The coordinator's ledger does not model the 1.4× overhead, 0.85 cap, live hardware VRAM, or intra-dispatcher concurrency — and on Mac (BypassGpuLease=true) the coordinator isn't in the path at all, so the Python gate is the sole admission control there.

So demoting it now = a live OOM-safety regression, not an inert refactor. Correct sequencing: first teach GpuCoordinator (or the lease request) the overhead-factor + utilization-cap + concurrency accounting the Python gate owns, then demote _acquire_gpu to telemetry-only in the art_pipe repo behind a flag. Recommend splitting U4 into (a) backend: coordinator models the 1.4×/0.85 VRAM headroom, and (b) art_pipe: gate→telemetry once (a) lands. Nothing merged; master untouched.

**U4 (art_pipe gate → telemetry) — blocked, needs a design decision; not a clean flag-gated backend change.** Investigated against primary source: The admission gate U4 targets is `Dispatcher._acquire_gpu` (`art_pipe/src/artpipe/dispatcher.py:522-579`) in **Eric's `art_pipe` repo**, not spikersoft-backend. It is **load-bearing OOM protection, not a redundant check**: it reserves `vram_mb × 1.4` (allocator/CUDA-context/fragmentation headroom), caps total reservation at `0.85 × total VRAM`, reads **live** free VRAM via `nvidia-smi` (unified memory on Apple Silicon), and blocks on a condition variable — governing intra-dispatcher concurrency across worker subprocesses. Backend has **no lever** over it: `ArtPipeStageOrchestrator` only requests a `GpuCoordinator` lease (or bypasses on Mac); `SubprocessArtPipeStageExecutor` passes the Python worker no VRAM/admission env. The coordinator's ledger does **not** model the 1.4× overhead, 0.85 cap, live hardware VRAM, or intra-dispatcher concurrency — and on Mac (`BypassGpuLease=true`) the coordinator isn't in the path at all, so the Python gate is the *sole* admission control there. **So demoting it now = a live OOM-safety regression, not an inert refactor.** Correct sequencing: first teach `GpuCoordinator` (or the lease request) the overhead-factor + utilization-cap + concurrency accounting the Python gate owns, *then* demote `_acquire_gpu` to telemetry-only in the `art_pipe` repo behind a flag. Recommend splitting U4 into (a) backend: coordinator models the 1.4×/0.85 VRAM headroom, and (b) art_pipe: gate→telemetry once (a) lands. Nothing merged; master untouched.

spikerj referenced this issue

2026-07-09 00:08:57 +00:00

backend: expand integration coverage — art pipe messaging, safety RPC, GPU leases, quiz path #465

spikerj referenced this issue

2026-07-12 07:00:00 +00:00

[Bug][Infra][GPU] image-description lease starves — gpu-coordinator single-lane budget still on SERVER (8192MB) vs 16000MB needed; enable 4090 lane #500

spikerj referenced this issue

2026-07-12 07:02:25 +00:00

[Bug][Infra][GPU] image-description lease starves — gpu-coordinator single-lane budget still on SERVER (8192MB) vs 16000MB needed; enable 4090 lane #500

spikerj referenced this issue

2026-07-12 18:28:27 +00:00

[Bug][Infra][GPU] artpipe lane down on 4090 — workers Rejected on missing /mnt/fusionio/spikersoft/art_pipe bind after artpipe-gpu label move (collateral of #500) #501

spikerj referenced this issue

2026-07-13 00:40:41 +00:00

[Infra][MinIO] Migrate /mnt/fusionio/spikersoft/ai → ai-models bucket for embeddings + quiz-generation (epic #413) #527

spikerj referenced this issue

2026-07-13 01:14:15 +00:00

[CI][ArtPipe] Route baked model-image builds (ModelEnvImages + tier-3 finals) to the 4090 runner, off laptop-server #534

spikerj referenced this issue

2026-07-13 01:50:10 +00:00

[Enhancement][ArtPipe][Angular] AI QR code generation — SDXL qrcode_monster ControlNet per-model lane + authenticated AI button in the QR tool #535

spikerj referenced this issue

2026-07-13 20:09:31 +00:00

[Infra][MinIO] Migrate /mnt/fusionio/spikersoft/ai → ai-models bucket for embeddings + quiz-generation (epic #413) #527

spikerj referenced this issue

2026-07-13 20:33:03 +00:00

[Infra][MinIO] Migrate /mnt/fusionio/spikersoft/ai → ai-models bucket for embeddings + quiz-generation (epic #413) #527

spikerj referenced this issue

2026-07-17 02:07:00 +00:00

[Infra][MinIO] Migrate /mnt/fusionio/spikersoft/ai → ai-models bucket for embeddings + quiz-generation (epic #413) #527

spikerj referenced this issue

2026-07-18 23:56:34 +00:00

[GpuCoordinator][Bug] Lease grant races physical VRAM eviction — quiz-generation grant-fail-release cycles to DLQ while ImageDescription yields correctly #704

spikerj referenced this issue

2026-07-24 00:57:51 +00:00

GPU coordinator queue deadlocked after #467 node-routing deploy — Embeddings pinned to disabled SERVER lane wedges all GPU tenants #822

spikerj referenced this issue

2026-07-29 15:59:41 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting #582

spikerj commented

2026-07-29 18:28:33 +00:00

Epic status roll-up — audited against origin/master. U2's node routing shipped and its fallout is resolved; the elastic-capacity behaviour it enables is only partly switched on.

U2 (worker-side node routing, PR #467) is live, and I can trace its consequences through three tickets audited this week:

It made every lease request carry RequiredNodeId, which immediately caused #822 — Embeddings pinned to a disabled SERVER lane wedged every GPU tenant.
#822 is now closed: VramBudgetTracker.CanEverFit (:465) rejects permanently-unsatisfiable requests at intake (LeaseManagerService.cs:317-324), and the MaxSkipCount block is per-node scoped at :748-751 rather than global — so one pinned tenant can no longer block requests destined for other nodes. Regression test at LeaseManagerServiceTests.cs:780-799.
Both lanes are now enabled in config: spikersoft-gpu-coordinator/docker-stack.yml:95-101 has Nodes__0__NodeId=SERVER / Enabled=true alongside the 4090.

So the two-lane elastic model this epic describes is configured and its known deadlock is fixed.

But one switch the epic depends on is still off, and its precondition has quietly changed. GpuSchedulingConfig.cs:314 UseHeartbeatCapacity is false (CLR default and explicitly at GpuCoordinator/appsettings.json:34). The doc comment at :310-313 says flipping it is "inert until either a node goes offline or a second GPU node is enabled."

#822 enabled the second GPU node. So it is no longer inert — heartbeat-based capacity is exactly what makes scheduling adapt when a node's real VRAM diverges from its configured budget, which is this epic's "elastic" premise. Whatever reasoning kept it off was written for a single-lane world and deserves a fresh decision rather than inheriting a stale default.

Related and still open: #704. Its three defects (grant racing physical eviction, release-does-not-imply-freed, and a failure-path CUDA leak) are a different axis from #822's work — CanEverFit is an admission test, the MaxSkipCount change is queue fairness; neither addresses the coordinator's accounting diverging from actual VRAM. I found the leak precisely: QuizGenerationService.cs:179-198 — if LoadFromFileAsync succeeds and CreateContext throws, weights holds VRAM and is never disposed. Worst possible timing, since CreateContext is exactly where VRAM runs out, so the failure that triggers a retry strands the allocation that caused it. One-line fix (catch { weights.Dispose(); throw; }), and it matters more once both lanes are live.

Two operator prerequisites remain unverifiable from git, both recorded on #822: the NVML Driver/library version mismatch on SERVER, and the stack file's own ROLLOUT GUARD at docker-stack.yml:90-94 ("only enable BOTH lanes when every shared-mode GPU tenant runs an image with #467's routing").

Suggested: fix #704's disposal leak, decide on UseHeartbeatCapacity now that its precondition holds, confirm the two operator items, then this epic is down to live observation.

**Epic status roll-up** — audited against `origin/master`. **U2's node routing shipped and its fallout is resolved; the elastic-capacity behaviour it enables is only partly switched on.** **U2 (worker-side node routing, PR #467) is live**, and I can trace its consequences through three tickets audited this week: - It made every lease request carry `RequiredNodeId`, which immediately caused **#822** — Embeddings pinned to a disabled SERVER lane wedged every GPU tenant. - **#822 is now closed**: `VramBudgetTracker.CanEverFit` (`:465`) rejects permanently-unsatisfiable requests at intake (`LeaseManagerService.cs:317-324`), and the `MaxSkipCount` block is **per-node scoped** at `:748-751` rather than global — so one pinned tenant can no longer block requests destined for other nodes. Regression test at `LeaseManagerServiceTests.cs:780-799`. - **Both lanes are now enabled in config**: `spikersoft-gpu-coordinator/docker-stack.yml:95-101` has `Nodes__0__NodeId=SERVER` / `Enabled=true` alongside the 4090. So the two-lane elastic model this epic describes is configured and its known deadlock is fixed. **But one switch the epic depends on is still off, and its precondition has quietly changed.** `GpuSchedulingConfig.cs:314` `UseHeartbeatCapacity` is `false` (CLR default *and* explicitly at `GpuCoordinator/appsettings.json:34`). The doc comment at `:310-313` says flipping it is *"inert until either a node goes offline or a second GPU node is enabled."* **#822 enabled the second GPU node.** So it is no longer inert — heartbeat-based capacity is exactly what makes scheduling adapt when a node's real VRAM diverges from its configured budget, which is this epic's "elastic" premise. Whatever reasoning kept it off was written for a single-lane world and deserves a fresh decision rather than inheriting a stale default. **Related and still open: #704.** Its three defects (grant racing physical eviction, release-does-not-imply-freed, and a failure-path CUDA leak) are a **different axis** from #822's work — `CanEverFit` is an admission test, the `MaxSkipCount` change is queue fairness; neither addresses the coordinator's accounting diverging from actual VRAM. I found the leak precisely: `QuizGenerationService.cs:179-198` — if `LoadFromFileAsync` succeeds and `CreateContext` throws, `weights` holds VRAM and is never disposed. Worst possible timing, since `CreateContext` is exactly where VRAM runs out, so the failure that triggers a retry strands the allocation that caused it. One-line fix (`catch { weights.Dispose(); throw; }`), and it matters more once both lanes are live. **Two operator prerequisites remain unverifiable from git**, both recorded on #822: the NVML `Driver/library version mismatch` on SERVER, and the stack file's own ROLLOUT GUARD at `docker-stack.yml:90-94` ("only enable BOTH lanes when every shared-mode GPU tenant runs an image with #467's routing"). **Suggested:** fix #704's disposal leak, decide on `UseHeartbeatCapacity` now that its precondition holds, confirm the two operator items, then this epic is down to live observation.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#367