Fix up in spikersoft-backend PR #229, implementing exactly the recommended combination — (1)+(3) durable, (2) defense-in-depth:
AttributeErroris no longer a permanent failure.…
Follow-up #506 traced to this incident's fallout path: the image-description worker's GPU-lease client can be None when the GpuCoordinator (SERVER-pinned) is transiently unreachable during a flap…
Related: filed #506 — a separate image-description defect surfaced while investigating this service. When the GPU-lease client is momentarily None (startup race before _gpu_client is set, or…
Recovery note (2026-07-12 ~08:57): SERVER back at ~08:51 (~25 min outage), but 6 services wedged post-recovery
SERVER rejoined ~08:51; minio/backend/mongo-router self-recovered (minio…
ESCALATION (2026-07-12 ~08:29): SERVER is HARD DOWN again — recurrence, live outage
After ~45 min of stability, SERVER went Down / Unreachable again at ~08:26 (docker node ls → `SERVER…
Correction / update (2026-07-12 ~08:08)
Per node owner: SERVER was under super heavy load; the fusionio drive recovered on its own and the node is healthy again — so this reads as a…
Update (2026-07-12 ~08:05): storage side of the same fault — MinIO lost its drive
The SERVER node is Ready and the daemon has been stable ~26 min (no flap since ~07:39), but **`minio_minio…
Update (2026-07-12 ~07:18): the coordinator fix appears live — the image-description worker now gets its GPU lease (no more position 1/1 starvation) and proceeds to load the model. The…
PR opened from the pre-pushed branch: spikersoft-infrastructure #40 (fix/gpu-coordinator-4090-burst, commit 83916a4 — used as-is; it merges cleanly, the coordinator stack was untouched by…
Verified end-to-end: go-forward cutover complete (Joey opted to skip migrating old artifacts — all 143 pre-existing package versions were wiped so Gitea's blob dedup can't reference ghost files,…