Follow-up #506 traced to this incident's fallout path: the image-description worker's GPU-lease client can be None when the GpuCoordinator (SERVER-pinned) is transiently unreachable during a flap…
Recovery note (2026-07-12 ~08:57): SERVER back at ~08:51 (~25 min outage), but 6 services wedged post-recovery
SERVER rejoined ~08:51; minio/backend/mongo-router self-recovered (minio…
ESCALATION (2026-07-12 ~08:29): SERVER is HARD DOWN again — recurrence, live outage
After ~45 min of stability, SERVER went Down / Unreachable again at ~08:26 (docker node ls → `SERVER…
Correction / update (2026-07-12 ~08:08)
Per node owner: SERVER was under super heavy load; the fusionio drive recovered on its own and the node is healthy again — so this reads as a…
Update (2026-07-12 ~08:05): storage side of the same fault — MinIO lost its drive
The SERVER node is Ready and the daemon has been stable ~26 min (no flap since ~07:39), but **`minio_minio…
Update (2026-07-12 ~07:18): the coordinator fix appears live — the image-description worker now gets its GPU lease (no more position 1/1 starvation) and proceeds to load the model. The…
PR opened from the pre-pushed branch: spikersoft-infrastructure #40 (fix/gpu-coordinator-4090-burst, commit 83916a4 — used as-is; it merges cleanly, the coordinator stack was untouched by…
Verified end-to-end: go-forward cutover complete (Joey opted to skip migrating old artifacts — all 143 pre-existing package versions were wiped so Gitea's blob dedup can't reference ghost files,…
Verified deployed: Seq shows DeletedBookPurgeService active: purging books soft-deleted more than 30 days ago, every 1.00:00:00 on the rolled file-movement (backend #227 + infra #39, both…
Option 1 implemented (Joey's pick — retention purge, 30-day undelete window):
- spikersoft-backend #227 —
DeletedBookPurgeServicein file-movement (the file-lifecycle owner with the…