Fixed and closing: PR #234 (merged, auto-deployed) — WasDuplicate flows through BookCreatedResult → book.created → orchestrator, which deletes the redundant extraction and completes the…
Both guards up: backend PR #238 (post-push docker manifest inspect retry gate — the deploy step can no longer start until the digest is actually servable; a push that never lands fails the…
Priority escalation — tonight made this ticket expensive. Three resource-exhaustion incidents in one session, all in the disk/exhaustion class the #381 probe already detects: (1) runner…
Scanner-flip preflight complete — plus a sequencing hazard to defuse:
✅ security.scan.requested has exactly ONE publisher (UploadOrchestrator) — and merged PR #236 mirrors the staging…
Concrete recurring instance of item C filed as 511 — quiz-generation's rolling update races its own image push: spec pinned to @sha256:X → swarm 'No such image' → same digest pullable minutes…
Filed exploration #510 — resilience to these SERVER flaps. Scopes the fixes cheapest-first: (A) restart-policy window on the 11 on-failure/maxAttempts=3/no-window services that keep wedging,…
Analysis + decision needed (this is an ops choice, not a code fix — laying out the paths):
Why it broke: the six artpipe stacks follow the movable artpipe-gpu label AND bind `/mnt/fusionio/…
Recurrence 2026-07-12 ~18:21 UTC (node owner reports another SERVER crash). Much milder than the 08:26 hard-down: SERVER came back Ready/Active/Reachable quickly (daemon flap, not a hard-down).…
Progress sweep: #500/#502/#508/#506/#497 all closed today with production evidence — captions flow from the 4090 with images AND model sourced from MinIO (110 completed captions in Mongo).…
All models verified loading in production — closing. nomic: 'Embedding model loaded successfully with 999 GPU layers' (17:25:25). Mistral: quiz generation produced real questions ('Total…
Verified and closing. Since #230 (COPY *.py + the image import smoke gate) the worker boots, consumes, leases, and captions — 110 completed captions in Mongo. The CI gate now runs `python3 -c…
Fix (#229) deployed and behaving — closing. The worker has been through multiple restart cycles since (deploys + a MinIO interruption mid-model-sync) with clean lease release/re-request each time…
Superseded by the MinIO model architecture and verified working — closing. The gated-repo problem was eliminated rather than tokened-around: the worker now syncs Qwen/Qwen3-VL-8B-Instruct…
Verified in production and closing. Since infra #40 deployed (SERVER lane parked, 4090 lane enabled, budget 24576 MB): the 16000 MB ImageDescription lease grants immediately (17:48:15,…