Fixed across three merged PRs and deployed via CI: #231 (0-question generations marked Failed + StartQuizAttempt guard + 400 mapping), #233 (page window no longer skips past small books — the…
Both guards up: backend PR #238 (post-push docker manifest inspect retry gate — the deploy step can no longer start until the digest is actually servable; a push that never lands fails the…
Priority escalation — tonight made this ticket expensive. Three resource-exhaustion incidents in one session, all in the disk/exhaustion class the #381 probe already detects: (1) runner…
Scanner-flip preflight complete — plus a sequencing hazard to defuse:
✅ security.scan.requested has exactly ONE publisher (UploadOrchestrator) — and merged PR #236 mirrors the staging…
Concrete recurring instance of item C filed as 511 — quiz-generation's rolling update races its own image push: spec pinned to @sha256:X → swarm 'No such image' → same digest pullable minutes…
Filed exploration #510 — resilience to these SERVER flaps. Scopes the fixes cheapest-first: (A) restart-policy window on the 11 on-failure/maxAttempts=3/no-window services that keep wedging,…
Analysis + decision needed (this is an ops choice, not a code fix — laying out the paths):
Why it broke: the six artpipe stacks follow the movable artpipe-gpu label AND bind `/mnt/fusionio/…
Recurrence 2026-07-12 ~18:21 UTC (node owner reports another SERVER crash). Much milder than the 08:26 hard-down: SERVER came back Ready/Active/Reachable quickly (daemon flap, not a hard-down).…
Progress sweep: #500/#502/#508/#506/#497 all closed today with production evidence — captions flow from the 4090 with images AND model sourced from MinIO (110 completed captions in Mongo).…
All models verified loading in production — closing. nomic: 'Embedding model loaded successfully with 999 GPU layers' (17:25:25). Mistral: quiz generation produced real questions ('Total…
Fix (#229) deployed and behaving — closing. The worker has been through multiple restart cycles since (deploys + a MinIO interruption mid-model-sync) with clean lease release/re-request each time…
Verified and closing. Since #230 (COPY *.py + the image import smoke gate) the worker boots, consumes, leases, and captions — 110 completed captions in Mongo. The CI gate now runs `python3 -c…
Verified in production and closing. Since infra #40 deployed (SERVER lane parked, 4090 lane enabled, budget 24576 MB): the 16000 MB ImageDescription lease grants immediately (17:48:15,…
Superseded by the MinIO model architecture and verified working — closing. The gated-repo problem was eliminated rather than tokened-around: the worker now syncs Qwen/Qwen3-VL-8B-Instruct…
Root-caused WITH production data and fixed — PR #234. Forensics (read-only via mongo-router): the bad book holds pages [1,1,2,2,3,3] under TWO ISBNs (both uploads of the same PDF); the…