Fixed, deployed, and verified as far as possible without a real blip — closing. Infra #42 (merged, deployed live during the session): the healthcheck now runs an authenticated mc ls data-path…
Fixed and closing: PR #234 (merged, auto-deployed) — WasDuplicate flows through BookCreatedResult → book.created → orchestrator, which deletes the redundant extraction and completes the…
Fixed across three merged PRs and deployed via CI: #231 (0-question generations marked Failed + StartQuizAttempt guard + 400 mapping), #233 (page window no longer skips past small books — the…
Both guards up: backend PR #238 (post-push docker manifest inspect retry gate — the deploy step can no longer start until the digest is actually servable; a push that never lands fails the…
Priority escalation — tonight made this ticket expensive. Three resource-exhaustion incidents in one session, all in the disk/exhaustion class the #381 probe already detects: (1) runner…
Scanner-flip preflight complete — plus a sequencing hazard to defuse:
✅ security.scan.requested has exactly ONE publisher (UploadOrchestrator) — and merged PR #236 mirrors the staging…
Concrete recurring instance of item C filed as 511 — quiz-generation's rolling update races its own image push: spec pinned to @sha256:X → swarm 'No such image' → same digest pullable minutes…
Filed exploration #510 — resilience to these SERVER flaps. Scopes the fixes cheapest-first: (A) restart-policy window on the 11 on-failure/maxAttempts=3/no-window services that keep wedging,…
Analysis + decision needed (this is an ops choice, not a code fix — laying out the paths):
Why it broke: the six artpipe stacks follow the movable artpipe-gpu label AND bind `/mnt/fusionio/…
Recurrence 2026-07-12 ~18:21 UTC (node owner reports another SERVER crash). Much milder than the 08:26 hard-down: SERVER came back Ready/Active/Reachable quickly (daemon flap, not a hard-down).…
Progress sweep: #500/#502/#508/#506/#497 all closed today with production evidence — captions flow from the 4090 with images AND model sourced from MinIO (110 completed captions in Mongo).…
All models verified loading in production — closing. nomic: 'Embedding model loaded successfully with 999 GPU layers' (17:25:25). Mistral: quiz generation produced real questions ('Total…
Fix (#229) deployed and behaving — closing. The worker has been through multiple restart cycles since (deploys + a MinIO interruption mid-model-sync) with clean lease release/re-request each time…
Verified and closing. Since #230 (COPY *.py + the image import smoke gate) the worker boots, consumes, leases, and captions — 110 completed captions in Mongo. The CI gate now runs `python3 -c…