QA watch 2026-07-13 ~19:41Z — dreamstream6 REJOINED at ~19:38Z (outage ~16:30→19:38Z, ≈3h). Host pings again, node State=ready.
Workload recovery: redis-cluster_redis-node-6 Running (one…
Additional finding while explaining the image size (2026-07-13 ~19:35Z) — the bloat has an exact inverse bug: the weights the model DOES need at runtime are NOT all baked.
Bloat mechanics,…
Pre-flight hardening PR: spikersoft-infrastructure#59 — --preserve-digests on every copy + optional authenticated Hub pulls (32/37 refs are Hub; an anonymous multi-arch sweep is ~100+ manifest…
Postal half appears RESOLVED in prod: /healthz (checked 2026-07-13 ~15:35Z) shows 'Postal Code Data' Healthy — only TileServerGL (#484) is non-Healthy. GeoIP half: fix chain is in review (backend…
Status split (verified live 2026-07-13 ~15:35Z):
FIXED — the 503 half: 41a25ab (failureStatus: Degraded + check honors Registration.FailureStatus) is deployed; /healthz now returns HTTP 200 with…
Hardening PR: spikersoft-infrastructure#58 — builder prune size cap (--keep-storage 60GB, LRU; the until=48h filter was actually exempting the hours-old outage cache), hourly timer, emergency…
Status: (1) exit-0 root cause fixed on master in 28c62fb (missing IDockerClient DI registration; ValidateOnBuild regression tests added). (3) CI stays-up smoke test: backend PR #255 — 15s no-deps…
QA watch 2026-07-13 ~19:18Z — the mirror is EMPTY and all three runs so far failed as outage collateral. MirrorUpstreamImages runs 10776/10779/10780 (02:10, 03:07, 04:00Z — manual dispatches;…
Backend PR spikersoft-backend#254 (Dockerfile + workflow, stacked on #253) + infra PR spikersoft-infrastructure#57 (stack file, GATED). The new workflow deliberately does NOT push tags/manifest/dep…
QA watch 2026-07-13 ~19:10Z — still reproducing, exactly 6/min: 60 'authorization not found' in the 10 min to 19:07Z on influxDB_influxdb (dreamstream4), steady 10s cadence. The server log line…
Infra half: spikersoft-infrastructure PR #56 (GATED) — API stack Storage__ env + reference-data provisioning recipe in the migration doc. Backend PR #253 also gained the API_S3_SECRET_KEY deploy…
QA watch 2026-07-13 ~19:03Z — CUTOVER VERIFIED (spec-level). The three stood-down stacks were spec-updated at 18:57:48-51Z and are digest-pinned to the 18:53Z finals: artpipe-model-blender@bf4238f…
GeoIP half: backend PR #253 (under #530) makes GeoIpService self-provision GeoLite2 from the MinIO reference-data bucket — fresh deploys stop needing /app/data. Postal half remains: it's a Mongo…
PR spikersoft-backend#253 (feat/530-geolite2-reference-data, stacked on #252/#533): GeoIpUpdateTaskRunner uploads each fresh mmdb to the reference-data bucket; GeoIpService lazily self-provisions…
QA watch 2026-07-13 ~18:58Z — RECOMMEND CLOSE: scaffolding proven end-to-end TWICE. (1) Phase 1: safetycheck env → cascade → final → deployed 1/1 on the 4090, consumers verified (#517, 07-13…
QA watch 2026-07-13 ~18:58Z — FULL image gate satisfied. All four finals published at 18:53Z, version 14556a4-746b1b6 (backend head + artpipe head): artpipe-model-blender, artpipe-model-sdxllightn…
QA watch 2026-07-13 ~18:55Z — env-image gate SATISFIED, finals in flight. All 3 Phase-2 env manifests are published from artpipe 746b1b6: blender (15:27Z), sdxllightning (18:36Z, 65.2 GB — see…
QA watch 2026-07-13 ~18:55Z — DOWNGRADE from hard blocker to optimization: the push now SUCCEEDS from the 4090. Run 10821 (success 18:48Z) published artpipe-model-env-sdxllightning:latest at…