Backend PR spikersoft-backend#254 (Dockerfile + workflow, stacked on #253) + infra PR spikersoft-infrastructure#57 (stack file, GATED). The new workflow deliberately does NOT push tags/manifest/dep…
QA watch 2026-07-13 ~19:10Z — still reproducing, exactly 6/min: 60 'authorization not found' in the 10 min to 19:07Z on influxDB_influxdb (dreamstream4), steady 10s cadence. The server log line…
Infra half: spikersoft-infrastructure PR #56 (GATED) — API stack Storage__ env + reference-data provisioning recipe in the migration doc. Backend PR #253 also gained the API_S3_SECRET_KEY deploy…
QA watch 2026-07-13 ~19:03Z — CUTOVER VERIFIED (spec-level). The three stood-down stacks were spec-updated at 18:57:48-51Z and are digest-pinned to the 18:53Z finals: artpipe-model-blender@bf4238f…
GeoIP half: backend PR #253 (under #530) makes GeoIpService self-provision GeoLite2 from the MinIO reference-data bucket — fresh deploys stop needing /app/data. Postal half remains: it's a Mongo…
PR spikersoft-backend#253 (feat/530-geolite2-reference-data, stacked on #252/#533): GeoIpUpdateTaskRunner uploads each fresh mmdb to the reference-data bucket; GeoIpService lazily self-provisions…
QA watch 2026-07-13 ~18:58Z — RECOMMEND CLOSE: scaffolding proven end-to-end TWICE. (1) Phase 1: safetycheck env → cascade → final → deployed 1/1 on the 4090, consumers verified (#517, 07-13…
QA watch 2026-07-13 ~18:58Z — FULL image gate satisfied. All four finals published at 18:53Z, version 14556a4-746b1b6 (backend head + artpipe head): artpipe-model-blender, artpipe-model-sdxllightn…
QA watch 2026-07-13 ~18:55Z — env-image gate SATISFIED, finals in flight. All 3 Phase-2 env manifests are published from artpipe 746b1b6: blender (15:27Z), sdxllightning (18:36Z, 65.2 GB — see…
QA watch 2026-07-13 ~18:55Z — DOWNGRADE from hard blocker to optimization: the push now SUCCEEDS from the 4090. Run 10821 (success 18:48Z) published artpipe-model-env-sdxllightning:latest at…
PR spikersoft-backend#252 (feat/533-shared-objectstore) implements this: new SpikerSoft.Storage project with IObjectStore + S3ObjectStore + shared ObjectKeys.TryMapKey shim + AddS3ObjectStore()/Use…
QA watch 2026-07-13 ~18:40Z — removal is now zero-risk. The typo'd stack's task (spiekersoft-docker-monitor, on SERVER) exited cleanly ~16:40Z and sits 0/1 Complete (on-failure policy won't…
QA watch 2026-07-13 ~18:40Z — filter fix NOT landed, failures continuing.
artpipe main venv_setup.py:263-264 is still the unfiltered call: snapshot_download(weights, local_dir_use_symlinks=False)…
QA watch 2026-07-13 ~18:40Z — status: HALF landed, keep open.
✅ ModelEnvImages half: artpipe 268f9ad (merge f7c0b15) is in main — .gitea/workflows/model-env-images.yml:50 = runs-on: [ubuntu-amd…
Verdict: bad RAM, quarantined; card/driver exonerated. Crash cluster resolved pending DIMM replacement.
- Root cause: defective physical memory region ~0xFDFC00000 (identical pfn neighborhood…
Exit-0 wedge occurrence #3 (2026-07-13 ~18:2xZ, post-GRUB-reboot recovery): metadata-extractor + security-scanner + upload-coordinator all Complete-wedged again — upload pipeline down until…
ROOT CAUSE FOUND + MITIGATION APPLIED (2026-07-13 ~18:1xZ, by joey): a bad memory sector was identified on SERVER and a GRUB memory exclusion applied for the affected 16 MB region (badram/memma…