Tonight's runs add the live evidence on top of the version check: build_env_images run 6612 (the #537 weight-patterns…
Live production proof from tonight, closing the loop on verification: the backend tasks list shows the superseded feat(media): S3-backed deliver… merge's arm64v8 build_and_publish tasks…
QA Team — new casualty + revised storage hypothesis 2026-07-14 ~03:25Z:
Seq was hit too. On dreamstream2, Seq's event store threw Transport endpoint is not connected (os error 107)…
QA Team — major timeline update 2026-07-14 ~03:20Z, from dreamstream1's kernel journal (pulled by ops):
dreamstream1 crashed and rebooted tonight at 02:28:06Z — its journal begins at…
Consumer audit done (repo-side) — recommendation: retire, don't migrate.
Findings:
- Zero code consumers: not one reference to sftp (path, host, or port) anywhere in spikersoft-backen…
QA Team — alert-tuning addendum 2026-07-14 ~03:15Z: the fleet kernel errors turn out to be PCIe AER corrected errors (see #552 for the full diagnosis). AER corrected errors can burst by…
Fixed and CI-verified. Angular PR #191 (merged): the three i18n asset imports in testing/transloco-test-setup.ts are now the workspace's second documented boundary exemption (inline…
QA Team — kernel-error signature identified 2026-07-14 ~03:15Z (kernel log pulled from a dreamstream node by ops):
The fleet 'kernel errors' are **PCIe AER corrected-error bursts on the NVMe…
QA Team — refinement/correction 2026-07-14 ~03:10Z on the SignalR errors noted earlier: Jaeger traces show the 02:27:48–49Z ForwardClusterEventToSignalR failures were **Redis PUBLISH…
Problem 1 (exit-0) is fixed on master — Program.cs now registers IDockerClient with the annotation explaining exactly this failure: ServiceAutoReconciler (#510-B) resolved `IDockerClient…
Code-side this is fixed on master (198ad66 — backups pinned node.hostname == SERVER like its siblings, plus the $$ escaping so pg_dump's vars survive the compose parser). What remains…
QA Team — corruption verdict 2026-07-14 ~03:00Z: durable, not transient. The missing chunk number 0 for toast value 131293 in pg_toast_2619 error re-fired at 02:44:50Z — well after…
Root cause (confirmed in code + config)
Three numbers that don't add up:
- Declared: image-description leases
GPU_VRAM_REQUIREMENT_MB=16000(its stack file) - Real: Qwen3-VL-8B fp16…
QA Team — predictive heads-up 2026-07-14 ~02:55Z: the kernel-error class that preceded dreamstream4's panic is actively accumulating on other nodes. system-remediation in the last 4h:
-…
QA Team — evidence update + topology correction 2026-07-14 ~02:55Z:
Correction: the contended GPU is the 4090 node's card (the #500 "4090 lane" is in service), not a GPU on SERVER —…
Registry/source findings from repo-side investigation (no host access from this session):
1. The image is unpullable — its namespace no longer exists. The stack pins `git.spikersoft.com/jspik…
Cross-linking evidence from tonight: dreamstream4 being Down is very likely the cause of the GlusterFS /mnt/infrastructure distress Joey observed ~02:14-02:25Z (and possibly of the brief…