spikerj commented on issue spikerj/spikersoft-issues#538

2026-07-14 04:02:45 +00:00

[Infra][Network] LAN traffic to *.spikersoft.com NAT-hairpins through the UDM — local DNS records + direct Gitea→MinIO path

Confirmed live tonight (2026-07-14) during a large artpipe env-image push: the registry write path pays this tax twice, because since #498 Gitea does not store package blobs — it relays them to…

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 03:30:42 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

Follow-up finding worth its own fix — quiz-gen's lease is under-declared too (same "books lied to" class as this ticket's root cause).

Surfaced while implementing PR #272: quiz-generation…

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 03:28:30 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

Silent-degradation fix up: spikersoft-backend PR #272

spikerj/spikersoft-backend#272 — the follow-up flagged in my root-cause comment (option (a), plus the…

spikerj commented on issue spikerj/spikersoft-issues#519

2026-07-14 03:27:30 +00:00

[ArtPipe] Baked images Phase 3: prodstages image + live modeling-lane cutover (epic #515)

QA Team — active blocker found 2026-07-14 ~03:35Z: spikersoft-artpipe-modeling_artpipe-modeling is in a hard reject-loop and the modeling lane is down.

Every ~5s since the **18:57Z Jul…

spikerj commented on issue spikerj/spikersoft-issues#538

2026-07-14 03:22:05 +00:00

[Infra][Network] LAN traffic to *.spikersoft.com NAT-hairpins through the UDM — local DNS records + direct Gitea→MinIO path

QA Team — stronger evidence 2026-07-14 ~03:30Z that this is more than LAN-hairpin fragility: during dreamstream1's crash/reboot window tonight (down ~02:27:50Z, boot 02:28:06Z — see #552),…

spikerj commented on issue spikerj/spikersoft-issues#546

2026-07-14 03:21:58 +00:00

[Security][Backend] OpenBao Phase 2: SpikerSoft.Secrets host-builder seam — service creds from Bao (epic #543)

Phase 2 seam is up for review: spikersoft-backend PR #270. New SpikerSoft.Secrets project (ISecretSource + VaultSecretSource:…

spikerj commented on issue spikerj/spikersoft-issues#510

2026-07-14 03:21:36 +00:00

[Exploration][Infra] Resilience to SERVER node flaps — restart-policy windows, auto-reconcile sweep, de-SPOF the SERVER-pinned stateful tier

Infra-wide swarm-resilience audit (#510) — stack-file sweep, 2026-07-13

Sweep of every docker-stack*.yml in spikersoft-infrastructure @ master 4192b90 (67 files parsed), classified against…

spikerj opened issue spikerj/spikersoft-issues#556

2026-07-14 03:17:39 +00:00

[Bug][Backend][Infra] tiles.spikersoft.com has no backing service — API status/health canary fetches a terrain tile every few minutes and 404s 100% of the time

spikerj closed issue spikerj/spikersoft-issues#548

2026-07-14 03:04:14 +00:00

[CI][Backend] Multi-arch create_manifest amends MUTABLE :amd64/:arm64v8 tags — out-of-order arch pushes can stitch a mixed-commit :latest (all backend multi-arch workflows)

spikerj closed issue spikerj/spikersoft-issues#536

2026-07-14 03:04:14 +00:00

[CI][ArtPipe] ModelEnvImages has no concurrency guard — concurrent runs race the same :latest tags (observed live)

spikerj commented on issue spikerj/spikersoft-issues#536

2026-07-14 03:04:03 +00:00

[CI][ArtPipe] ModelEnvImages has no concurrency guard — concurrent runs race the same :latest tags (observed live)

Tonight's runs add the live evidence on top of the version check: build_env_images run 6612 (the #537 weight-patterns…

spikerj commented on issue spikerj/spikersoft-issues#548

2026-07-14 03:04:02 +00:00

[CI][Backend] Multi-arch create_manifest amends MUTABLE :amd64/:arm64v8 tags — out-of-order arch pushes can stitch a mixed-commit :latest (all backend multi-arch workflows)

Live production proof from tonight, closing the loop on verification: the backend tasks list shows the superseded feat(media): S3-backed deliver… merge's arm64v8 build_and_publish tasks…

spikerj commented on issue spikerj/spikersoft-issues#552

2026-07-14 03:03:29 +00:00

[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes

QA Team — new casualty + revised storage hypothesis 2026-07-14 ~03:25Z:

Seq was hit too. On dreamstream2, Seq's event store threw Transport endpoint is not connected (os error 107)…

spikerj commented on issue spikerj/spikersoft-issues#552

2026-07-14 03:01:34 +00:00

[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes

QA Team — major timeline update 2026-07-14 ~03:20Z, from dreamstream1's kernel journal (pulled by ops):

dreamstream1 crashed and rebooted tonight at 02:28:06Z — its journal begins at…

spikerj commented on issue spikerj/spikersoft-issues#532

2026-07-14 03:00:30 +00:00

[Infra][MinIO] Optional: replace sftp stack with MinIO SFTP mode (epic #413)

Consumer audit done (repo-side) — recommendation: retire, don't migrate.

Findings:

Zero code consumers: not one reference to sftp (path, host, or port) anywhere in spikersoft-backen…

spikerj commented on issue spikerj/spikersoft-issues#555

2026-07-14 02:59:09 +00:00

[Bug][Infra][Alerting] Node outages are invisible to ops alerting — no node-down alerts, kernel alerts arrive after recovery and carry no kernel message, cooldown hides escalation

QA Team — alert-tuning addendum 2026-07-14 ~03:15Z: the fleet kernel errors turn out to be PCIe AER corrected errors (see #552 for the full diagnosis). AER corrected errors can burst by…

spikerj closed issue spikerj/spikersoft-issues#550

2026-07-14 02:59:09 +00:00

[Lint][Angular] testing/transloco-test-setup.ts violates @nx/enforce-module-boundaries x3 — surfaced by first-ever root-project lint

spikerj commented on issue spikerj/spikersoft-issues#550

2026-07-14 02:59:08 +00:00

[Lint][Angular] testing/transloco-test-setup.ts violates @nx/enforce-module-boundaries x3 — surfaced by first-ever root-project lint

Fixed and CI-verified. Angular PR #191 (merged): the three i18n asset imports in testing/transloco-test-setup.ts are now the workspace's second documented boundary exemption (inline…

spikerj commented on issue spikerj/spikersoft-issues#552

2026-07-14 02:59:04 +00:00

[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes

QA Team — kernel-error signature identified 2026-07-14 ~03:15Z (kernel log pulled from a dreamstream node by ops):

The fleet 'kernel errors' are **PCIe AER corrected-error bursts on the NVMe…

spikerj commented on issue spikerj/spikersoft-issues#541

2026-07-14 02:57:46 +00:00

[Bug][Backend][DockerMonitor] All swarm node events published as 'docker.node.unknown' with empty hostname — ~21 junk RabbitMQ msgs/min for healthy nodes

QA Team — refinement/correction 2026-07-14 ~03:10Z on the SignalR errors noted earlier: Jaeger traces show the 02:27:48–49Z ForwardClusterEventToSignalR failures were **Redis PUBLISH…