Joseph Spiker

spikerj commented on issue spikerj/spikersoft-issues#513

2026-07-14 02:55:41 +00:00

[Bug][Infra][Deploy] docker-monitor: new correctly-named stack exits 0 on startup (stuck 0/1) + duplicate misspelled 'spiekersoft-docker-monitor' stack running in parallel

Problem 1 (exit-0) is fixed on master — Program.cs now registers IDockerClient with the annotation explaining exactly this failure: ServiceAutoReconciler (#510-B) resolved `IDockerClient…

spikerj commented on issue spikerj/spikersoft-issues#479

2026-07-14 02:54:59 +00:00

[Bug][Infra] sonarqube_backups stuck in reject-loop — no Sonar backups running (missing placement constraint)

Code-side this is fixed on master (198ad66 — backups pinned node.hostname == SERVER like its siblings, plus the $$ escaping so pg_dump's vars survive the compose parser). What remains…

spikerj opened issue spikerj/spikersoft-issues#555

2026-07-14 02:53:00 +00:00

[Bug][Infra][Alerting] Node outages are invisible to ops alerting — no node-down alerts, kernel alerts arrive after recovery and carry no kernel message, cooldown hides escalation

spikerj commented on issue spikerj/spikersoft-issues#552

2026-07-14 02:49:03 +00:00

[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes

QA Team — corruption verdict 2026-07-14 ~03:00Z: durable, not transient. The missing chunk number 0 for toast value 131293 in pg_toast_2619 error re-fired at 02:44:50Z — well after…

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 02:47:22 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

Root cause (confirmed in code + config)

Three numbers that don't add up:

Declared: image-description leases GPU_VRAM_REQUIREMENT_MB=16000 (its stack file)
Real: Qwen3-VL-8B fp16…

spikerj commented on issue spikerj/spikersoft-issues#552

2026-07-14 02:44:48 +00:00

[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes

QA Team — predictive heads-up 2026-07-14 ~02:55Z: the kernel-error class that preceded dreamstream4's panic is actively accumulating on other nodes. system-remediation in the last 4h:

-…

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 02:44:40 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — evidence update + topology correction 2026-07-14 ~02:55Z:

Correction: the contended GPU is the 4090 node's card (the #500 "4090 lane" is in service), not a GPU on SERVER —…

spikerj commented on issue spikerj/spikersoft-issues#554

2026-07-14 02:44:06 +00:00

[Bug][Infra] jetson-influxdb fleet exit-0 churn — all 7 replicas print banner, hit 'I/O error', exit and restart every ~10s (since the 2026-07-13 17:01Z redeploy)

Registry/source findings from repo-side investigation (no host access from this session):

1. The image is unpullable — its namespace no longer exists. The stack pins `git.spikersoft.com/jspik…

spikerj commented on issue spikerj/spikersoft-issues#552

2026-07-14 02:42:23 +00:00

[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes

Cross-linking evidence from tonight: dreamstream4 being Down is very likely the cause of the GlusterFS /mnt/infrastructure distress Joey observed ~02:14-02:25Z (and possibly of the brief…

spikerj commented on issue spikerj/spikersoft-issues#552

2026-07-14 02:38:37 +00:00

[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes

QA Team — de-escalation update 2026-07-14 ~02:50Z (follow-up to the URGENT note above):

The postgres transport errors stopped at 02:29:51Z and have not recurred; timing tracks…

spikerj commented on issue spikerj/spikersoft-issues#539

2026-07-14 02:35:26 +00:00

[Infra][Node] dreamstream6 (aarch64) down since ~16:30Z 2026-07-13 — heartbeat failure, host unreachable

QA Team — watch 2026-07-14 ~02:45Z: gitea-runners_arm_v8_2_act_runner is still crash-looping on dreamstream6 (task: non-zero exit (1) every ~20s, currently 0/1) — roughly 6 hours…

spikerj commented on issue spikerj/spikersoft-issues#554

2026-07-14 02:35:23 +00:00

[Bug][Infra] jetson-influxdb fleet exit-0 churn — all 7 replicas print banner, hit 'I/O error', exit and restart every ~10s (since the 2026-07-13 17:01Z redeploy)

QA Team — root cause CONFIRMED 2026-07-14 ~02:45Z: the 'I/O error' was the reporters failing to reach InfluxDB itself. The main influxDB_influxdb service is pinned to dreamstream4 and…

spikerj commented on issue spikerj/spikersoft-issues#552

2026-07-14 02:35:16 +00:00

[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes

QA Team — URGENT escalation 2026-07-14 ~02:45Z. The data-integrity question from this ticket is now CONFIRMED as an active failure.

1. Node status: dreamstream4 rejoined ~02:40Z (all…

spikerj commented on issue spikerj/spikersoft-issues#538

2026-07-14 02:31:59 +00:00

[Infra][Network] LAN traffic to *.spikersoft.com NAT-hairpins through the UDM — local DNS records + direct Gitea→MinIO path

QA Team — supporting datapoint 2026-07-14 ~02:32Z: from the swarm manager (LAN), https://git.spikersoft.com hard-refused connections (`dial tcp 204.197.150.99:443: connect: connection…

spikerj commented on issue spikerj/spikersoft-issues#541

2026-07-14 02:31:54 +00:00

[Bug][Backend][DockerMonitor] All swarm node events published as 'docker.node.unknown' with empty hostname — ~21 junk RabbitMQ msgs/min for healthy nodes

QA Team — watch 2026-07-14 ~02:35Z: the junk docker.node.unknown stream now has a measurable downstream cost beyond RabbitMQ spam. The API is throwing on it:

spikerj opened issue spikerj/spikersoft-issues#554

2026-07-14 02:31:35 +00:00

[Bug][Infra] jetson-influxdb fleet exit-0 churn — all 7 replicas print banner, hit 'I/O error', exit and restart every ~10s (since the 2026-07-13 17:01Z redeploy)

spikerj opened issue spikerj/spikersoft-issues#553

2026-07-14 02:31:17 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

spikerj commented on issue spikerj/spikersoft-issues#542

2026-07-14 02:24:12 +00:00

[Security][Prod] nginx root bind publicly serves quarantine, raw uploads, dlls — /spikersoft/<private-tree> downloadable

QA Team — post-close verification 2026-07-14 ~02:20Z: the fix holds in prod.

https://learn.spikersoft.com/spikersoft/quarantine/ → 404
`https://learn.spikersoft.com/spikersoft/uplo…

spikerj commented on issue spikerj/spikersoft-issues#513

2026-07-14 02:24:09 +00:00

[Bug][Infra][Deploy] docker-monitor: new correctly-named stack exits 0 on startup (stuck 0/1) + duplicate misspelled 'spiekersoft-docker-monitor' stack running in parallel

QA Team — state update 2026-07-14 ~02:20Z:

The correctly-named stack is now healthy: spikersoft-docker-monitor_spikersoft-docker-monitor shows 1/1 Running (no more exit-0-on-start…

spikerj commented on issue spikerj/spikersoft-issues#539

2026-07-14 02:24:04 +00:00

[Infra][Node] dreamstream6 (aarch64) down since ~16:30Z 2026-07-13 — heartbeat failure, host unreachable

QA Team — watch 2026-07-14 ~02:20Z:

dreamstream6 is back: docker node ls shows Ready/Active.
More evidence for the ds6 DNS/pull problem discussed above: at ~01:20Z swarm tried…