Joseph Spiker spikerj
  • Joined on 2025-11-03
spikerj commented on issue spikerj/spikersoft-issues#554 2026-07-14 02:35:23 +00:00
[Bug][Infra] jetson-influxdb fleet exit-0 churn — all 7 replicas print banner, hit 'I/O error', exit and restart every ~10s (since the 2026-07-13 17:01Z redeploy)

QA Team — root cause CONFIRMED 2026-07-14 ~02:45Z: the 'I/O error' was the reporters failing to reach InfluxDB itself. The main influxDB_influxdb service is pinned to dreamstream4 and…

spikerj commented on issue spikerj/spikersoft-issues#552 2026-07-14 02:35:16 +00:00
[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes

QA Team — URGENT escalation 2026-07-14 ~02:45Z. The data-integrity question from this ticket is now CONFIRMED as an active failure.

1. Node status: dreamstream4 rejoined ~02:40Z (all…

spikerj commented on issue spikerj/spikersoft-issues#538 2026-07-14 02:31:59 +00:00
[Infra][Network] LAN traffic to *.spikersoft.com NAT-hairpins through the UDM — local DNS records + direct Gitea→MinIO path

QA Team — supporting datapoint 2026-07-14 ~02:32Z: from the swarm manager (LAN), https://git.spikersoft.com hard-refused connections (`dial tcp 204.197.150.99:443: connect: connection…

spikerj commented on issue spikerj/spikersoft-issues#541 2026-07-14 02:31:54 +00:00
[Bug][Backend][DockerMonitor] All swarm node events published as 'docker.node.unknown' with empty hostname — ~21 junk RabbitMQ msgs/min for healthy nodes

QA Team — watch 2026-07-14 ~02:35Z: the junk docker.node.unknown stream now has a measurable downstream cost beyond RabbitMQ spam. The API is throwing on it:

[02:27:59 ERR] [Production/…
spikerj opened issue spikerj/spikersoft-issues#554 2026-07-14 02:31:35 +00:00
[Bug][Infra] jetson-influxdb fleet exit-0 churn — all 7 replicas print banner, hit 'I/O error', exit and restart every ~10s (since the 2026-07-13 17:01Z redeploy)
spikerj opened issue spikerj/spikersoft-issues#553 2026-07-14 02:31:17 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090
spikerj commented on issue spikerj/spikersoft-issues#542 2026-07-14 02:24:12 +00:00
[Security][Prod] nginx root bind publicly serves quarantine, raw uploads, dlls — /spikersoft/<private-tree> downloadable

QA Team — post-close verification 2026-07-14 ~02:20Z: the fix holds in prod.

  • https://learn.spikersoft.com/spikersoft/quarantine/404
  • `https://learn.spikersoft.com/spikersoft/uplo…
spikerj commented on issue spikerj/spikersoft-issues#513 2026-07-14 02:24:09 +00:00
[Bug][Infra][Deploy] docker-monitor: new correctly-named stack exits 0 on startup (stuck 0/1) + duplicate misspelled 'spiekersoft-docker-monitor' stack running in parallel

QA Team — state update 2026-07-14 ~02:20Z:

  • The correctly-named stack is now healthy: spikersoft-docker-monitor_spikersoft-docker-monitor shows 1/1 Running (no more exit-0-on-start…
spikerj commented on issue spikerj/spikersoft-issues#539 2026-07-14 02:24:04 +00:00
[Infra][Node] dreamstream6 (aarch64) down since ~16:30Z 2026-07-13 — heartbeat failure, host unreachable

QA Team — watch 2026-07-14 ~02:20Z:

  • dreamstream6 is back: docker node ls shows Ready/Active.
  • More evidence for the ds6 DNS/pull problem discussed above: at ~01:20Z swarm tried…
spikerj opened issue spikerj/spikersoft-issues#552 2026-07-14 02:23:50 +00:00
[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes
spikerj opened issue spikerj/spikersoft-issues#551 2026-07-14 02:23:32 +00:00
[Bug][Infra][Jaeger] jaeger_elasticsearch crash-loops (0/1) — ES 7.17.0 image started against a data volume written by ES 8.11.0, restarting every ~30s
spikerj commented on issue spikerj/spikersoft-issues#518 2026-07-14 02:18:51 +00:00
[ArtPipe] Baked images Phase 2: stood-down residents blender/sdxl/triposr, hunyuan deferred (epic #515)

All Phase 2 images now exist in the registry (2026-07-14): the #537 fix (artpipe PR #19, weight_patterns → fp16 allow-list) let build_env_images run 6612 complete, and the ArtPipeProcessor…

spikerj closed issue spikerj/spikersoft-issues#537 2026-07-14 02:07:53 +00:00
[ArtPipe][CI] Phase 2 blocked: sdxllightning env image is 98 GB — unfiltered snapshot_download bakes the entire SDXL repo; push dies
spikerj commented on issue spikerj/spikersoft-issues#537 2026-07-14 02:07:53 +00:00
[ArtPipe][CI] Phase 2 blocked: sdxllightning env image is 98 GB — unfiltered snapshot_download bakes the entire SDXL repo; push dies

Fixed and verified end-to-end. spikersoft-artpipe PR #19 (install.weight_patternssnapshot_download(allow_patterns=…)) merged and the auto-triggered build_env_images run [6612](https:…

spikerj commented on issue spikerj/spikersoft-issues#537 2026-07-14 00:10:55 +00:00
[ArtPipe][CI] Phase 2 blocked: sdxllightning env image is 98 GB — unfiltered snapshot_download bakes the entire SDXL repo; push dies

Recurred tonight (run 6554: sequential step died 23:58Z pushing sdxllightning — same signature). Fix PR: spikersoft-artpipe#19 — install.weight_patterns allow-list in venv_setup + SDXLLightning…

spikerj commented on issue spikerj/spikersoft-issues#531 2026-07-14 00:02:15 +00:00
[Infra][MinIO] Migrate Angular nginx root bind — presigned/proxy, LAST (epic #413)

Frontend deploy pipeline is green and the revert-state config is LIVE (private trees 404, public trees on the bind). #178 re-apply path when cutover gates hold: revert-the-revert of angular #179,…

spikerj commented on issue spikerj/spikersoft-issues#549 2026-07-14 00:02:15 +00:00
[CI][Angular] Frontend pipeline does 4x duplicate Angular builds + an unnecessary arm-runner build leg — dedupe via build-once/artifact + buildx multi-arch (PR incoming)

Pipeline proven end-to-end on run 6610: build-once artifact (v3) → e2e gates → test-and-lint (recursion fixed) → buildx multi-arch atomic publish → deploy — all green, first full-green master…

spikerj commented on issue spikerj/spikersoft-issues#542 2026-07-14 00:01:59 +00:00
[Security][Prod] nginx root bind publicly serves quarantine, raw uploads, dlls — /spikersoft/<private-tree> downloadable

VERIFIED LIVE (run 6610 deployed): /spikersoft/{quarantine,uploads,dlls,assets,ai,art_pipe} all return 404 — the private trees are no longer publicly downloadable. Public trees unaffected (blogs…

spikerj closed issue spikerj/spikersoft-issues#542 2026-07-14 00:01:59 +00:00
[Security][Prod] nginx root bind publicly serves quarantine, raw uploads, dlls — /spikersoft/<private-tree> downloadable
spikerj opened issue spikerj/spikersoft-issues#550 2026-07-13 23:32:25 +00:00
[Lint][Angular] testing/transloco-test-setup.ts violates @nx/enforce-module-boundaries x3 — surfaced by first-ever root-project lint