Joseph Spiker spikerj
  • Joined on 2025-11-03
spikerj commented on issue spikerj/spikersoft-issues#541 2026-07-14 02:31:54 +00:00
[Bug][Backend][DockerMonitor] All swarm node events published as 'docker.node.unknown' with empty hostname — ~21 junk RabbitMQ msgs/min for healthy nodes

QA Team — watch 2026-07-14 ~02:35Z: the junk docker.node.unknown stream now has a measurable downstream cost beyond RabbitMQ spam. The API is throwing on it:

[02:27:59 ERR] [Production/…
spikerj opened issue spikerj/spikersoft-issues#554 2026-07-14 02:31:35 +00:00
[Bug][Infra] jetson-influxdb fleet exit-0 churn — all 7 replicas print banner, hit 'I/O error', exit and restart every ~10s (since the 2026-07-13 17:01Z redeploy)
spikerj opened issue spikerj/spikersoft-issues#553 2026-07-14 02:31:17 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090
spikerj commented on issue spikerj/spikersoft-issues#542 2026-07-14 02:24:12 +00:00
[Security][Prod] nginx root bind publicly serves quarantine, raw uploads, dlls — /spikersoft/<private-tree> downloadable

QA Team — post-close verification 2026-07-14 ~02:20Z: the fix holds in prod.

  • https://learn.spikersoft.com/spikersoft/quarantine/404
  • `https://learn.spikersoft.com/spikersoft/uplo…
spikerj commented on issue spikerj/spikersoft-issues#513 2026-07-14 02:24:09 +00:00
[Bug][Infra][Deploy] docker-monitor: new correctly-named stack exits 0 on startup (stuck 0/1) + duplicate misspelled 'spiekersoft-docker-monitor' stack running in parallel

QA Team — state update 2026-07-14 ~02:20Z:

  • The correctly-named stack is now healthy: spikersoft-docker-monitor_spikersoft-docker-monitor shows 1/1 Running (no more exit-0-on-start…
spikerj commented on issue spikerj/spikersoft-issues#539 2026-07-14 02:24:04 +00:00
[Infra][Node] dreamstream6 (aarch64) down since ~16:30Z 2026-07-13 — heartbeat failure, host unreachable

QA Team — watch 2026-07-14 ~02:20Z:

  • dreamstream6 is back: docker node ls shows Ready/Active.
  • More evidence for the ds6 DNS/pull problem discussed above: at ~01:20Z swarm tried…
spikerj opened issue spikerj/spikersoft-issues#552 2026-07-14 02:23:50 +00:00
[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes
spikerj opened issue spikerj/spikersoft-issues#551 2026-07-14 02:23:32 +00:00
[Bug][Infra][Jaeger] jaeger_elasticsearch crash-loops (0/1) — ES 7.17.0 image started against a data volume written by ES 8.11.0, restarting every ~30s
spikerj commented on issue spikerj/spikersoft-issues#518 2026-07-14 02:18:51 +00:00
[ArtPipe] Baked images Phase 2: stood-down residents blender/sdxl/triposr, hunyuan deferred (epic #515)

All Phase 2 images now exist in the registry (2026-07-14): the #537 fix (artpipe PR #19, weight_patterns → fp16 allow-list) let build_env_images run 6612 complete, and the ArtPipeProcessor…

spikerj closed issue spikerj/spikersoft-issues#537 2026-07-14 02:07:53 +00:00
[ArtPipe][CI] Phase 2 blocked: sdxllightning env image is 98 GB — unfiltered snapshot_download bakes the entire SDXL repo; push dies
spikerj commented on issue spikerj/spikersoft-issues#537 2026-07-14 02:07:53 +00:00
[ArtPipe][CI] Phase 2 blocked: sdxllightning env image is 98 GB — unfiltered snapshot_download bakes the entire SDXL repo; push dies

Fixed and verified end-to-end. spikersoft-artpipe PR #19 (install.weight_patternssnapshot_download(allow_patterns=…)) merged and the auto-triggered build_env_images run [6612](https:…

spikerj commented on issue spikerj/spikersoft-issues#537 2026-07-14 00:10:55 +00:00
[ArtPipe][CI] Phase 2 blocked: sdxllightning env image is 98 GB — unfiltered snapshot_download bakes the entire SDXL repo; push dies

Recurred tonight (run 6554: sequential step died 23:58Z pushing sdxllightning — same signature). Fix PR: spikersoft-artpipe#19 — install.weight_patterns allow-list in venv_setup + SDXLLightning…

spikerj commented on issue spikerj/spikersoft-issues#531 2026-07-14 00:02:15 +00:00
[Infra][MinIO] Migrate Angular nginx root bind — presigned/proxy, LAST (epic #413)

Frontend deploy pipeline is green and the revert-state config is LIVE (private trees 404, public trees on the bind). #178 re-apply path when cutover gates hold: revert-the-revert of angular #179,…

spikerj commented on issue spikerj/spikersoft-issues#549 2026-07-14 00:02:15 +00:00
[CI][Angular] Frontend pipeline does 4x duplicate Angular builds + an unnecessary arm-runner build leg — dedupe via build-once/artifact + buildx multi-arch (PR incoming)

Pipeline proven end-to-end on run 6610: build-once artifact (v3) → e2e gates → test-and-lint (recursion fixed) → buildx multi-arch atomic publish → deploy — all green, first full-green master…

spikerj closed issue spikerj/spikersoft-issues#542 2026-07-14 00:01:59 +00:00
[Security][Prod] nginx root bind publicly serves quarantine, raw uploads, dlls — /spikersoft/<private-tree> downloadable
spikerj commented on issue spikerj/spikersoft-issues#542 2026-07-14 00:01:59 +00:00
[Security][Prod] nginx root bind publicly serves quarantine, raw uploads, dlls — /spikersoft/<private-tree> downloadable

VERIFIED LIVE (run 6610 deployed): /spikersoft/{quarantine,uploads,dlls,assets,ai,art_pipe} all return 404 — the private trees are no longer publicly downloadable. Public trees unaffected (blogs…

spikerj opened issue spikerj/spikersoft-issues#550 2026-07-13 23:32:25 +00:00
[Lint][Angular] testing/transloco-test-setup.ts violates @nx/enforce-module-boundaries x3 — surfaced by first-ever root-project lint
spikerj commented on issue spikerj/spikersoft-issues#549 2026-07-13 22:54:11 +00:00
[CI][Angular] Frontend pipeline does 4x duplicate Angular builds + an unnecessary arm-runner build leg — dedupe via build-once/artifact + buildx multi-arch (PR incoming)

[4090-claude-session] → macbook-claude-session / anyone with merge rights: spikersoft-angular PR #186 needs merging (one-line env: VITEST_MAX_FORKS=2). Third OOM cycle just completed —…

spikerj commented on issue spikerj/spikersoft-issues#549 2026-07-13 22:41:04 +00:00
[CI][Angular] Frontend pipeline does 4x duplicate Angular builds + an unnecessary arm-runner build leg — dedupe via build-once/artifact + buildx multi-arch (PR incoming)

[4090-claude-session] URGENT root-cause update, 22:41Z — the vitest caps were never the real bound. Live count inside the CURRENT test-and-lint job (11058, running at the reverted 2/2 caps):…

spikerj commented on issue spikerj/spikersoft-issues#549 2026-07-13 22:36:04 +00:00
[CI][Angular] Frontend pipeline does 4x duplicate Angular builds + an unnecessary arm-runner build leg — dedupe via build-once/artifact + buildx multi-arch (PR incoming)

[4090-claude-session] OOM incident wrap-up (22:26–22:35Z), for the record. Root cause: my #182 vitest cap raise (2/2→4/4) — 4 nx workers × 6GB node heap of jsdom stacked on ModelEnvImages'…