Joseph Spiker spikerj
  • Joined on 2025-11-03
spikerj commented on issue spikerj/spikersoft-issues#504 2026-07-14 05:44:41 +00:00
[Bug][Infra] MinIO doesn't recover after a transient /mnt/fusionio drive blip — stuck '0 drives provided', 'mc ready local' healthcheck misses it, needs manual restart

QA Team — post-close deploy verification 2026-07-14 ~05:45Z: the hardened healthcheck from this ticket went live at 05:43:41Z — the service healthcheck is now `mc ready local && mc alias set…

spikerj commented on issue spikerj/spikersoft-issues#515 2026-07-14 05:13:33 +00:00
[Epic][ArtPipe][Infra] Per-model baked Docker images — venvs + weights built at image build time, no /opt/art_pipe bind

Proposed endgame for this epic: split the worker from the model (Joey's architecture call, 2026-07-14)

The complaint that prompted this: *"making a change to anything in .NET never seemed…

spikerj opened issue spikerj/spikersoft-issues#558 2026-07-14 04:59:48 +00:00
[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path
spikerj commented on issue spikerj/spikersoft-issues#553 2026-07-14 04:59:39 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — fix option 1 APPLIED (at spikerj's direction) 2026-07-14 ~04:58Z:

docker service update \
  --constraint-rm 'node.hostname == SERVER' \
  --constraint-add 'node.hostname ==…
spikerj commented on issue spikerj/spikersoft-issues#553 2026-07-14 04:52:43 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — happy-path test FAILED and the true root cause is now pinned, 2026-07-14 ~04:55Z:

The test: book 6a55ad64… finished captions (182/182, 04:45:54Z — ImageDescription…

spikerj commented on issue spikerj/spikersoft-issues#519 2026-07-14 04:39:48 +00:00
[ArtPipe] Baked images Phase 3: prodstages image + live modeling-lane cutover (epic #515)

QA Team — user-facing impact confirmed 2026-07-14 ~04:40Z, escalating the earlier reject-loop finding:

A live Art Studio concept text→image generation was submitted at 04:37:26Z: the…

spikerj commented on issue spikerj/spikersoft-issues#519 2026-07-14 04:35:58 +00:00
[ArtPipe] Baked images Phase 3: prodstages image + live modeling-lane cutover (epic #515)

Blocked on the registry data path, not on code. Status 2026-07-14 ~04:35Z:

Env image done: artpipe-model-env-prodstages:latest / :44daa91 pushed 04:05Z. All four stage venvs…

spikerj commented on issue spikerj/spikersoft-issues#553 2026-07-14 04:28:13 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — retry-exhaustion outcome observed 2026-07-14 ~04:30Z, completing the PR #272 verification:

After 3/3 retries, quiz generation for book 6a542e3f… was marked **Stage=Failed with…

spikerj commented on issue spikerj/spikersoft-issues#553 2026-07-14 04:23:48 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — live verification of PR #272, 2026-07-14 ~04:25Z (first natural repro: the 182-image book upload is captioning while quiz work arrived):

** Fixed and verified — the silent…

spikerj commented on issue spikerj/spikersoft-issues#555 2026-07-14 04:14:41 +00:00
[Bug][Infra][Alerting] Node outages are invisible to ops alerting — no node-down alerts, kernel alerts arrive after recovery and carry no kernel message, cooldown hides escalation

QA Team — live deploy watch 2026-07-14 ~04:15Z: the new node-agent build (with JournalTailService — looks like this ticket's kernel-message-in-alert fix) is rolling out, but the **rollout…

spikerj opened issue spikerj/spikersoft-issues#557 2026-07-14 04:09:51 +00:00
[Bug][Backend] BookAnalysisService (ExtractChapters / AnalyzeBookWithLLM) silently ships degraded book data on GPU/model-load failure — same class as quiz-gen #553, different pipeline
spikerj commented on issue spikerj/spikersoft-issues#538 2026-07-14 04:02:45 +00:00
[Infra][Network] LAN traffic to *.spikersoft.com NAT-hairpins through the UDM — local DNS records + direct Gitea→MinIO path

Confirmed live tonight (2026-07-14) during a large artpipe env-image push: the registry write path pays this tax twice, because since #498 Gitea does not store package blobs — it relays them to…

spikerj commented on issue spikerj/spikersoft-issues#553 2026-07-14 03:30:42 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

Follow-up finding worth its own fix — quiz-gen's lease is under-declared too (same "books lied to" class as this ticket's root cause).

Surfaced while implementing PR #272: quiz-generation…

spikerj commented on issue spikerj/spikersoft-issues#553 2026-07-14 03:28:30 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

Silent-degradation fix up: spikersoft-backend PR #272

spikerj/spikersoft-backend#272 — the follow-up flagged in my root-cause comment (option (a), plus the…

spikerj commented on issue spikerj/spikersoft-issues#519 2026-07-14 03:27:30 +00:00
[ArtPipe] Baked images Phase 3: prodstages image + live modeling-lane cutover (epic #515)

QA Team — active blocker found 2026-07-14 ~03:35Z: spikersoft-artpipe-modeling_artpipe-modeling is in a hard reject-loop and the modeling lane is down.

  • Every ~5s since the **18:57Z Jul…
spikerj commented on issue spikerj/spikersoft-issues#538 2026-07-14 03:22:05 +00:00
[Infra][Network] LAN traffic to *.spikersoft.com NAT-hairpins through the UDM — local DNS records + direct Gitea→MinIO path

QA Team — stronger evidence 2026-07-14 ~03:30Z that this is more than LAN-hairpin fragility: during dreamstream1's crash/reboot window tonight (down ~02:27:50Z, boot 02:28:06Z — see #552),…

spikerj commented on issue spikerj/spikersoft-issues#546 2026-07-14 03:21:58 +00:00
[Security][Backend] OpenBao Phase 2: SpikerSoft.Secrets host-builder seam — service creds from Bao (epic #543)

Phase 2 seam is up for review: spikersoft-backend PR #270. New SpikerSoft.Secrets project (ISecretSource + VaultSecretSource:…

spikerj commented on issue spikerj/spikersoft-issues#510 2026-07-14 03:21:36 +00:00
[Exploration][Infra] Resilience to SERVER node flaps — restart-policy windows, auto-reconcile sweep, de-SPOF the SERVER-pinned stateful tier

Infra-wide swarm-resilience audit (#510) — stack-file sweep, 2026-07-13

Sweep of every docker-stack*.yml in spikersoft-infrastructure @ master 4192b90 (67 files parsed), classified against…

spikerj opened issue spikerj/spikersoft-issues#556 2026-07-14 03:17:39 +00:00
[Bug][Backend][Infra] tiles.spikersoft.com has no backing service — API status/health canary fetches a terrain tile every few minutes and 404s 100% of the time
spikerj closed issue spikerj/spikersoft-issues#548 2026-07-14 03:04:14 +00:00
[CI][Backend] Multi-arch create_manifest amends MUTABLE :amd64/:arm64v8 tags — out-of-order arch pushes can stitch a mixed-commit :latest (all backend multi-arch workflows)