spikerj commented on issue spikerj/spikersoft-issues#551

2026-07-14 06:16:55 +00:00

[Bug][Infra][Jaeger] jaeger_elasticsearch crash-loops (0/1) — ES 7.17.0 image started against a data volume written by ES 8.11.0, restarting every ~30s

Root cause is deeper than the crash loop, and the crash loop was hiding it.

The image pin is fixed — ES 8.11.0 (matching what wrote the data volume) is now 1/1, cluster green, stable through…

spikerj commented on issue spikerj/spikersoft-issues#556

2026-07-14 06:13:52 +00:00

[Bug][Backend][Infra] tiles.spikersoft.com has no backing service — API status/health canary fetches a terrain tile every few minutes and 404s 100% of the time

I need to rebuild this image from my windows desktop, and include the information and repo into the spikersoft infra repo.. put this on hold until I go this

spikerj closed issue spikerj/spikersoft-issues#558

2026-07-14 06:13:01 +00:00

[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

spikerj commented on issue spikerj/spikersoft-issues#558

2026-07-14 06:07:57 +00:00

[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

QA Team — symptom resolved 2026-07-14 ~06:10Z: remediation's disk-space incidents for SERVER:/ stopped firing (last occurrence #98; the detector fires every ~2 min while above threshold,…

spikerj opened issue spikerj/spikersoft-issues#559

2026-07-14 05:58:45 +00:00

[Bug][Backend][Quality] Quiz pages whose LLM output fails JSON parsing silently contribute 0 questions (WRN only) — quizzes ship smaller than requested

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 05:58:38 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — ✅ END-TO-END VERIFIED, 2026-07-14 05:58:03Z: quiz generation for book 6a55ad64… completed with 42 questions — Stage=Completed, user notified. First successful quiz since…

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 05:52:21 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — second integration gap found & patched during the 4090 verification, 2026-07-14 ~05:52Z:

The first re-triggered quiz on the 4090 failed with DirectoryNotFoundException (not…

spikerj commented on issue spikerj/spikersoft-issues#505

2026-07-14 05:45:33 +00:00

[Bug][Infra] MinIO single drive faulted under first sustained concurrent write load — writers aborted mid-stream, task left LB

QA Team — post-close root-cause record 2026-07-14 ~05:50Z, so the story is findable later: the drive faults this ticket documented were traced to bad RAM on SERVER, now isolated via GRUB…

spikerj commented on issue spikerj/spikersoft-issues#558

2026-07-14 05:45:29 +00:00

[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

QA Team — likely root cause identified 2026-07-14 ~05:50Z: MinIO's data dir had been temporarily relocated onto SERVER's root filesystem while the fusion card was suspect (bad RAM on…

spikerj commented on issue spikerj/spikersoft-issues#504

2026-07-14 05:44:41 +00:00

[Bug][Infra] MinIO doesn't recover after a transient /mnt/fusionio drive blip — stuck '0 drives provided', 'mc ready local' healthcheck misses it, needs manual restart

QA Team — post-close deploy verification 2026-07-14 ~05:45Z: the hardened healthcheck from this ticket went live at 05:43:41Z — the service healthcheck is now `mc ready local && mc alias set…

spikerj commented on issue spikerj/spikersoft-issues#515

2026-07-14 05:13:33 +00:00

[Epic][ArtPipe][Infra] Per-model baked Docker images — venvs + weights built at image build time, no /opt/art_pipe bind

Proposed endgame for this epic: split the worker from the model (Joey's architecture call, 2026-07-14)

The complaint that prompted this: *"making a change to anything in .NET never seemed…

spikerj opened issue spikerj/spikersoft-issues#558

2026-07-14 04:59:48 +00:00

[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 04:59:39 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — fix option 1 APPLIED (at spikerj's direction) 2026-07-14 ~04:58Z:

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 04:52:43 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — happy-path test FAILED and the true root cause is now pinned, 2026-07-14 ~04:55Z:

The test: book 6a55ad64… finished captions (182/182, 04:45:54Z — ImageDescription…

spikerj commented on issue spikerj/spikersoft-issues#519

2026-07-14 04:39:48 +00:00

[ArtPipe] Baked images Phase 3: prodstages image + live modeling-lane cutover (epic #515)

QA Team — user-facing impact confirmed 2026-07-14 ~04:40Z, escalating the earlier reject-loop finding:

A live Art Studio concept text→image generation was submitted at 04:37:26Z: the…

spikerj commented on issue spikerj/spikersoft-issues#519

2026-07-14 04:35:58 +00:00

[ArtPipe] Baked images Phase 3: prodstages image + live modeling-lane cutover (epic #515)

Blocked on the registry data path, not on code. Status 2026-07-14 ~04:35Z:

✅ Env image done: artpipe-model-env-prodstages:latest / :44daa91 pushed 04:05Z. All four stage venvs…

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 04:28:13 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — retry-exhaustion outcome observed 2026-07-14 ~04:30Z, completing the PR #272 verification:

After 3/3 retries, quiz generation for book 6a542e3f… was marked **Stage=Failed with…

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 04:23:48 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — live verification of PR #272, 2026-07-14 ~04:25Z (first natural repro: the 182-image book upload is captioning while quiz work arrived):

**✅ Fixed and verified — the silent…

spikerj commented on issue spikerj/spikersoft-issues#555

2026-07-14 04:14:41 +00:00

[Bug][Infra][Alerting] Node outages are invisible to ops alerting — no node-down alerts, kernel alerts arrive after recovery and carry no kernel message, cooldown hides escalation

QA Team — live deploy watch 2026-07-14 ~04:15Z: the new node-agent build (with JournalTailService — looks like this ticket's kernel-message-in-alert fix) is rolling out, but the **rollout…

spikerj opened issue spikerj/spikersoft-issues#557

2026-07-14 04:09:51 +00:00

[Bug][Backend] BookAnalysisService (ExtractChapters / AnalyzeBookWithLLM) silently ships degraded book data on GPU/model-load failure — same class as quiz-gen #553, different pipeline