spikerj opened issue spikerj/spikersoft-issues#560

2026-07-14 06:20:40 +00:00

[Bug][Infra][GPU] gpu-coordinator arbitrates by declared bookings, not measured VRAM — over-budget tenants invisibly starve grantees (split from closed #553)

spikerj closed issue spikerj/spikersoft-issues#539

2026-07-14 06:19:38 +00:00

[Infra][Node] dreamstream6 (aarch64) down since ~16:30Z 2026-07-13 — heartbeat failure, host unreachable

spikerj commented on issue spikerj/spikersoft-issues#558

2026-07-14 06:18:42 +00:00

[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

Acute condition resolved: SERVER root is now 32% used, 286 GB free (was 94% and climbing, alerting every ~2 min).

Reclaimed by moving the MinIO data root onto the FusionIO card and deleting…

spikerj closed issue spikerj/spikersoft-issues#553

2026-07-14 06:18:41 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 06:18:41 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

Both halves are fixed and verified.

The VRAM lease under-declaration — image-description declared a 16 GB lease while actually holding 18.2 GiB, so the coordinator believed there was room to…

spikerj commented on issue spikerj/spikersoft-issues#551

2026-07-14 06:16:55 +00:00

[Bug][Infra][Jaeger] jaeger_elasticsearch crash-loops (0/1) — ES 7.17.0 image started against a data volume written by ES 8.11.0, restarting every ~30s

Root cause is deeper than the crash loop, and the crash loop was hiding it.

The image pin is fixed — ES 8.11.0 (matching what wrote the data volume) is now 1/1, cluster green, stable through…

spikerj commented on issue spikerj/spikersoft-issues#556

2026-07-14 06:13:52 +00:00

[Bug][Backend][Infra] tiles.spikersoft.com has no backing service — API status/health canary fetches a terrain tile every few minutes and 404s 100% of the time

I need to rebuild this image from my windows desktop, and include the information and repo into the spikersoft infra repo.. put this on hold until I go this

spikerj closed issue spikerj/spikersoft-issues#558

2026-07-14 06:13:01 +00:00

[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

spikerj commented on issue spikerj/spikersoft-issues#558

2026-07-14 06:07:57 +00:00

[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

QA Team — symptom resolved 2026-07-14 ~06:10Z: remediation's disk-space incidents for SERVER:/ stopped firing (last occurrence #98; the detector fires every ~2 min while above threshold,…

spikerj opened issue spikerj/spikersoft-issues#559

2026-07-14 05:58:45 +00:00

[Bug][Backend][Quality] Quiz pages whose LLM output fails JSON parsing silently contribute 0 questions (WRN only) — quizzes ship smaller than requested

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 05:58:38 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — ✅ END-TO-END VERIFIED, 2026-07-14 05:58:03Z: quiz generation for book 6a55ad64… completed with 42 questions — Stage=Completed, user notified. First successful quiz since…

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 05:52:21 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — second integration gap found & patched during the 4090 verification, 2026-07-14 ~05:52Z:

The first re-triggered quiz on the 4090 failed with DirectoryNotFoundException (not…

spikerj commented on issue spikerj/spikersoft-issues#505

2026-07-14 05:45:33 +00:00

[Bug][Infra] MinIO single drive faulted under first sustained concurrent write load — writers aborted mid-stream, task left LB

QA Team — post-close root-cause record 2026-07-14 ~05:50Z, so the story is findable later: the drive faults this ticket documented were traced to bad RAM on SERVER, now isolated via GRUB…

spikerj commented on issue spikerj/spikersoft-issues#558

2026-07-14 05:45:29 +00:00

[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

QA Team — likely root cause identified 2026-07-14 ~05:50Z: MinIO's data dir had been temporarily relocated onto SERVER's root filesystem while the fusion card was suspect (bad RAM on…

spikerj commented on issue spikerj/spikersoft-issues#504

2026-07-14 05:44:41 +00:00

[Bug][Infra] MinIO doesn't recover after a transient /mnt/fusionio drive blip — stuck '0 drives provided', 'mc ready local' healthcheck misses it, needs manual restart

QA Team — post-close deploy verification 2026-07-14 ~05:45Z: the hardened healthcheck from this ticket went live at 05:43:41Z — the service healthcheck is now `mc ready local && mc alias set…

spikerj commented on issue spikerj/spikersoft-issues#515

2026-07-14 05:13:33 +00:00

[Epic][ArtPipe][Infra] Per-model baked Docker images — venvs + weights built at image build time, no /opt/art_pipe bind

Proposed endgame for this epic: split the worker from the model (Joey's architecture call, 2026-07-14)

The complaint that prompted this: *"making a change to anything in .NET never seemed…

spikerj opened issue spikerj/spikersoft-issues#558

2026-07-14 04:59:48 +00:00

[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 04:59:39 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — fix option 1 APPLIED (at spikerj's direction) 2026-07-14 ~04:58Z:

docker service update \
  --constraint-rm 'node.hostname == SERVER' \
  --constraint-add 'node.hostname ==…

spikerj commented on issue spikerj/spikersoft-issues#553

2026-07-14 04:52:43 +00:00

[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — happy-path test FAILED and the true root cause is now pinned, 2026-07-14 ~04:55Z:

The test: book 6a55ad64… finished captions (182/182, 04:45:54Z — ImageDescription…

spikerj commented on issue spikerj/spikersoft-issues#519

2026-07-14 04:39:48 +00:00

[ArtPipe] Baked images Phase 3: prodstages image + live modeling-lane cutover (epic #515)

QA Team — user-facing impact confirmed 2026-07-14 ~04:40Z, escalating the earlier reject-loop finding:

A live Art Studio concept text→image generation was submitted at 04:37:26Z: the…