Joseph Spiker spikerj
  • Joined on 2025-11-03
spikerj commented on issue spikerj/spikersoft-issues#557 2026-07-14 06:30:02 +00:00
[Bug][Backend] BookAnalysisService (ExtractChapters / AnalyzeBookWithLLM) silently ships degraded book data on GPU/model-load failure — same class as quiz-gen #553, different pipeline

Diagnosis + fix in PR #275 (with #559 — same mechanism, one PR).

Root cause — three separate holes, all in BookAnalysisService:

  1. **ExtractChaptersWithLLM had a blanket `catch…
spikerj commented on issue spikerj/spikersoft-issues#554 2026-07-14 06:29:56 +00:00
[Bug][Infra] jetson-influxdb fleet exit-0 churn — all 7 replicas print banner, hit 'I/O error', exit and restart every ~10s (since the 2026-07-13 17:01Z redeploy)

Diagnosis: host-side jtop dependency, not config and not credentials

The reporter reads each board's stats from the host's jtop daemon over a bind-mounted unix socket (/run/jtop.sock).…

spikerj commented on issue spikerj/spikersoft-issues#559 2026-07-14 06:29:34 +00:00
[Bug][Backend][Quality] Quiz pages whose LLM output fails JSON parsing silently contribute 0 questions (WRN only) — quizzes ship smaller than requested

Diagnosis + fix in PR #275 (with #557 — same mechanism, one PR).

Root cause. QuizGenerationService.ParseGeneratedQuestions returned an empty list whenever the model's JSON was…

spikerj closed issue spikerj/spikersoft-issues#484 2026-07-14 06:23:15 +00:00
[Bug][Infra] API /healthz returns 503 — TileServerGL check failing (tiles.spikersoft.com 404s, no tile-server deployed)
spikerj commented on issue spikerj/spikersoft-issues#484 2026-07-14 06:23:15 +00:00
[Bug][Infra] API /healthz returns 503 — TileServerGL check failing (tiles.spikersoft.com 404s, no tile-server deployed)

Verified against the live endpoint before touching anything — this is already fixed and deployed.

/healthz currently returns HTTP 200 with status Degraded (12 checks Healthy,…

spikerj commented on issue spikerj/spikersoft-issues#556 2026-07-14 06:21:29 +00:00
[Bug][Backend][Infra] tiles.spikersoft.com has no backing service — API status/health canary fetches a terrain tile every few minutes and 404s 100% of the time

Diagnosis — your option 1 is correct

The tile service was never deployed / was decommissioned. Verified read-only against the swarm: a scan of all **94 services matches nothing on…

spikerj commented on issue spikerj/spikersoft-issues#484 2026-07-14 06:20:59 +00:00
[Bug][Infra] API /healthz returns 503 — TileServerGL check failing (tiles.spikersoft.com 404s, no tile-server deployed)

Diagnosis

Confirmed, and the headline symptom is already resolved.

The 503 is gone. /healthz currently returns HTTP 200 / "status": "Degraded" — 12 checks Healthy, TileServerG…

spikerj opened issue spikerj/spikersoft-issues#560 2026-07-14 06:20:40 +00:00
[Bug][Infra][GPU] gpu-coordinator arbitrates by declared bookings, not measured VRAM — over-budget tenants invisibly starve grantees (split from closed #553)
spikerj closed issue spikerj/spikersoft-issues#539 2026-07-14 06:19:38 +00:00
[Infra][Node] dreamstream6 (aarch64) down since ~16:30Z 2026-07-13 — heartbeat failure, host unreachable
spikerj commented on issue spikerj/spikersoft-issues#558 2026-07-14 06:18:42 +00:00
[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

Acute condition resolved: SERVER root is now 32% used, 286 GB free (was 94% and climbing, alerting every ~2 min).

Reclaimed by moving the MinIO data root onto the FusionIO card and deleting…

spikerj closed issue spikerj/spikersoft-issues#553 2026-07-14 06:18:41 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090
spikerj commented on issue spikerj/spikersoft-issues#553 2026-07-14 06:18:41 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

Both halves are fixed and verified.

The VRAM lease under-declaration — image-description declared a 16 GB lease while actually holding 18.2 GiB, so the coordinator believed there was room to…

spikerj commented on issue spikerj/spikersoft-issues#551 2026-07-14 06:16:55 +00:00
[Bug][Infra][Jaeger] jaeger_elasticsearch crash-loops (0/1) — ES 7.17.0 image started against a data volume written by ES 8.11.0, restarting every ~30s

Root cause is deeper than the crash loop, and the crash loop was hiding it.

The image pin is fixed — ES 8.11.0 (matching what wrote the data volume) is now 1/1, cluster green, stable through…

spikerj commented on issue spikerj/spikersoft-issues#556 2026-07-14 06:13:52 +00:00
[Bug][Backend][Infra] tiles.spikersoft.com has no backing service — API status/health canary fetches a terrain tile every few minutes and 404s 100% of the time

I need to rebuild this image from my windows desktop, and include the information and repo into the spikersoft infra repo.. put this on hold until I go this

spikerj closed issue spikerj/spikersoft-issues#558 2026-07-14 06:13:01 +00:00
[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path
spikerj commented on issue spikerj/spikersoft-issues#558 2026-07-14 06:07:57 +00:00
[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

QA Team — symptom resolved 2026-07-14 ~06:10Z: remediation's disk-space incidents for SERVER:/ stopped firing (last occurrence #98; the detector fires every ~2 min while above threshold,…

spikerj opened issue spikerj/spikersoft-issues#559 2026-07-14 05:58:45 +00:00
[Bug][Backend][Quality] Quiz pages whose LLM output fails JSON parsing silently contribute 0 questions (WRN only) — quizzes ship smaller than requested
spikerj commented on issue spikerj/spikersoft-issues#553 2026-07-14 05:58:38 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team END-TO-END VERIFIED, 2026-07-14 05:58:03Z: quiz generation for book 6a55ad64… completed with 42 questionsStage=Completed, user notified. First successful quiz since…

spikerj commented on issue spikerj/spikersoft-issues#553 2026-07-14 05:52:21 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

QA Team — second integration gap found & patched during the 4090 verification, 2026-07-14 ~05:52Z:

The first re-triggered quiz on the 4090 failed with DirectoryNotFoundException (not…

spikerj commented on issue spikerj/spikersoft-issues#505 2026-07-14 05:45:33 +00:00
[Bug][Infra] MinIO single drive faulted under first sustained concurrent write load — writers aborted mid-stream, task left LB

QA Team — post-close root-cause record 2026-07-14 ~05:50Z, so the story is findable later: the drive faults this ticket documented were traced to bad RAM on SERVER, now isolated via GRUB…