spikerj commented on issue spikerj/spikersoft-issues#562 2026-07-14 09:40:54 +00:00
[Bug][Infra][Deploy] Dreamstream nodes reject fresh tasks with 'No such image' for 1-3 minutes after (re)scheduling — 4th distinct service tonight; pulls likely fighting the UDM hairpin

QA Team — diagnosis sharpened 2026-07-14 ~09:45Z, and severity is worse than filed:

  • The coderunner occurrence is not a 1-3 min transient: replica .4 has now been bouncing between…
spikerj commented on issue spikerj/spikersoft-issues#543 2026-07-14 09:37:34 +00:00
[Epic][Security][Infra] Central secrets management — OpenBao vault + PKI, retire scattered passwords

QA Team — inventory input for this epic, 2026-07-14: a full fact-checking audit of every spikersoft-backend project (README rebuild, 100+ agents reading every csproj/appsettings/Program.cs)…

spikerj opened issue spikerj/spikersoft-issues#562 2026-07-14 09:36:43 +00:00
[Bug][Infra][Deploy] Dreamstream nodes reject fresh tasks with 'No such image' for 1-3 minutes after (re)scheduling — 4th distinct service tonight; pulls likely fighting the UDM hairpin
spikerj opened issue spikerj/spikersoft-issues#561 2026-07-14 06:34:47 +00:00
[Bug][Infra][Data] Keycloak postgres pg_statistic toast corruption persists after #552 closed — ClearExpiredEvents/session purge failing every 15-min cycle since 02:29Z, repair never run
spikerj commented on issue spikerj/spikersoft-issues#539 2026-07-14 06:32:27 +00:00
[Infra][Node] dreamstream6 (aarch64) down since ~16:30Z 2026-07-13 — heartbeat failure, host unreachable

Recovered. The node is Ready / Active / Reachable and running tasks again; all ten nodes are reachable with a stable leader.

Closing on observed state, with the same caveat I left on #552:…

spikerj closed issue spikerj/spikersoft-issues#552 2026-07-14 06:32:27 +00:00
[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes
spikerj commented on issue spikerj/spikersoft-issues#552 2026-07-14 06:32:26 +00:00
[Bug][Infra][Node] dreamstream4 (manager) Down/Unreachable — keycloak postgres forced failover: 'No such image' reject-loop on ds6 (#511 signature), stale 2/1 task, DB changed nodes

Recovered. The node is Ready / Active / Reachable, it is carrying tasks again, and the whole manager set is reachable with a stable leader.

The concrete casualty in this ticket is also…

spikerj commented on issue spikerj/spikersoft-issues#483 2026-07-14 06:30:36 +00:00
[Bug][Infra] InfluxDB rejecting a client every ~10s ("authorization not found") — stale/missing token on a poller

Found it: it's Traefik's InfluxDB metrics push

Traefik's metrics.influxdb2 exporter pushes on a 10s interval by default — an exact match for the reported cadence (90 rejections per…

spikerj commented on issue spikerj/spikersoft-issues#557 2026-07-14 06:30:02 +00:00
[Bug][Backend] BookAnalysisService (ExtractChapters / AnalyzeBookWithLLM) silently ships degraded book data on GPU/model-load failure — same class as quiz-gen #553, different pipeline

Diagnosis + fix in PR #275 (with #559 — same mechanism, one PR).

Root cause — three separate holes, all in BookAnalysisService:

  1. **ExtractChaptersWithLLM had a blanket `catch…
spikerj commented on issue spikerj/spikersoft-issues#554 2026-07-14 06:29:56 +00:00
[Bug][Infra] jetson-influxdb fleet exit-0 churn — all 7 replicas print banner, hit 'I/O error', exit and restart every ~10s (since the 2026-07-13 17:01Z redeploy)

Diagnosis: host-side jtop dependency, not config and not credentials

The reporter reads each board's stats from the host's jtop daemon over a bind-mounted unix socket (/run/jtop.sock).…

spikerj commented on issue spikerj/spikersoft-issues#559 2026-07-14 06:29:34 +00:00
[Bug][Backend][Quality] Quiz pages whose LLM output fails JSON parsing silently contribute 0 questions (WRN only) — quizzes ship smaller than requested

Diagnosis + fix in PR #275 (with #557 — same mechanism, one PR).

Root cause. QuizGenerationService.ParseGeneratedQuestions returned an empty list whenever the model's JSON was…

spikerj closed issue spikerj/spikersoft-issues#484 2026-07-14 06:23:15 +00:00
[Bug][Infra] API /healthz returns 503 — TileServerGL check failing (tiles.spikersoft.com 404s, no tile-server deployed)
spikerj commented on issue spikerj/spikersoft-issues#484 2026-07-14 06:23:15 +00:00
[Bug][Infra] API /healthz returns 503 — TileServerGL check failing (tiles.spikersoft.com 404s, no tile-server deployed)

Verified against the live endpoint before touching anything — this is already fixed and deployed.

/healthz currently returns HTTP 200 with status Degraded (12 checks Healthy,…

spikerj commented on issue spikerj/spikersoft-issues#556 2026-07-14 06:21:29 +00:00
[Bug][Backend][Infra] tiles.spikersoft.com has no backing service — API status/health canary fetches a terrain tile every few minutes and 404s 100% of the time

Diagnosis — your option 1 is correct

The tile service was never deployed / was decommissioned. Verified read-only against the swarm: a scan of all **94 services matches nothing on…

spikerj commented on issue spikerj/spikersoft-issues#484 2026-07-14 06:20:59 +00:00
[Bug][Infra] API /healthz returns 503 — TileServerGL check failing (tiles.spikersoft.com 404s, no tile-server deployed)

Diagnosis

Confirmed, and the headline symptom is already resolved.

The 503 is gone. /healthz currently returns HTTP 200 / "status": "Degraded" — 12 checks Healthy, TileServerG…

spikerj opened issue spikerj/spikersoft-issues#560 2026-07-14 06:20:40 +00:00
[Bug][Infra][GPU] gpu-coordinator arbitrates by declared bookings, not measured VRAM — over-budget tenants invisibly starve grantees (split from closed #553)
spikerj closed issue spikerj/spikersoft-issues#539 2026-07-14 06:19:38 +00:00
[Infra][Node] dreamstream6 (aarch64) down since ~16:30Z 2026-07-13 — heartbeat failure, host unreachable
spikerj commented on issue spikerj/spikersoft-issues#558 2026-07-14 06:18:42 +00:00
[Bug][Infra] SERVER root filesystem at 94% and climbing — remediation alerting every ~2min (occurrences 46+), alert-only with no cleanup path

Acute condition resolved: SERVER root is now 32% used, 286 GB free (was 94% and climbing, alerting every ~2 min).

Reclaimed by moving the MinIO data root onto the FusionIO card and deleting…

spikerj closed issue spikerj/spikersoft-issues#553 2026-07-14 06:18:41 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090
spikerj commented on issue spikerj/spikersoft-issues#553 2026-07-14 06:18:41 +00:00
[Bug][Prod][GPU] quiz-generation silently degrades ALL quizzes — Mistral-7B load fails (CUDA alloc), image-description holds 18.2GiB > its 16GB lease on SERVER's 4090

Both halves are fixed and verified.

The VRAM lease under-declaration — image-description declared a 16 GB lease while actually holding 18.2 GiB, so the coordinator believed there was room to…