Joseph Spiker

spikerj commented on issue spikerj/spikersoft-issues#589

2026-07-29 15:59:56 +00:00

[Bug][Prod][Data-loss] 10 messages are permanently dead in DLQs with ZERO consumers — 5 of them are real user quizzes killed by #553, now replayable

Audited against origin/master — NOT DONE. Neither the replay nor the systemic fix exists. Flagging the time-sensitive part first.

No replay tooling of any kind. `git grep -iE…

spikerj commented on issue spikerj/spikersoft-issues#582

2026-07-29 15:59:41 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting

Audited against origin/master — the premise has gone stale; this ticket needs restating rather than simply closing or leaving.

**The pin was deliberately reverted, and that's now…

spikerj commented on issue spikerj/spikersoft-issues#610

2026-07-29 15:59:26 +00:00

[Security][Infra] OpenBao: centralize public Let's Encrypt cert storage/distribution in Bao (Traefik acme.json → KV; mailserver + consumers pull) (epic #543)

Audited against origin/master — NOT DONE. Zero implementing change; all four acceptance criteria unstarted. Recording the greps so this doesn't need re-deriving:

**No KV path, policy or…

spikerj commented on issue spikerj/spikersoft-issues#615

2026-07-29 15:58:01 +00:00

[Bug][CI][Infra] #584 not fixed — deploy job's git pull runs unprivileged on a root-owned /mnt/infrastructure, so every infra deploy silently ships a stale stack

Audited against origin/master — the literal complaint was fixed; the impact persists. Staying open.

Fixed: every workflow carrying the pull now uses the sudo form. `git grep -l "sudo…

spikerj commented on issue spikerj/spikersoft-issues#521

2026-07-29 15:57:45 +00:00

[Infra][Registry] Gitea upstream-image mirror (skopeo → mirror/* org) + incremental stack repointing

Audited against origin/master — the workflow shipped, but the mirror is empty and zero stacks were repointed. Staying open. Concrete numbers so the remaining work is measurable:

Built:…

spikerj commented on issue spikerj/spikersoft-issues#546

2026-07-29 15:57:28 +00:00

[Security][Backend] OpenBao Phase 2: SpikerSoft.Secrets host-builder seam — service creds from Bao (epic #543)

Audited against origin/master — the seam itself is adopted fleet-wide, but 6 stacks were left behind and Stage 4 is unswept. Staying open.

**Adopted centrally rather than per-service,…

spikerj commented on issue spikerj/spikersoft-issues#633

2026-07-29 15:57:12 +00:00

[Go-Live Checklist][Security] Rotate + Bao-migrate the dev-environment secrets before production

New finding — a sixth item, not in the original five. Turned up while auditing #610. Locations only below, no values.

spikersoft-infrastructure/mailserver/docker-stack.yml commits **five…

spikerj commented on issue spikerj/spikersoft-issues#561

2026-07-29 15:56:17 +00:00

[Bug][Infra][Data] Keycloak postgres pg_statistic toast corruption persists after #552 closed — ClearExpiredEvents/session purge failing every 15-min cycle since 02:29Z, repair never run

Audited against origin/master — NOT DONE, and this is the same live defect as #482, not a distinct one. Recommending a merge.

Same corruption, same object, same failing job. Both…

spikerj commented on issue spikerj/spikersoft-issues#579

2026-07-29 15:56:03 +00:00

[Bug][CI][Infra] gitea-runners arm_v8_2 crash-looping on dreamstream6 since the #539 outage — ARM CI capacity silently down 3→2

Noting for the record — this was closed today (2026-07-29 ~15:36Z) with no closing comment, and I'm leaving it closed on the assumption that was deliberate. But the **durable fix is definitively…

spikerj closed issue spikerj/spikersoft-issues#824

2026-07-29 15:55:40 +00:00

CI gap: strictTemplates / app-build errors merge to master green (nx build spikersoft not gated)

spikerj commented on issue spikerj/spikersoft-issues#824

2026-07-29 15:55:27 +00:00

CI gap: strictTemplates / app-build errors merge to master green (nx build spikersoft not gated)

Resolved — all three suggested fixes have landed, via the epic #722 CI restructure (the workflow is now .gitea/workflows/main.yml; the old tests.yml this ticket was written against no…

spikerj commented on issue spikerj/spikersoft-issues#554

2026-07-29 15:53:02 +00:00

[Bug][Infra] jetson-influxdb fleet exit-0 churn — all 7 replicas print banner, hit 'I/O error', exit and restart every ~10s (since the 2026-07-13 17:01Z redeploy)

Audited against origin/master — the stack was hardened; the actual bug cannot be fixed from any repo we have. Staying open, and recording why, because this one is stuck for a structural…

spikerj commented on issue spikerj/spikersoft-issues#503

2026-07-29 15:52:50 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services

Audited against origin/master — root cause found and mitigated; the hardware fix is outstanding. Staying open.

Recorded in repo:

minio/docker-stack.yml:20-26 documents the bad RAM…

spikerj commented on issue spikerj/spikersoft-issues#514

2026-07-29 15:52:33 +00:00

[Infra][CI] Gitea Actions leaves GITEA-ACTION* volumes behind → build nodes recurrently fill up (need runner auto-cleanup / scheduled prune)

Audited against origin/master — the prune mechanism shipped; three items remain. Staying open.

Landed (infra PRs #48, #58): gitea-act-runner/cleanup/gitea-actions-prune.sh —…

spikerj commented on issue spikerj/spikersoft-issues#460

2026-07-29 15:52:20 +00:00

art_pipe observability: install OpenTelemetry into model venvs so worker spans (#430) actually export (#428)

Audited against artpipe origin/main — the code shipped; the acceptance criterion has never been reachable. Staying open.

Landed (PRs #9 850773b/85bd96e and #30 a2f4450/d15e2ec):…

spikerj commented on issue spikerj/spikersoft-issues#830

2026-07-29 15:52:08 +00:00

Flaky under load: keycloak-admin user-list spec settles resources with a fixed hop count (reddens full-suite SonarQube scan); 9 sibling specs share the pattern

Re-verified against origin/master — only the named spec was fixed; all nine siblings still carry the blind two-hop helper. Staying open, which matches what you asked for in comments…

spikerj commented on issue spikerj/spikersoft-issues#783

2026-07-29 15:51:58 +00:00

[Infra][MinIO] art-asset-artifacts: the ACTUAL last GridFS bucket → MinIO (ArtStudio stage outputs, ~4.5GB) (epic #413)

Audited against origin/master — code, infra and the data move are all done and recorded; three operator steps remain. Staying open on those.

Done:

`SpikerSoft.Api/Extensions/ServiceCo…

spikerj commented on issue spikerj/spikersoft-issues#530

2026-07-29 15:51:46 +00:00

[Backend][MinIO] GeoLite2 → reference-data bucket; self-provisioning GeoIP (epic #413, resolves GeoIP half of #489)

Audited against origin/master — the code is complete and the #540 blocker is genuinely gone, but nothing shows the job has ever run. Staying open on that last step.

Code, confirmed: -…

spikerj commented on issue spikerj/spikersoft-issues#529

2026-07-29 15:51:33 +00:00

[Infra][MinIO] Uploads-tree coordinated batch cutover — 7 services off /mnt/fusionio (epic #413)

Audited against origin/master — phase 2 has not started on any of the 7 services. Per-service state, so the remaining work is concrete rather than "the cutover":

Storage__UseS3=true…

spikerj commented on issue spikerj/spikersoft-issues#696

2026-07-29 15:51:17 +00:00

[Infra][MinIO] Cut over book/ebook artifacts to MinIO as sole store (ebooks slice of #413) + fix inline-mirror pipeline stall

Audited against origin/master. Correcting a stale note first: the 2026-07-18 comment saying the internal route was "NOT started" is wrong as of today.

Landed since that comment: -…