spikerj commented on issue spikerj/spikersoft-issues#717 2026-07-29 16:04:22 +00:00
[CI][Backend] Tier-3 artpipe finals builds keep getting superseded/cancelled before pushing — dino fix (#702) never reached the registry

Audited against origin/master — NOT DONE, none of the three fix options. And the audit turned up something sharper than "nobody got to it":

**This is the one backend workflow with no…

spikerj commented on issue spikerj/spikersoft-issues#774 2026-07-29 16:04:07 +00:00
[Bug][Infra][Deploy] backend deploy reports success while swarm rolls back on transient image-pull failure (silent non-deploy, #584 class)

Audited against origin/master — NOT DONE, and the blast radius is fleet-wide. Staying open.

No convergence check exists. The shared deploy path is `spikersoft-backend/.gitea/workflows/_…

spikerj commented on issue spikerj/spikersoft-issues#825 2026-07-29 16:03:52 +00:00
Art Studio MinIO migration: reader cutover preceded writer cutover → gap-window artifacts orphaned (500 on download)

Audited against origin/master — the incident was remediated; none of the four prevention items landed. Staying open, and the prevention half is the part that matters.

The remediation is…

spikerj commented on issue spikerj/spikersoft-issues#756 2026-07-29 16:02:56 +00:00
[Critical][Ops][Notifications] Ops alerting is silently dead — every system-remediation-alert email fails SMTP AUTH 535; critical disk/OOM alerts never delivered

Audited against origin/master — NOT DONE, and fix 3 as written is currently impossible. That's the useful finding, so leading with it.

**Fix 3 — "alert on the notifications DLQ / failure…

spikerj commented on issue spikerj/spikersoft-issues#613 2026-07-29 16:01:22 +00:00
[Bug][Infra][MinIO] file-movement's metadata-svc key can't write the quarantine bucket (Access Denied) — blocks #526/#529 cutover (epic #413)

Audited against origin/master — all the code and provisioning landed; only the one-time key rotation remains. Staying open on that single step.

Committed and complete:

  • `spikersoft-fil…
spikerj commented on issue spikerj/spikersoft-issues#538 2026-07-29 16:01:02 +00:00
[Infra][Network] LAN traffic to *.spikersoft.com NAT-hairpins through the UDM — local DNS records + direct Gitea→MinIO path

Audited against origin/master — tier 2 is done and annotated; tier 1 (the headline fix) has no evidence of any kind. Staying open.

Tier 2 — Gitea→MinIO direct path: DONE, and…

spikerj commented on issue spikerj/spikersoft-issues#519 2026-07-29 16:00:48 +00:00
[ArtPipe] Baked images Phase 3: prodstages image + live modeling-lane cutover (epic #515)

Audited against origin/master — the blocking gate has cleared; only the live cutover verification remains. Staying open on that.

The tier-3 image now exists, which the 2026-07-22 board…

spikerj commented on issue spikerj/spikersoft-issues#603 2026-07-29 16:00:34 +00:00
[Bug][Infra][Observability][QA] node-agent shows a green 7/7 while missing on ds1/ds4/ds6 — global-service desired-count hides a 30% monitoring blind spot on exactly the degraded nodes

Audited against origin/master — NOT DONE. No implementing change of any kind.

  • git log origin/master --grep='#603' across all repos: zero commits.
  • No global-service coverage guard…
spikerj commented on issue spikerj/spikersoft-issues#628 2026-07-29 16:00:21 +00:00
[Bug][CI][E2E] e2e.staff credential drift — OpenBao staff_password no longer authenticates against Keycloak (student from same secret works)

Audited — UNVERIFIABLE FROM GIT, with no positive evidence of a fix. Treat as still open.

Passwords live in OpenBao and Keycloak, neither of which is in git — correctly so, per the central…

spikerj commented on issue spikerj/spikersoft-issues#605 2026-07-29 16:00:08 +00:00
[Test][E2E] Split authenticated e2e into staff + student roles (two identities)

Audited against origin/master — PARTIAL, exactly as the 2026-07-22 reconciliation comment says. The credential tooling shipped; the suite split has not started.

**Done — persona…

spikerj commented on issue spikerj/spikersoft-issues#589 2026-07-29 15:59:56 +00:00
[Bug][Prod][Data-loss] 10 messages are permanently dead in DLQs with ZERO consumers — 5 of them are real user quizzes killed by #553, now replayable

Audited against origin/master — NOT DONE. Neither the replay nor the systemic fix exists. Flagging the time-sensitive part first.

No replay tooling of any kind. `git grep -iE…

spikerj commented on issue spikerj/spikersoft-issues#582 2026-07-29 15:59:41 +00:00
[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting

Audited against origin/master — the premise has gone stale; this ticket needs restating rather than simply closing or leaving.

**The pin was deliberately reverted, and that's now…

spikerj commented on issue spikerj/spikersoft-issues#610 2026-07-29 15:59:26 +00:00
[Security][Infra] OpenBao: centralize public Let's Encrypt cert storage/distribution in Bao (Traefik acme.json → KV; mailserver + consumers pull) (epic #543)

Audited against origin/master — NOT DONE. Zero implementing change; all four acceptance criteria unstarted. Recording the greps so this doesn't need re-deriving:

  • **No KV path, policy or…
spikerj commented on issue spikerj/spikersoft-issues#615 2026-07-29 15:58:01 +00:00
[Bug][CI][Infra] #584 not fixed — deploy job's git pull runs unprivileged on a root-owned /mnt/infrastructure, so every infra deploy silently ships a stale stack

Audited against origin/master — the literal complaint was fixed; the impact persists. Staying open.

Fixed: every workflow carrying the pull now uses the sudo form. `git grep -l "sudo…

spikerj commented on issue spikerj/spikersoft-issues#521 2026-07-29 15:57:45 +00:00
[Infra][Registry] Gitea upstream-image mirror (skopeo → mirror/* org) + incremental stack repointing

Audited against origin/master — the workflow shipped, but the mirror is empty and zero stacks were repointed. Staying open. Concrete numbers so the remaining work is measurable:

Built:…

spikerj commented on issue spikerj/spikersoft-issues#546 2026-07-29 15:57:28 +00:00
[Security][Backend] OpenBao Phase 2: SpikerSoft.Secrets host-builder seam — service creds from Bao (epic #543)

Audited against origin/master — the seam itself is adopted fleet-wide, but 6 stacks were left behind and Stage 4 is unswept. Staying open.

**Adopted centrally rather than per-service,…

spikerj commented on issue spikerj/spikersoft-issues#633 2026-07-29 15:57:12 +00:00
[Go-Live Checklist][Security] Rotate + Bao-migrate the dev-environment secrets before production

New finding — a sixth item, not in the original five. Turned up while auditing #610. Locations only below, no values.

spikersoft-infrastructure/mailserver/docker-stack.yml commits **five…

spikerj commented on issue spikerj/spikersoft-issues#561 2026-07-29 15:56:17 +00:00
[Bug][Infra][Data] Keycloak postgres pg_statistic toast corruption persists after #552 closed — ClearExpiredEvents/session purge failing every 15-min cycle since 02:29Z, repair never run

Audited against origin/master — NOT DONE, and this is the same live defect as #482, not a distinct one. Recommending a merge.

Same corruption, same object, same failing job. Both…

spikerj commented on issue spikerj/spikersoft-issues#579 2026-07-29 15:56:03 +00:00
[Bug][CI][Infra] gitea-runners arm_v8_2 crash-looping on dreamstream6 since the #539 outage — ARM CI capacity silently down 3→2

Noting for the record — this was closed today (2026-07-29 ~15:36Z) with no closing comment, and I'm leaving it closed on the assumption that was deliberate. But the **durable fix is definitively…

spikerj closed issue spikerj/spikersoft-issues#824 2026-07-29 15:55:40 +00:00
CI gap: strictTemplates / app-build errors merge to master green (nx build spikersoft not gated)