Joseph Spiker

spikerj commented on issue spikerj/spikersoft-issues#590

2026-07-14 21:35:20 +00:00

[Bug][CI][Deploy] cancel-in-progress on master kills DEPLOY jobs — back-to-back merges cancel each other and fixes silently never ship

QA Team — 🚨 ESCALATION. This is not an intermittent race. Master's CI has been in a total cancellation cascade all day. NOTHING merged today has shipped.

The measurement

Every one of…

spikerj opened issue spikerj/spikersoft-issues#597

2026-07-14 21:31:22 +00:00

[Bug][Backend][Observability] #575 was fixed in ONE publisher — BookController (5 publish sites) and 3 others still ship RabbitMQ messages with no traceparent, so user-initiated flows appear in Jaeger as disconnected orphan traces

spikerj closed issue spikerj/spikersoft-issues#569

2026-07-14 21:28:37 +00:00

[Bug][Backend][KeycloakEvents] Queue binds routing key '#' on amq.topic — subscribes to EVERY message on the exchange; a landmine for any future amq.topic publisher

spikerj commented on issue spikerj/spikersoft-issues#569

2026-07-14 21:28:34 +00:00

[Bug][Backend][KeycloakEvents] Queue binds routing key '#' on amq.topic — subscribes to EVERY message on the exchange; a landmine for any future amq.topic publisher

Merged to master in spikersoft-backend PR #289. The keycloak.events queue now binds KK.EVENT.# and the legacy catch-all # binding is explicitly unbound on startup, so foreign messages on…

spikerj commented on issue spikerj/spikersoft-issues#583

2026-07-14 21:28:24 +00:00

[Bug][CI] 28 of 30 path-filtered workflows still don't re-run when their own workflow file changes — #538 fixed only artpipe

Fix in spikersoft-backend PR #291 (open, awaiting merge).

Confirmed as filed, with one small correction to the scope: 30 workflows use paths: filters and 2 already self-trigger — but…

spikerj opened issue spikerj/spikersoft-issues#596

2026-07-14 21:27:28 +00:00

[Bug][CI][Registry] #548 was closed but its fix NEVER LANDED — all 23 of 23 multi-arch workflows still --amend mutable :amd64/:arm64v8 tags, and the cancelled-run window from #590 makes mixed-commit :latest reachable today

spikerj commented on issue spikerj/spikersoft-issues#587

2026-07-14 21:26:54 +00:00

[Bug][Prod][Registry] 14 images have no :latest manifest (manifest unknown / 404) — any reschedule onto a fresh node will fail to start

QA Team — ROOT CAUSE FOUND, and it is NOT registry corruption. The images are all fine.

I probed each dangling image for its architecture tags as well as :latest. The result is…

spikerj commented on issue spikerj/spikersoft-issues#590

2026-07-14 21:26:54 +00:00

[Bug][CI][Deploy] cancel-in-progress on master kills DEPLOY jobs — back-to-back merges cancel each other and fixes silently never ship

QA Team — this ticket's blast radius is bigger than "deploy jobs". It is also silently destroying multi-arch image manifests.

cancel-in-progress: true does not just skip deploys. In the…

spikerj commented on issue spikerj/spikersoft-issues#584

2026-07-14 21:25:08 +00:00

[Bug][CI][Deploy] 28 of 29 deploy jobs never 'git pull' /mnt/infrastructure — they deploy STALE stack files and report success

Fix in spikersoft-backend PR #290 (open, awaiting merge).

Confirmed exactly as filed, count and all: 29 workflows deploy from /mnt/infrastructure; 1 pulls (artpipe-processor); 28 do not.…

spikerj commented on issue spikerj/spikersoft-issues#595

2026-07-14 21:24:53 +00:00

[Bug][Prod][ArtPipe] Concept-art is dead because ArtStudio:StageModelMap is EMPTY — API routes to the legacy in-process stage queue whose consumer can't boot, while the 5 resident model containers sit idle on queues nobody publishes to

QA Team — correction to my note above re: #537 / the 98 GB image.

The repo owner clarified: the 98 GB figure in #537 was a bug, not the real image size — an unfiltered `snapshot_down…

spikerj commented on issue spikerj/spikersoft-issues#569

2026-07-14 21:20:20 +00:00

[Bug][Backend][KeycloakEvents] Queue binds routing key '#' on amq.topic — subscribes to EVERY message on the exchange; a landmine for any future amq.topic publisher

Fix in spikersoft-backend PR #289 (open, awaiting merge).

Confirmed exactly as filed. KeycloakEventHostedService.cs:120 bound the queue with `ConfigureQueueWithDlqAsync(_channel, "amq.topic",…

spikerj commented on issue spikerj/spikersoft-issues#595

2026-07-14 21:19:48 +00:00

[Bug][Prod][ArtPipe] Concept-art is dead because ArtStudio:StageModelMap is EMPTY — API routes to the legacy in-process stage queue whose consumer can't boot, while the 5 resident model containers sit idle on queues nobody publishes to

QA Team — ROOT CAUSE FOUND. This supersedes the fix I proposed above; that advice was wrong.

I originally suggested recreating the /mnt/fusionio bind on the 4090 or replacing it with MinIO…

spikerj opened issue spikerj/spikersoft-issues#595

2026-07-14 21:15:33 +00:00

[Bug][Prod][ArtPipe] Concept-art generation is silently dead — consumer can NEVER start (artpipe-gpu pin vs SERVER-only /mnt/fusionio bind, same class as #553) and SDXL generator is at replicas=0; live user request stuck unread in queue

spikerj opened issue spikerj/spikersoft-issues#594

2026-07-14 21:12:24 +00:00

[Bug][Backend][Observability] Serilog MinimumLevel is hardcoded to Debug in EventHandlerHostBuilder — EVERY appsettings log-level setting is dead config, and ~2.9M Debug events/day are shipped to Seq

spikerj commented on issue spikerj/spikersoft-issues#557

2026-07-14 21:12:22 +00:00

[Bug][Backend] BookAnalysisService (ExtractChapters / AnalyzeBookWithLLM) silently ships degraded book data on GPU/model-load failure — same class as quiz-gen #553, different pipeline

Status check after PR #275 (merged to master at 06:31Z, b3287f5).

The silent-degradation half is fixed and shipped. Verified on master: the blanket `catch (Exception) { return new…

spikerj opened issue spikerj/spikersoft-issues#593

2026-07-14 21:12:06 +00:00

[Bug][Backend][Books] Chapter extraction is orphaned end-to-end — nothing ever populates Book.Chapters, and GET /books/{id}/chapters returns [] for every book, always

spikerj closed issue spikerj/spikersoft-issues#559

2026-07-14 21:11:14 +00:00

[Bug][Backend][Quality] Quiz pages whose LLM output fails JSON parsing silently contribute 0 questions (WRN only) — quizzes ship smaller than requested

spikerj commented on issue spikerj/spikersoft-issues#559

2026-07-14 21:11:10 +00:00

[Bug][Backend][Quality] Quiz pages whose LLM output fails JSON parsing silently contribute 0 questions (WRN only) — quizzes ship smaller than requested

Resolved in spikersoft-backend PR #275, merged to master at 06:31Z (b3287f5). Verified present on master: QuizResponseParser, LlmOutputUnparsableException, Quiz.DegradedPageNumbers,…

spikerj closed issue spikerj/spikersoft-issues#591

2026-07-14 21:09:15 +00:00

[Bug][Tests][Curriculum] CurriculumSmokeTests picks whatever python3 is first on PATH — macOS system Python 3.9 fails the match-statement lessons (21300/21301) with a raw SyntaxError

spikerj commented on issue spikerj/spikersoft-issues#591

2026-07-14 21:09:10 +00:00

[Bug][Tests][Curriculum] CurriculumSmokeTests picks whatever python3 is first on PATH — macOS system Python 3.9 fails the match-statement lessons (21300/21301) with a raw SyntaxError

Resolved in spikersoft-backend PR #287 (merged to master).

Root cause was neither the curriculum nor the product code: the test harness invoked bare python3, which on macOS resolves to the…