Joseph Spiker

spikerj opened issue spikerj/spikersoft-issues#587

2026-07-14 20:24:06 +00:00

[Bug][Prod][Registry] 14 services have DANGLING :latest manifests — the index points at child digests that 404. Any fresh pull fails. This is the root cause of #562

spikerj commented on issue spikerj/spikersoft-issues#585

2026-07-14 20:19:07 +00:00

[Bug][Prod][AI] Book embeddings are WRITE-ONLY — nothing in production ever reads them; RedisVectorSearchService is registered nowhere

QA Team — correcting one claim in my own report above, and adding the runtime evidence.

Correction. I wrote that the embeddings work "sits on the same 24 GB card that quiz-generation now…

spikerj commented on issue spikerj/spikersoft-issues#582

2026-07-14 20:17:27 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting

QA Team — 2026-07-14 ~21:00Z. Deployed and verified. This is fixed.

/mnt/infrastructure was synced and the stack redeployed at 20:12:26Z. Verified against the live service, not the…

spikerj opened issue spikerj/spikersoft-issues#586

2026-07-14 20:17:13 +00:00

[Bug][Tests][QuizGeneration] ValidRequest_IsConsumedAcked_QuizCompletesWithFakedQuestions_AndNotifies fails on master — quiz contains questions from pages other than the expected page 4

spikerj commented on issue spikerj/spikersoft-issues#564

2026-07-14 20:16:59 +00:00

[Bug][Backend][Quality] QuizGenerationService validation is fail-open — if the validation LLM call throws, the unvalidated question ships in the quiz

Fix in backend PR #283 — took the "retry once then fail-closed" shape, and the completion-notification count, because the second one turned out not to be optional.

**The gate now fails…

spikerj opened issue spikerj/spikersoft-issues#585

2026-07-14 20:16:58 +00:00

[Bug][Prod][AI] Book embeddings are WRITE-ONLY — nothing in production ever reads them; RedisVectorSearchService is registered nowhere

spikerj commented on issue spikerj/spikersoft-issues#582

2026-07-14 20:12:14 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting

QA Team — correcting my own previous note, 2026-07-14 ~20:45Z. The root cause is now identified: #584.

In my last comment I said a normal push would not redeploy quiz-generation and…

spikerj opened issue spikerj/spikersoft-issues#584

2026-07-14 20:11:45 +00:00

[Bug][CI][Deploy] 28 of 29 deploy jobs never 'git pull' /mnt/infrastructure — they deploy STALE stack files and report success

spikerj commented on issue spikerj/spikersoft-issues#582

2026-07-14 20:06:21 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting

QA Team — update, 2026-07-14 ~20:30Z.

Step 1 is done: PR #281 is merged. The quiz-generation deploy now passes METADATA_S3_SECRET_KEY and refuses to deploy if it is empty, so the…

spikerj opened issue spikerj/spikersoft-issues#583

2026-07-14 20:05:57 +00:00

[Bug][CI] 28 of 30 path-filtered workflows still don't re-run when their own workflow file changes — #538 fixed only artpipe

spikerj closed issue spikerj/spikersoft-issues#565

2026-07-14 20:03:41 +00:00

[Bug][Backend] RedisVectorSearchService in-memory fallback caps the candidate pool at TopK keys BEFORE scoring — fallback returns arbitrary chunks, not the most similar ones

spikerj commented on issue spikerj/spikersoft-issues#565

2026-07-14 20:03:38 +00:00

[Bug][Backend] RedisVectorSearchService in-memory fallback caps the candidate pool at TopK keys BEFORE scoring — fallback returns arbitrary chunks, not the most similar ones

Resolved in spikersoft-backend PR #279 (merged to master 2026-07-14 as 81507f8). ExecuteInMemorySearchAsync now scores every matching BookPageChunk:* key and then takes the top TopK by…

spikerj commented on issue spikerj/spikersoft-issues#573

2026-07-14 20:02:48 +00:00

[Bug][Backend][API] CodeExecutionResultConsumer uses autoAck:true — a student's execution result is silently lost if the API dies mid-handle

Worked in spikersoft-backend PR #282 (open, not yet merged). The premise here is inverted — flagging it because the audit that produced this ticket will produce more like it.

`SpikerSoft.Busi…

spikerj commented on issue spikerj/spikersoft-issues#579

2026-07-14 20:01:27 +00:00

[Bug][CI][Infra] gitea-runners arm_v8_2 crash-looping on dreamstream6 since the #539 outage — ARM CI capacity silently down 3→2

QA Team — sweep 2026-07-14 ~20:15Z. Still broken, and it lies to the obvious check.

docker service ls showed this at 1/1 during my sweep and I briefly recorded it as recovered.…

spikerj opened issue spikerj/spikersoft-issues#582

2026-07-14 20:01:01 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting

spikerj opened issue spikerj/spikersoft-issues#581

2026-07-14 19:57:14 +00:00

[Bug][CI][Deploy] lesson-video-processor has NEVER deployed — the VIDEOS_S3_SECRET_KEY Actions secret was never created, so its own deploy guard rejects every run

spikerj opened issue spikerj/spikersoft-issues#580

2026-07-14 17:43:50 +00:00

[Bug][CI] arm64v8 image builds intermittently fail pulling mcr.microsoft.com base image over IPv6 — 'network is unreachable', ~1 in 6 arm jobs

spikerj opened issue spikerj/spikersoft-issues#579

2026-07-14 17:43:18 +00:00

[Bug][CI][Infra] gitea-runners arm_v8_2 crash-looping on dreamstream6 since the #539 outage — ARM CI capacity silently down 3→2

spikerj commented on issue spikerj/spikersoft-issues#561

2026-07-14 17:42:49 +00:00

[Bug][Infra][Data] Keycloak postgres pg_statistic toast corruption persists after #552 closed — ClearExpiredEvents/session purge failing every 15-min cycle since 02:29Z, repair never run

QA Team — sweep 2026-07-14 ~17:45Z. Still firing, ~15h after this ticket was opened, and the error has moved.

Latest occurrence, straight off keycloak_keycloak on dreamstream5 at…

spikerj commented on issue spikerj/spikersoft-issues#562

2026-07-14 17:42:25 +00:00

[Bug][Infra][Deploy] Dreamstream nodes reject fresh tasks with 'No such image' for 1-3 minutes after (re)scheduling — 4th distinct service tonight; pulls likely fighting the UDM hairpin

QA Team — sweep 2026-07-14 ~17:45Z. This is no longer the transient pattern this ticket describes, at least for coderunner.

The ticket characterises the reject as self-healing: *"the…