Joseph Spiker

spikerj opened issue spikerj/spikersoft-issues#590

2026-07-14 20:38:46 +00:00

[Bug][CI][Deploy] cancel-in-progress on master kills DEPLOY jobs — back-to-back merges cancel each other and fixes silently never ship

spikerj opened issue spikerj/spikersoft-issues#589

2026-07-14 20:34:46 +00:00

[Bug][Prod][Data-loss] 10 messages are permanently dead in DLQs with ZERO consumers — 5 of them are real user quizzes killed by #553, now replayable

spikerj commented on issue spikerj/spikersoft-issues#575

2026-07-14 20:30:55 +00:00

[Bug][Backend][Scheduler] ScheduledTaskPublisher does not propagate W3C trace context on publish — violates AGENTS.md non-negotiable #1; scheduled-task work is untraceable end-to-end

Fix in backend PR #285. Diagnosis was right. Two corrections to the fix shape, both of which would have bitten:

1. "Inject via ActivityHelper" alone would have been a no-op here.…

spikerj opened issue spikerj/spikersoft-issues#588

2026-07-14 20:30:13 +00:00

[Bug][Telemetry] Every service reports service.namespace='1.0.0' — AddService()'s 2nd positional arg is serviceNamespace, not serviceVersion. Spans also carry no node identity

spikerj commented on issue spikerj/spikersoft-issues#562

2026-07-14 20:24:28 +00:00

[Bug][Infra][Deploy] Dreamstream nodes reject fresh tasks with 'No such image' for 1-3 minutes after (re)scheduling — 4th distinct service tonight; pulls likely fighting the UDM hairpin

QA Team — 2026-07-14 ~21:15Z. Root-caused. It is not the UDM hairpin, and it is not transient. Full analysis in #587.

This ticket's premise — *"the node rejects it 2-4 times over ~1-3…

spikerj closed issue spikerj/spikersoft-issues#564

2026-07-14 20:24:28 +00:00

[Bug][Backend][Quality] QuizGenerationService validation is fail-open — if the validation LLM call throws, the unvalidated question ships in the quiz

spikerj commented on issue spikerj/spikersoft-issues#564

2026-07-14 20:24:25 +00:00

[Bug][Backend][Quality] QuizGenerationService validation is fail-open — if the validation LLM call throws, the unvalidated question ships in the quiz

Resolved in spikersoft-backend PR #283 (merged to master as d633e5d). The validation gate now fails closed (drop the question after a bounded retry, AI:MaxValidationAttempts), GPU/VRAM…

spikerj closed issue spikerj/spikersoft-issues#573

2026-07-14 20:24:20 +00:00

[Bug][Backend][API] CodeExecutionResultConsumer uses autoAck:true — a student's execution result is silently lost if the API dies mid-handle

spikerj commented on issue spikerj/spikersoft-issues#573

2026-07-14 20:24:17 +00:00

[Bug][Backend][API] CodeExecutionResultConsumer uses autoAck:true — a student's execution result is silently lost if the API dies mid-handle

Resolved in spikersoft-backend PR #282 (merged to master as 760667c). Deleted the dead CodeExecutionResultConsumer + ICodeExecutionResultConsumer, and added `CodeExecutionResultDeliveryTest…

spikerj opened issue spikerj/spikersoft-issues#587

2026-07-14 20:24:06 +00:00

[Bug][Prod][Registry] 14 services have DANGLING :latest manifests — the index points at child digests that 404. Any fresh pull fails. This is the root cause of #562

spikerj commented on issue spikerj/spikersoft-issues#585

2026-07-14 20:19:07 +00:00

[Bug][Prod][AI] Book embeddings are WRITE-ONLY — nothing in production ever reads them; RedisVectorSearchService is registered nowhere

QA Team — correcting one claim in my own report above, and adding the runtime evidence.

Correction. I wrote that the embeddings work "sits on the same 24 GB card that quiz-generation now…

spikerj commented on issue spikerj/spikersoft-issues#582

2026-07-14 20:17:27 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting

QA Team — 2026-07-14 ~21:00Z. Deployed and verified. This is fixed.

/mnt/infrastructure was synced and the stack redeployed at 20:12:26Z. Verified against the live service, not the…

spikerj opened issue spikerj/spikersoft-issues#586

2026-07-14 20:17:13 +00:00

[Bug][Tests][QuizGeneration] ValidRequest_IsConsumedAcked_QuizCompletesWithFakedQuestions_AndNotifies fails on master — quiz contains questions from pages other than the expected page 4

spikerj commented on issue spikerj/spikersoft-issues#564

2026-07-14 20:16:59 +00:00

[Bug][Backend][Quality] QuizGenerationService validation is fail-open — if the validation LLM call throws, the unvalidated question ships in the quiz

Fix in backend PR #283 — took the "retry once then fail-closed" shape, and the completion-notification count, because the second one turned out not to be optional.

**The gate now fails…

spikerj opened issue spikerj/spikersoft-issues#585

2026-07-14 20:16:58 +00:00

[Bug][Prod][AI] Book embeddings are WRITE-ONLY — nothing in production ever reads them; RedisVectorSearchService is registered nowhere

spikerj commented on issue spikerj/spikersoft-issues#582

2026-07-14 20:12:14 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting

QA Team — correcting my own previous note, 2026-07-14 ~20:45Z. The root cause is now identified: #584.

In my last comment I said a normal push would not redeploy quiz-generation and…

spikerj opened issue spikerj/spikersoft-issues#584

2026-07-14 20:11:45 +00:00

[Bug][CI][Deploy] 28 of 29 deploy jobs never 'git pull' /mnt/infrastructure — they deploy STALE stack files and report success

spikerj commented on issue spikerj/spikersoft-issues#582

2026-07-14 20:06:21 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting

QA Team — update, 2026-07-14 ~20:30Z.

Step 1 is done: PR #281 is merged. The quiz-generation deploy now passes METADATA_S3_SECRET_KEY and refuses to deploy if it is empty, so the…

spikerj opened issue spikerj/spikersoft-issues#583

2026-07-14 20:05:57 +00:00

[Bug][CI] 28 of 30 path-filtered workflows still don't re-run when their own workflow file changes — #538 fixed only artpipe

spikerj closed issue spikerj/spikersoft-issues#565

2026-07-14 20:03:41 +00:00

[Bug][Backend] RedisVectorSearchService in-memory fallback caps the candidate pool at TopK keys BEFORE scoring — fallback returns arbitrary chunks, not the most similar ones