Joseph Spiker

spikerj commented on issue spikerj/spikersoft-issues#562

2026-07-14 21:06:34 +00:00

[Bug][Infra][Deploy] Dreamstream nodes reject fresh tasks with 'No such image' for 1-3 minutes after (re)scheduling — 4th distinct service tonight; pulls likely fighting the UDM hairpin

QA Team — root cause CONFIRMED. Your original hairpin theory was right. Evidence below.

git.spikersoft.com resolves to a public IP from inside the cluster:

spikerj commented on issue spikerj/spikersoft-issues#588

2026-07-14 21:04:47 +00:00

[Bug][Telemetry] Every service reports service.namespace='1.0.0' — AddService()'s 2nd positional arg is serviceNamespace, not serviceVersion. Spans also carry no node identity

Fixed across two PRs (both open, awaiting merge). All three items in this ticket are covered.

spikersoft-backend #288 — the code fix
spikersoft-infrastructure #82 — the…

spikerj commented on issue spikerj/spikersoft-issues#579

2026-07-14 20:49:35 +00:00

[Bug][CI][Infra] gitea-runners arm_v8_2 crash-looping on dreamstream6 since the #539 outage — ARM CI capacity silently down 3→2

QA Team — still flapping. Re-verified 2026-07-14.

gitea-runners_arm_v8_2_act_runner on dreamstream6 is still crash-looping, ~3 days on:

spikerj commented on issue spikerj/spikersoft-issues#587

2026-07-14 20:49:35 +00:00

[Bug][Prod][Registry] 14 images have no :latest manifest (manifest unknown / 404) — any reschedule onto a fresh node will fail to start

QA Team — still live, still causing #562. Re-verified 2026-07-14.

The dangling manifests have not self-healed. Spot-check from a swarm node:

spikerj commented on issue spikerj/spikersoft-issues#582

2026-07-14 20:48:34 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting

QA Team — update: the placement half of this ticket is RESOLVED, but do not close it yet.

Infra #81 merged and deployed at 2026-07-14 20:11 UTC. Verified live on the swarm:

spikerj closed issue spikerj/spikersoft-issues#586

2026-07-14 20:48:16 +00:00

[Bug][Tests][QuizGeneration] ValidRequest_IsConsumedAcked_QuizCompletesWithFakedQuestions_AndNotifies fails on master — quiz contains questions from pages other than the expected page 4

spikerj opened issue spikerj/spikersoft-issues#592

2026-07-14 20:48:11 +00:00

[Bug][Prod][GPU] quiz-generation is green, consuming, and CANNOT generate a quiz — deployed with an empty Storage__SecretKey and no model bind (follow-on to #553 / infra #81)

spikerj commented on issue spikerj/spikersoft-issues#586

2026-07-14 20:48:11 +00:00

[Bug][Tests][QuizGeneration] ValidRequest_IsConsumedAcked_QuizCompletesWithFakedQuestions_AndNotifies fails on master — quiz contains questions from pages other than the expected page 4

Resolved in spikersoft-backend PR #284 (merged to master as 5085e57).

Answering the two hypotheses in the report: neither. Not test isolation, and not a front-matter bug — the worker was…

spikerj closed issue spikerj/spikersoft-issues#575

2026-07-14 20:48:02 +00:00

[Bug][Backend][Scheduler] ScheduledTaskPublisher does not propagate W3C trace context on publish — violates AGENTS.md non-negotiable #1; scheduled-task work is untraceable end-to-end

spikerj commented on issue spikerj/spikersoft-issues#575

2026-07-14 20:47:54 +00:00

[Bug][Backend][Scheduler] ScheduledTaskPublisher does not propagate W3C trace context on publish — violates AGENTS.md non-negotiable #1; scheduled-task work is untraceable end-to-end

Resolved in spikersoft-backend PR #285 (merged to master as f3f5166). ScheduledTaskPublisher now starts a real Producer span and injects W3C trace context; ScheduledTaskConsumer extracts…

spikerj opened issue spikerj/spikersoft-issues#591

2026-07-14 20:43:33 +00:00

[Bug][Tests][Curriculum] CurriculumSmokeTests picks whatever python3 is first on PATH — macOS system Python 3.9 fails the match-statement lessons (21300/21301) with a raw SyntaxError

spikerj opened issue spikerj/spikersoft-issues#590

2026-07-14 20:38:46 +00:00

[Bug][CI][Deploy] cancel-in-progress on master kills DEPLOY jobs — back-to-back merges cancel each other and fixes silently never ship

spikerj opened issue spikerj/spikersoft-issues#589

2026-07-14 20:34:46 +00:00

[Bug][Prod][Data-loss] 10 messages are permanently dead in DLQs with ZERO consumers — 5 of them are real user quizzes killed by #553, now replayable

spikerj commented on issue spikerj/spikersoft-issues#575

2026-07-14 20:30:55 +00:00

[Bug][Backend][Scheduler] ScheduledTaskPublisher does not propagate W3C trace context on publish — violates AGENTS.md non-negotiable #1; scheduled-task work is untraceable end-to-end

Fix in backend PR #285. Diagnosis was right. Two corrections to the fix shape, both of which would have bitten:

1. "Inject via ActivityHelper" alone would have been a no-op here.…

spikerj opened issue spikerj/spikersoft-issues#588

2026-07-14 20:30:13 +00:00

[Bug][Telemetry] Every service reports service.namespace='1.0.0' — AddService()'s 2nd positional arg is serviceNamespace, not serviceVersion. Spans also carry no node identity

spikerj closed issue spikerj/spikersoft-issues#564

2026-07-14 20:24:28 +00:00

[Bug][Backend][Quality] QuizGenerationService validation is fail-open — if the validation LLM call throws, the unvalidated question ships in the quiz

spikerj commented on issue spikerj/spikersoft-issues#562

2026-07-14 20:24:28 +00:00

[Bug][Infra][Deploy] Dreamstream nodes reject fresh tasks with 'No such image' for 1-3 minutes after (re)scheduling — 4th distinct service tonight; pulls likely fighting the UDM hairpin

QA Team — 2026-07-14 ~21:15Z. Root-caused. It is not the UDM hairpin, and it is not transient. Full analysis in #587.

This ticket's premise — *"the node rejects it 2-4 times over ~1-3…

spikerj commented on issue spikerj/spikersoft-issues#564

2026-07-14 20:24:25 +00:00

[Bug][Backend][Quality] QuizGenerationService validation is fail-open — if the validation LLM call throws, the unvalidated question ships in the quiz

Resolved in spikersoft-backend PR #283 (merged to master as d633e5d). The validation gate now fails closed (drop the question after a bounded retry, AI:MaxValidationAttempts), GPU/VRAM…

spikerj closed issue spikerj/spikersoft-issues#573

2026-07-14 20:24:20 +00:00

[Bug][Backend][API] CodeExecutionResultConsumer uses autoAck:true — a student's execution result is silently lost if the API dies mid-handle

spikerj commented on issue spikerj/spikersoft-issues#573

2026-07-14 20:24:17 +00:00

[Bug][Backend][API] CodeExecutionResultConsumer uses autoAck:true — a student's execution result is silently lost if the API dies mid-handle

Resolved in spikersoft-backend PR #282 (merged to master as 760667c). Deleted the dead CodeExecutionResultConsumer + ICodeExecutionResultConsumer, and added `CodeExecutionResultDeliveryTest…