Joseph Spiker spikerj
  • Joined on 2025-11-03
spikerj commented on issue spikerj/spikersoft-issues#512 2026-07-12 18:59:47 +00:00
[Bug][Frontend] Author's own staff-removed blog posts show on the dashboard with no moderation indicator

Audit of the /blog list page complete: already coveredBlogEntryComponent carries a status-removed badge + removed-post card styling for isDeleted posts (shipped with the moderation…

spikerj opened issue spikerj/spikersoft-issues#512 2026-07-12 18:55:21 +00:00
[Bug][Frontend] Author's own staff-removed blog posts show on the dashboard with no moderation indicator
spikerj commented on issue spikerj/spikersoft-issues#511 2026-07-12 18:49:42 +00:00
[Bug][Infra][Deploy] quiz-generation rolling update races the image push — 'No such image' reject loops + UpdateStatus=paused

Fleet sweep up: PR #239 gates all 28 image-publishing workflows (also re-lands quiz-generation's gate — #238 had merged with only the accidental empty commit). With infra #44's rollback…

spikerj commented on issue spikerj/spikersoft-issues#504 2026-07-12 18:48:12 +00:00
[Bug][Infra] MinIO doesn't recover after a transient /mnt/fusionio drive blip — stuck '0 drives provided', 'mc ready local' healthcheck misses it, needs manual restart

Fixed, deployed, and verified as far as possible without a real blip — closing. Infra #42 (merged, deployed live during the session): the healthcheck now runs an authenticated mc ls data-path…

spikerj closed issue spikerj/spikersoft-issues#504 2026-07-12 18:48:12 +00:00
[Bug][Infra] MinIO doesn't recover after a transient /mnt/fusionio drive blip — stuck '0 drives provided', 'mc ready local' healthcheck misses it, needs manual restart
spikerj closed issue spikerj/spikersoft-issues#507 2026-07-12 18:48:11 +00:00
[Bug][Backend] Zero-question quizzes are marked Completed and takeable — starting one insta-expires at 0/0
spikerj closed issue spikerj/spikersoft-issues#509 2026-07-12 18:48:11 +00:00
[Bug][Backend][Data] BookPages inflation — quiz saw 6 pages on a 3-page book; one book linked 570 pages (staleness-retry double-processing suspected)
spikerj commented on issue spikerj/spikersoft-issues#509 2026-07-12 18:48:11 +00:00
[Bug][Backend][Data] BookPages inflation — quiz saw 6 pages on a 3-page book; one book linked 570 pages (staleness-retry double-processing suspected)

Fixed and closing: PR #234 (merged, auto-deployed) — WasDuplicate flows through BookCreatedResult → book.created → orchestrator, which deletes the redundant extraction and completes the…

spikerj commented on issue spikerj/spikersoft-issues#507 2026-07-12 18:48:11 +00:00
[Bug][Backend] Zero-question quizzes are marked Completed and takeable — starting one insta-expires at 0/0

Fixed across three merged PRs and deployed via CI: #231 (0-question generations marked Failed + StartQuizAttempt guard + 400 mapping), #233 (page window no longer skips past small books — the…

spikerj commented on issue spikerj/spikersoft-issues#511 2026-07-12 18:46:08 +00:00
[Bug][Infra][Deploy] quiz-generation rolling update races the image push — 'No such image' reject loops + UpdateStatus=paused

Both guards up: backend PR #238 (post-push docker manifest inspect retry gate — the deploy step can no longer start until the digest is actually servable; a push that never lands fails the…

spikerj commented on issue spikerj/spikersoft-issues#480 2026-07-12 18:44:33 +00:00
[Bug][Infra] spikersoft-node-agent running on 1/10 nodes — rejected on missing /var/log/journal and /etc/timezone binds

Priority escalation — tonight made this ticket expensive. Three resource-exhaustion incidents in one session, all in the disk/exhaustion class the #381 probe already detects: (1) runner…

spikerj commented on issue spikerj/spikersoft-issues#493 2026-07-12 18:42:59 +00:00
[Infra][MinIO] Pilot: deploy MinIO + migrate security-scanner off /mnt/fusionio to S3 (proves the pattern for #413)

Scanner-flip preflight complete — plus a sequencing hazard to defuse:

security.scan.requested has exactly ONE publisher (UploadOrchestrator) — and merged PR #236 mirrors the staging…

spikerj commented on issue spikerj/spikersoft-issues#510 2026-07-12 18:32:28 +00:00
[Exploration][Infra] Resilience to SERVER node flaps — restart-policy windows, auto-reconcile sweep, de-SPOF the SERVER-pinned stateful tier

Concrete recurring instance of item C filed as 511 — quiz-generation's rolling update races its own image push: spec pinned to @sha256:X → swarm 'No such image' → same digest pullable minutes…

spikerj opened issue spikerj/spikersoft-issues#511 2026-07-12 18:32:28 +00:00
[Bug][Infra][Deploy] quiz-generation rolling update races the image push — 'No such image' reject loops + UpdateStatus=paused
spikerj commented on issue spikerj/spikersoft-issues#503 2026-07-12 18:29:51 +00:00
[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services

Filed exploration #510 — resilience to these SERVER flaps. Scopes the fixes cheapest-first: (A) restart-policy window on the 11 on-failure/maxAttempts=3/no-window services that keep wedging,…

spikerj opened issue spikerj/spikersoft-issues#510 2026-07-12 18:29:39 +00:00
[Exploration][Infra] Resilience to SERVER node flaps — restart-policy windows, auto-reconcile sweep, de-SPOF the SERVER-pinned stateful tier
spikerj commented on issue spikerj/spikersoft-issues#501 2026-07-12 18:28:27 +00:00
[Bug][Infra][GPU] artpipe lane down on 4090 — workers Rejected on missing /mnt/fusionio/spikersoft/art_pipe bind after artpipe-gpu label move (collateral of #500)

Analysis + decision needed (this is an ops choice, not a code fix — laying out the paths):

Why it broke: the six artpipe stacks follow the movable artpipe-gpu label AND bind `/mnt/fusionio/…

spikerj commented on issue spikerj/spikersoft-issues#503 2026-07-12 18:26:17 +00:00
[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services

Recurrence 2026-07-12 ~18:21 UTC (node owner reports another SERVER crash). Much milder than the 08:26 hard-down: SERVER came back Ready/Active/Reachable quickly (daemon flap, not a hard-down).…

spikerj commented on issue spikerj/spikersoft-issues#493 2026-07-12 18:25:10 +00:00
[Infra][MinIO] Pilot: deploy MinIO + migrate security-scanner off /mnt/fusionio to S3 (proves the pattern for #413)

Progress sweep: #500/#502/#508/#506/#497 all closed today with production evidence — captions flow from the 4090 with images AND model sourced from MinIO (110 completed captions in Mongo).…

spikerj commented on issue spikerj/spikersoft-issues#497 2026-07-12 18:22:56 +00:00
[Bug][Prod] /mnt/fusionio/spikersoft/ai missing embedding model — book pipeline dies at embeddings (nomic-embed-text-v2-moe.f16.gguf not found)

All models verified loading in production — closing. nomic: 'Embedding model loaded successfully with 999 GPU layers' (17:25:25). Mistral: quiz generation produced real questions ('Total…