Joseph Spiker

spikerj commented on issue spikerj/spikersoft-issues#503

2026-07-12 08:58:54 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services

Recovery note (2026-07-12 ~08:57): SERVER back at ~08:51 (~25 min outage), but 6 services wedged post-recovery

SERVER rejoined ~08:51; minio/backend/mongo-router self-recovered (minio…

spikerj commented on issue spikerj/spikersoft-issues#503

2026-07-12 08:30:32 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services

ESCALATION (2026-07-12 ~08:29): SERVER is HARD DOWN again — recurrence, live outage

After ~45 min of stability, SERVER went Down / Unreachable again at ~08:26 (docker node ls → `SERVER…

spikerj commented on issue spikerj/spikersoft-issues#503

2026-07-12 08:09:32 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services

Correction / update (2026-07-12 ~08:08)

Per node owner: SERVER was under super heavy load; the fusionio drive recovered on its own and the node is healthy again — so this reads as a…

spikerj opened issue spikerj/spikersoft-issues#504

2026-07-12 08:09:32 +00:00

[Bug][Infra] MinIO doesn't recover after a transient /mnt/fusionio drive blip — stuck '0 drives provided', 'mc ready local' healthcheck misses it, needs manual restart

spikerj commented on issue spikerj/spikersoft-issues#503

2026-07-12 08:06:53 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services

Update (2026-07-12 ~08:05): storage side of the same fault — MinIO lost its drive

The SERVER node is Ready and the daemon has been stable ~26 min (no flap since ~07:39), but **`minio_minio…

spikerj opened issue spikerj/spikersoft-issues#503

2026-07-12 07:45:34 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services

spikerj opened issue spikerj/spikersoft-issues#502

2026-07-12 07:19:24 +00:00

[Bug][Prod][GPU] image-description crash-loops on gated HuggingFace model — no HF token for Qwen3-VL-8B (401 GatedRepoError)

spikerj commented on issue spikerj/spikersoft-issues#500

2026-07-12 07:19:24 +00:00

[Bug][Infra][GPU] image-description lease starves — gpu-coordinator single-lane budget still on SERVER (8192MB) vs 16000MB needed; enable 4090 lane

Update (2026-07-12 ~07:18): the coordinator fix appears live — the image-description worker now gets its GPU lease (no more position 1/1 starvation) and proceeds to load the model. The…

spikerj opened issue spikerj/spikersoft-issues#501

2026-07-12 07:05:39 +00:00

[Bug][Infra][GPU] artpipe lane down on 4090 — workers Rejected on missing /mnt/fusionio/spikersoft/art_pipe bind after artpipe-gpu label move (collateral of #500)

spikerj commented on issue spikerj/spikersoft-issues#500

2026-07-12 07:02:25 +00:00

[Bug][Infra][GPU] image-description lease starves — gpu-coordinator single-lane budget still on SERVER (8192MB) vs 16000MB needed; enable 4090 lane

PR opened from the pre-pushed branch: spikersoft-infrastructure #40 (fix/gpu-coordinator-4090-burst, commit 83916a4 — used as-is; it merges cleanly, the coordinator stack was untouched by…

spikerj opened issue spikerj/spikersoft-issues#500

2026-07-12 07:00:00 +00:00

[Bug][Infra][GPU] image-description lease starves — gpu-coordinator single-lane budget still on SERVER (8192MB) vs 16000MB needed; enable 4090 lane

spikerj closed issue spikerj/spikersoft-issues#498

2026-07-12 06:30:39 +00:00

[Infra][MinIO] Move Gitea package/container-registry storage to the gitea-packages bucket

spikerj commented on issue spikerj/spikersoft-issues#498

2026-07-12 06:30:39 +00:00

[Infra][MinIO] Move Gitea package/container-registry storage to the gitea-packages bucket

Verified end-to-end: go-forward cutover complete (Joey opted to skip migrating old artifacts — all 143 pre-existing package versions were wiped so Gitea's blob dedup can't reference ghost files,…

spikerj closed issue spikerj/spikersoft-issues#499

2026-07-12 06:30:38 +00:00

[Bug][Backend] Book deletion leaves all artifacts behind — on disk AND in the MinIO ebooks bucket (no purge path exists)

spikerj commented on issue spikerj/spikersoft-issues#499

2026-07-12 06:30:38 +00:00

[Bug][Backend] Book deletion leaves all artifacts behind — on disk AND in the MinIO ebooks bucket (no purge path exists)

Verified deployed: Seq shows DeletedBookPurgeService active: purging books soft-deleted more than 30 days ago, every 1.00:00:00 on the rolled file-movement (backend #227 + infra #39, both…

spikerj commented on issue spikerj/spikersoft-issues#499

2026-07-12 06:21:26 +00:00

[Bug][Backend] Book deletion leaves all artifacts behind — on disk AND in the MinIO ebooks bucket (no purge path exists)

Option 1 implemented (Joey's pick — retention purge, 30-day undelete window):

spikersoft-backend #227 — DeletedBookPurgeService in file-movement (the file-lifecycle owner with the…

spikerj opened issue spikerj/spikersoft-issues#499

2026-07-12 06:08:29 +00:00

[Bug][Backend] Book deletion leaves all artifacts behind — on disk AND in the MinIO ebooks bucket (no purge path exists)

spikerj opened issue spikerj/spikersoft-issues#498

2026-07-12 05:09:38 +00:00

[Infra][MinIO] Move Gitea package/container-registry storage to the gitea-packages bucket

spikerj commented on issue spikerj/spikersoft-issues#403

2026-07-12 05:04:36 +00:00

[infra drift] image-description & trellis3d deploy to /mnt/infrastructure folders that don't exist

Image-description half resolved as (b) — and it's definitively LIVE, not superseded: tonight's book upload proved the UploadOrchestrator actively publishes image.description.requested (36…

spikerj commented on issue spikerj/spikersoft-issues#403

2026-07-12 04:55:32 +00:00

[infra drift] image-description & trellis3d deploy to /mnt/infrastructure folders that don't exist

Production evidence: image-description is still live (resolves the (a) superseded vs (b) still-live question) — and its absence is a user-facing book-upload failure

Surfaced today…