Joseph Spiker spikerj
  • Joined on 2025-11-03
spikerj commented on issue spikerj/spikersoft-issues#539 2026-07-13 20:57:29 +00:00
[Infra][Node] dreamstream6 (aarch64) down since ~16:30Z 2026-07-13 — heartbeat failure, host unreachable

QA watch 2026-07-13 ~20:57Z — redis cluster health after rejoin: VERIFIED OK. Cluster gossip shows no persistent failures; one replica briefly entered FAIL at 20:48:44Z and cleared at 20:49:40Z…

spikerj commented on issue spikerj/spikersoft-issues#536 2026-07-13 20:52:32 +00:00
[CI][ArtPipe] ModelEnvImages has no concurrency guard — concurrent runs race the same :latest tags (observed live)

QA watch 2026-07-13 ~20:53Z — guard merged (a3fba37) and version-compatible. Implementation: workflow-level concurrency group 'model-env-images' with cancel-in-progress: true — newest commit…

spikerj commented on issue spikerj/spikersoft-issues#534 2026-07-13 20:52:03 +00:00
[CI][ArtPipe] Route baked model-image builds (ModelEnvImages + tier-3 finals) to the 4090 runner, off laptop-server

Both halves MERGED (artpipe #16 + backend #266). Ticket closeable after the next finals cascade runs green on the 4090 — no further code work.

spikerj commented on issue spikerj/spikersoft-issues#534 2026-07-13 20:47:54 +00:00
[CI][ArtPipe] Route baked model-image builds (ModelEnvImages + tier-3 finals) to the 4090 runner, off laptop-server

QA watch 2026-07-13 ~20:48Z — BOTH halves now merged; code-complete. (1) ModelEnvImages → 4090: landed earlier (model-env-images.yml:50), live-proven by run 10821 executing on the 4090 (verified…

spikerj commented on issue spikerj/spikersoft-issues#531 2026-07-13 20:47:46 +00:00
[Infra][MinIO] Migrate Angular nginx root bind — presigned/proxy, LAST (epic #413)

QA watch 2026-07-13 ~20:48Z — revert verified, and it UNBLOCKS the security deploy. Revert 2d7caa63 (PR #179) removed only the proxy commit (d5fd39f3); the security deny block from 10f80c18 is…

spikerj commented on issue spikerj/spikersoft-issues#531 2026-07-13 20:47:09 +00:00
[Infra][MinIO] Migrate Angular nginx root bind — presigned/proxy, LAST (epic #413)

Race outcome: revert #179 MERGED before any of the three queued Frontend CI deploys landed — probes show the ORIGINAL config still serving (quarantine + blogs both return the bind's 403), so no…

spikerj commented on issue spikerj/spikersoft-issues#542 2026-07-13 20:47:09 +00:00
[Security][Prod] nginx root bind publicly serves quarantine, raw uploads, dlls — /spikersoft/<private-tree> downloadable

Fix (#177) merged; deploy pending in the Frontend CI queue (old config still live — quarantine currently 403 = bind directory). Verify after deploy: curl /spikersoft/quarantine/ -> 404, and a…

spikerj commented on issue spikerj/spikersoft-issues#531 2026-07-13 20:44:12 +00:00
[Infra][MinIO] Migrate Angular nginx root bind — presigned/proxy, LAST (epic #413)

HOLD: #178 merged ahead of its gates — verified live that minio.spikersoft.com/blogs/ returns AccessDenied (no anonymous read) and the old config is still serving (quarantine 403s = bind…

spikerj commented on issue spikerj/spikersoft-issues#541 2026-07-13 20:42:56 +00:00
[Bug][Backend][DockerMonitor] All swarm node events published as 'docker.node.unknown' with empty hostname — ~21 junk RabbitMQ msgs/min for healthy nodes

QA watch 2026-07-13 ~20:44Z — fix VERIFIED in master, not yet live. f0b846a0 lands all three recommendations: Message.Action-first verb resolution, hostname from Actor.Attributes['name'], and…

spikerj commented on issue spikerj/spikersoft-issues#534 2026-07-13 20:40:46 +00:00
[CI][ArtPipe] Route baked model-image builds (ModelEnvImages + tier-3 finals) to the 4090 runner, off laptop-server

Both halves now authored: ModelEnvImages → 4090 was artpipe PR #16 (merged); backend tier-3 finals → 4090 is backend PR #266 (open). With both on the same exclusive runner, tier-2 and tier-3…

spikerj commented on issue spikerj/spikersoft-issues#531 2026-07-13 20:39:43 +00:00
[Infra][MinIO] Migrate Angular nginx root bind — presigned/proxy, LAST (epic #413)

SECURITY — deploy needed, 2026-07-13 ~20:42Z QA watch. The nginx deny fix (angular 10f80c18, merged 20:29Z in PR #177) is NOT live yet: learn.spikersoft.com still runs the pre-fix config. Live…

spikerj commented on issue spikerj/spikersoft-issues#536 2026-07-13 20:39:18 +00:00
[CI][ArtPipe] ModelEnvImages has no concurrency guard — concurrent runs race the same :latest tags (observed live)

artpipe PR #18: workflow-level concurrency group with cancel-in-progress (ticket fix option 1) — newest commit owns :latest + provenance; superseded runs cancel instead of clobbering. Option 2…

spikerj commented on issue spikerj/spikersoft-issues#541 2026-07-13 20:36:32 +00:00
[Bug][Backend][DockerMonitor] All swarm node events published as 'docker.node.unknown' with empty hostname — ~21 junk RabbitMQ msgs/min for healthy nodes

Backend PR #265: Action-first verb mapping, node-only name-attr hostname fallback (container names can't masquerade as hostnames), and heartbeat debounce — node updates without a state transition…

spikerj commented on issue spikerj/spikersoft-issues#521 2026-07-13 20:33:05 +00:00
[Infra][Registry] Gitea upstream-image mirror (skopeo → mirror/* org) + incremental stack repointing

Hardening MERGED (infra #59 — --preserve-digests + optional authed Hub pulls). Mirror is still EMPTY: re-dispatch MirrorUpstreamImages once the artpipe pushes finish (optionally set DOCKERHUB_USER…

spikerj commented on issue spikerj/spikersoft-issues#493 2026-07-13 20:33:04 +00:00
[Infra][MinIO] Pilot: deploy MinIO + migrate security-scanner off /mnt/fusionio to S3 (proves the pattern for #413)

Pilot status: MinIO deployed+hardened (infra #54/#55), scanner dual-read code live since the flag flip, recipe proven and generalized into SpikerSoft.Storage (#533) — every later migration…

spikerj commented on issue spikerj/spikersoft-issues#514 2026-07-13 20:33:04 +00:00
[Infra][CI] Gitea Actions leaves GITEA-ACTION* volumes behind → build nodes recurrently fill up (need runner auto-cleanup / scheduled prune)

Prune hardening MERGED (infra #58): size-capped hourly prune + emergency full-cache drop + exit-2 ALERT. Remaining: re-run install.sh on each build node (wait for in-flight artpipe pushes on…

spikerj commented on issue spikerj/spikersoft-issues#413 2026-07-13 20:33:04 +00:00
[Infra][Architecture] Provider-agnostic object storage (MinIO -> S3/Azure Blob) to replace GlusterFS/host-path model

EPIC SCOREBOARD (2026-07-13, after 19 loop iterations): every child ticket is code-complete. MERGED: #533 seam (closed), #530 GeoIP, #540 scheduler, #526 quarantine, #524 dlls, #525 assets, #522…

spikerj commented on issue spikerj/spikersoft-issues#528 2026-07-13 20:33:04 +00:00
[Backend][MinIO] Replace API static-file media serving with S3-backed delivery (epic #413)

Code MERGED (backend #263 — S3MediaFallbackMiddleware behind the four disk mounts, Range pass-through). Live but inert until the API stack runs Storage__UseS3=true (infra #56 merged the env;…

spikerj commented on issue spikerj/spikersoft-issues#527 2026-07-13 20:33:03 +00:00
[Infra][MinIO] Migrate /mnt/fusionio/spikersoft/ai → ai-models bucket for embeddings + quiz-generation (epic #413)

Code MERGED (backend #261 — ObjectFileCache + embeddings direct + quiz via ModelPathResolver delegate). Remaining = ops: ai-models bucket + per-service ro keys, mc mirror /mnt/fusionio/spikersoft/…

spikerj commented on issue spikerj/spikersoft-issues#523 2026-07-13 20:32:40 +00:00
[Infra][MinIO] Migrate lesson-videos + LessonVideoProcessor to S3 (lesson-videos bucket, epic #413)

Code MERGED (backend #262). Remaining = ops via #529 (lesson-videos bucket + videos-svc key + VIDEOS_S3_SECRET_KEY — backend #264/infra #60 — backfill, phase-2 bind drop). S3-temp transcode…