Joseph Spiker spikerj
  • Joined on 2025-11-03
spikerj commented on issue spikerj/spikersoft-issues#534 2026-07-13 01:20:58 +00:00
[CI][ArtPipe] Route baked model-image builds (ModelEnvImages + tier-3 finals) to the 4090 runner, off laptop-server

Hard evidence for this routing change (2026-07-13 ~02:00Z): the ModelEnvImages retry (task 10766) failed again on the laptop runner — ~2 min in, inside 'Build + push artpipe-base', with NO outage…

spikerj commented on issue spikerj/spikersoft-issues#516 2026-07-13 01:20:57 +00:00
[ArtPipe] Baked images Phase 0: build scaffolding (Dockerfiles, CI, setup flags) + disk/limits go-no-go gates (epic #515)

Disk go/no-go: GO (measured on SERVER 2026-07-12):

/dev/fioa  2.9T total, 81G used, 2.7T avail (3%) on /mnt/fusionio
/mnt/fusionio/minio (MinIO data root incl. gitea-packages):…
spikerj opened issue spikerj/spikersoft-issues#534 2026-07-13 01:14:15 +00:00
[CI][ArtPipe] Route baked model-image builds (ModelEnvImages + tier-3 finals) to the 4090 runner, off laptop-server
spikerj commented on issue spikerj/spikersoft-issues#516 2026-07-13 01:09:23 +00:00
[ArtPipe] Baked images Phase 0: build scaffolding (Dockerfiles, CI, setup flags) + disk/limits go-no-go gates (epic #515)

Build-chain status (2026-07-13 ~01:35Z): tier 3 is red because tier 2 never published.

  • backend ArtPipeProcessor run for the tier-3 merge (79482d8f, 'Dockerfile.model + CI finals loop')…
spikerj commented on issue spikerj/spikersoft-issues#517 2026-07-13 01:06:52 +00:00
[ArtPipe] Baked images Phase 1: SafetyCheck pilot image + cutover, incl. #486 fault isolation (epic #515)

Deploy-gate status check (2026-07-13 ~01:25Z): gate NOT satisfied — hold the /mnt/infrastructure sync.

  • docker manifest inspect git.spikersoft.com/spikerj/artpipe-model-safetycheck:latest
spikerj commented on issue spikerj/spikersoft-issues#517 2026-07-13 01:00:09 +00:00
[ArtPipe] Baked images Phase 1: SafetyCheck pilot image + cutover, incl. #486 fault isolation (epic #515)

Pilot cutover PR is staged and GATED: spikersoft-infrastructure PR #52 (stack file: baked image, bind removed, HF_HUB_OFFLINE=1). Merge gates in the PR body — image must be verified pullable…

spikerj commented on issue spikerj/spikersoft-issues#516 2026-07-13 00:59:55 +00:00
[ArtPipe] Baked images Phase 0: build scaffolding (Dockerfiles, CI, setup flags) + disk/limits go-no-go gates (epic #515)

Phase 0 scaffolding landed/opened (2026-07-12):

  • spikersoft-artpipe PR #11 (MERGED): setup --no-weights/--weights-only flags, .dockerignore, docker/Dockerfile.base + Dockerfile.model-env,…
spikerj commented on issue spikerj/spikersoft-issues#503 2026-07-13 00:52:48 +00:00
[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services

New fallout data point (2026-07-13, ~00:50 local): SERVER's log RPC endpoint is dead while the node reports Ready and its tasks run. docker service logs for ANY SERVER-hosted task (minio_minio,…

spikerj commented on issue spikerj/spikersoft-issues#481 2026-07-13 00:52:35 +00:00
[Bug][Infra] jaeger_elasticsearch down ~24h on SERVER (exit 1) — MaxAttempts exhausted, tracing backend offline

Status 2026-07-13: the restart-policy fix (8a8889e, condition: any) IS deployed — jaeger_elasticsearch now retries forever instead of exhausting MaxAttempts. But the underlying failure persists:…

spikerj reopened issue spikerj/spikersoft-issues#479 2026-07-13 00:52:05 +00:00
[Bug][Infra] sonarqube_backups stuck in reject-loop — no Sonar backups running (missing placement constraint)
spikerj commented on issue spikerj/spikersoft-issues#479 2026-07-13 00:52:05 +00:00
[Bug][Infra] sonarqube_backups stuck in reject-loop — no Sonar backups running (missing placement constraint)

Reopening — the fix (198ad66, pin backups to SERVER) is merged to master but the sonarqube stack was NOT redeployed. Evidence right now (2026-07-13):

  • Deployed service UpdatedAt: 2026-07-12…
spikerj commented on issue spikerj/spikersoft-issues#493 2026-07-13 00:41:03 +00:00
[Infra][MinIO] Pilot: deploy MinIO + migrate security-scanner off /mnt/fusionio to S3 (proves the pattern for #413)

Status check while filing the #413 child tree (2026-07-12): metadata-extractor is already dual-running in prod (Storage__UseS3=true + metadata-svc key in its stack file), and the S3 code paths…

spikerj commented on issue spikerj/spikersoft-issues#413 2026-07-13 00:41:02 +00:00
[Infra][Architecture] Provider-agnostic object storage (MinIO -> S3/Azure Blob) to replace GlusterFS/host-path model

Child tickets filed for the remaining per-service migrations (2026-07-12), sequenced per docs/minio-storage-migration.md — standalone dirs first, uploads tree as one batch:

  • #522 blogs · #523…
spikerj opened issue spikerj/spikersoft-issues#533 2026-07-13 00:40:43 +00:00
[Backend][Refactor] Shared IObjectStore/S3 registration — dedupe per-service AmazonS3Client blocks (epic #413)
spikerj opened issue spikerj/spikersoft-issues#532 2026-07-13 00:40:43 +00:00
[Infra][MinIO] Optional: replace sftp stack with MinIO SFTP mode (epic #413)
spikerj opened issue spikerj/spikersoft-issues#530 2026-07-13 00:40:42 +00:00
[Backend][MinIO] GeoLite2 → reference-data bucket; self-provisioning GeoIP (epic #413, resolves GeoIP half of #489)
spikerj opened issue spikerj/spikersoft-issues#531 2026-07-13 00:40:42 +00:00
[Infra][MinIO] Migrate Angular nginx root bind — presigned/proxy, LAST (epic #413)
spikerj opened issue spikerj/spikersoft-issues#528 2026-07-13 00:40:41 +00:00
[Backend][MinIO] Replace API static-file media serving with S3-backed delivery (epic #413)
spikerj opened issue spikerj/spikersoft-issues#529 2026-07-13 00:40:41 +00:00
[Infra][MinIO] Uploads-tree coordinated batch cutover — 7 services off /mnt/fusionio (epic #413)
spikerj opened issue spikerj/spikersoft-issues#526 2026-07-13 00:40:41 +00:00
[Infra][MinIO] Migrate quarantine dir to S3 (quarantine bucket, epic #413)