Recurrence 2026-07-13 ~02:50Z (user-confirmed crash; node rejoined by ~02:53Z, recovery wave in progress — minio/backend/mongo-router/docker-monitor all restarting through the usual 'swarm…
MinIO down again, new failure mode (2026-07-13 ~02:31Z) — and it takes the container registry with it.
minio_miniotask exited cleanly (state Complete, exit 0) at ~02:31Z and swarm…
Phase 2 fully staged (2026-07-13), merge in this order:
- spikersoft-artpipe PR #14 — extends the ModelEnvImages default set (+blender +sdxllightning +triposr; hunyuan3dpaint stays…
DEPLOYED — soak started 2026-07-13 ~01:51 UTC. PR #52 merged, /mnt/infrastructure synced, stack redeployed. Verified from the manager:
- Service 1/1 Running on the 4090 — a node with NO…
Design correction (owner feedback 2026-07-13): no GridFS for QR artifacts — store in MinIO.
The integration doc above says 'artifacts up to GridFS (same as art_pipe stages)' — superseded:…
Safety-lane half of this is CLEARED (2026-07-13 01:51Z): the #517 baked-image cutover deployed and spikersoft-artpipe-model-safety is 1/1 Running on the 4090 with no /mnt/fusionio/spikersoft/art_pi…
Cutover deployed + verified live (2026-07-13 01:51Z). spikersoft-artpipe-model-safety now runs artpipe-model-safetycheck:latest on the 4090 — 1/1 Running (the reject-loop stopped at…
Pilot env image verified on SERVER (2026-07-13, fresh pull — gate 2 of infra PR #52 cleared for the env tier):
artpipe-base:latest+artpipe-model-env-safetycheck:{latest,5f0abb6}built…
Gate SATISFIED (2026-07-13 01:38Z) — clear to cut over:
- ModelEnvImages run 10768 succeeded (built on the 4090 runner; secret fix c27b8dd made the cascade work):
artpipe-base(12.7 GB) +…
Hard evidence for this routing change (2026-07-13 ~02:00Z): the ModelEnvImages retry (task 10766) failed again on the laptop runner — ~2 min in, inside 'Build + push artpipe-base', with NO outage…
Disk go/no-go: GO (measured on SERVER 2026-07-12):
Build-chain status (2026-07-13 ~01:35Z): tier 3 is red because tier 2 never published.
- backend ArtPipeProcessor run for the tier-3 merge (79482d8f, 'Dockerfile.model + CI finals loop')…
Deploy-gate status check (2026-07-13 ~01:25Z): gate NOT satisfied — hold the /mnt/infrastructure sync.
docker manifest inspect git.spikersoft.com/spikerj/artpipe-model-safetycheck:latest…
Pilot cutover PR is staged and GATED: spikersoft-infrastructure PR #52 (stack file: baked image, bind removed, HF_HUB_OFFLINE=1). Merge gates in the PR body — image must be verified pullable…
Phase 0 scaffolding landed/opened (2026-07-12):
- spikersoft-artpipe PR #11 (MERGED): setup --no-weights/--weights-only flags, .dockerignore, docker/Dockerfile.base + Dockerfile.model-env,…
New fallout data point (2026-07-13, ~00:50 local): SERVER's log RPC endpoint is dead while the node reports Ready and its tasks run. docker service logs for ANY SERVER-hosted task (minio_minio,…
Status 2026-07-13: the restart-policy fix (8a8889e, condition: any) IS deployed — jaeger_elasticsearch now retries forever instead of exhausting MaxAttempts. But the underlying failure persists:…