Third failure mode confirmed — 98 GB pushes don't just fail, they HANG (2026-07-13 06:15Z). Run 10784 (with the 576b4bd cache sweep but NOT the snapshot_download filter) rebuilt sdxllightning…
Escalation data point (2026-07-13 ~05:14Z): tonight this bit much harder than runner-volume leftovers — leftover ModelEnvImages layers/build-cache on laptop-server (two killed runs of the 98 GB…
New wedge class found (2026-07-13 ~04:45Z), invisible to both item A and item B: after the 04:2x SERVER hard-down recovery, metadata-extractor, security-monitor and system-remediation each…
Crash #3: ~2026-07-13 03:5x UTC — roughly ONE HOUR after the reboot from crash #2 (03:51 UTC boot). Intervals are shrinking: ~8h → ~1h, again under memory/IO load (a ~20 GB registry push was in…
Recurrence #4 (2026-07-13 ~04:2xZ): HARD DOWN again — docker node ls → SERVER Down/Unreachable (user-confirmed). Same full-outage mode as 07-12 08:29: everything hostname-pinned to SERVER…
Root cause found (2026-07-13 ~03:00Z) — diagnosable at last because the SERVER crash/reboot restored the node's log RPC endpoint:
java.lang.IllegalStateException: data path [/usr/share/el…
New crash data point: SERVER went down again ~2026-07-13 03:0x UTC — second crash in ~9h (uptime was 8h21m at 02:43 UTC, so the previous boot was ~18:22 UTC 2026-07-12). Correlates with tonight's…
Recurrence 2026-07-13 ~02:50Z (user-confirmed crash; node rejoined by ~02:53Z, recovery wave in progress — minio/backend/mongo-router/docker-monitor all restarting through the usual 'swarm…
MinIO down again, new failure mode (2026-07-13 ~02:31Z) — and it takes the container registry with it.
minio_miniotask exited cleanly (state Complete, exit 0) at ~02:31Z and swarm…
Phase 2 fully staged (2026-07-13), merge in this order:
- spikersoft-artpipe PR #14 — extends the ModelEnvImages default set (+blender +sdxllightning +triposr; hunyuan3dpaint stays…
DEPLOYED — soak started 2026-07-13 ~01:51 UTC. PR #52 merged, /mnt/infrastructure synced, stack redeployed. Verified from the manager:
- Service 1/1 Running on the 4090 — a node with NO…
Design correction (owner feedback 2026-07-13): no GridFS for QR artifacts — store in MinIO.
The integration doc above says 'artifacts up to GridFS (same as art_pipe stages)' — superseded:…
Safety-lane half of this is CLEARED (2026-07-13 01:51Z): the #517 baked-image cutover deployed and spikersoft-artpipe-model-safety is 1/1 Running on the 4090 with no /mnt/fusionio/spikersoft/art_pi…
Cutover deployed + verified live (2026-07-13 01:51Z). spikersoft-artpipe-model-safety now runs artpipe-model-safetycheck:latest on the 4090 — 1/1 Running (the reject-loop stopped at…
Pilot env image verified on SERVER (2026-07-13, fresh pull — gate 2 of infra PR #52 cleared for the env tier):
artpipe-base:latest+artpipe-model-env-safetycheck:{latest,5f0abb6}built…
Gate SATISFIED (2026-07-13 01:38Z) — clear to cut over:
- ModelEnvImages run 10768 succeeded (built on the 4090 runner; secret fix c27b8dd made the cascade work):
artpipe-base(12.7 GB) +…