Registry blob corruption: artpipe-model-env-safetycheck 4.8 GB weights layer truncated in MinIO (unexpected EOF) #821

New Issue

2026-07-23T23:52:58Z

spikerj commented

2026-07-23 23:52:58 +00:00

Symptom

Every docker pull git.spikersoft.com/spikerj/artpipe-model-env-safetycheck:latest dies mid-download of layer sha256:f3d9300873ec9cef90715060dec532074b4a4d50ce949293db1c49e673640fc4 with unexpected EOF, after ~10 min per attempt. This killed backend ArtPipeProcessor master runs 17872 (both attempts, 2026-07-23) once the env image was no longer in the 4090's local docker cache.

Root cause — corrupt stored object, not the network

Verified by reading the blob at every hop:

Through the full chain (Traefik -> Gitea -> MinIO), from the 4090 over the LAN: HTTP 200, stream dies at 4,614,782,976 of 5,184,057,166 bytes.
Directly from MinIO (mc cat against minio:9000 on the overlay, bypassing Gitea and Traefik entirely): dies at exactly the same byte offset with unexpected EOF.

mc stat metadata claims the full 4.8 GiB (multipart upload, 309 parts, ETag 260d13fb63affd958ad08c78127fb16e-309, stored 2026-07-22 05:10 UTC by env-image run 17775) — but the object's tail cannot be read back. I.e. the multipart object in the gitea-packages bucket is missing/corrupt past ~byte 4.6 G. Possibly related to SERVER's root-LV disk-full episode / MinIO data-root cutover.

NAT-hairpin (#538) made pulls slow and flaky, but is NOT the cause here — the LAN path fails identically at the same offset.

Why a plain rebuild doesn't fix it

Pull retries can never succeed (the registry cannot serve the bytes).
A cached rebuild reuses the same layer digest; docker push then sees "layer already exists" (Gitea's blob dedup is DB-backed) and skips the upload — the corrupt blob stays authoritative.
BuildKit --cache-from env:latest is poisoned too: cache-hit layers must be materialized by pulling the corrupt blob (observed live in artpipe run 17905, which spun on pull-retry loops).

Fix

model-env-images.yml gains a no_cache workflow_dispatch input: builds with --no-cache (and without --cache-from), so the weights layer is regenerated with new content -> new digest -> fresh blob upload; the :latest manifest then references the healthy blob. Corrupt blob becomes unreferenced garbage for the package prune (gitea/cleanup/prune-container-packages.py) to age out.
Dispatched with images: safetycheck, no_cache: true.
Mitigation for the hairpin while at it: the 4090's /etc/hosts now pins git.spikersoft.com to its LAN address, so multi-GB registry transfers on the build node no longer traverse the router (#538).

Follow-ups

Verify remaining env images' big blobs are readable (prodstages 15.7 GB + 17.2 GB, photostack 289 MB) — check running as of this writing.
Consider a post-push read-back verification step in the env workflow (HEAD is not enough — needs a ranged GET of the blob tail) to catch truncated multipart uploads at push time.
Audit other buckets/objects written around the disk-full window for the same truncation.

## Symptom Every `docker pull git.spikersoft.com/spikerj/artpipe-model-env-safetycheck:latest` dies mid-download of layer `sha256:f3d9300873ec9cef90715060dec532074b4a4d50ce949293db1c49e673640fc4` with `unexpected EOF`, after ~10 min per attempt. This killed backend ArtPipeProcessor master runs 17872 (both attempts, 2026-07-23) once the env image was no longer in the 4090's local docker cache. ## Root cause — corrupt stored object, not the network Verified by reading the blob at every hop: 1. Through the full chain (Traefik -> Gitea -> MinIO), from the 4090 over the LAN: HTTP 200, stream dies at **4,614,782,976 of 5,184,057,166 bytes**. 2. Directly from MinIO (`mc cat` against `minio:9000` on the overlay, bypassing Gitea and Traefik entirely): dies at **exactly the same byte offset** with `unexpected EOF`. `mc stat` metadata claims the full 4.8 GiB (multipart upload, 309 parts, `ETag 260d13fb63affd958ad08c78127fb16e-309`, stored 2026-07-22 05:10 UTC by env-image run 17775) — but the object's tail cannot be read back. I.e. the multipart object in the `gitea-packages` bucket is missing/corrupt past ~byte 4.6 G. Possibly related to SERVER's root-LV disk-full episode / MinIO data-root cutover. NAT-hairpin (#538) made pulls slow and flaky, but is NOT the cause here — the LAN path fails identically at the same offset. ## Why a plain rebuild doesn't fix it - Pull retries can never succeed (the registry cannot serve the bytes). - A cached rebuild reuses the same layer digest; `docker push` then sees "layer already exists" (Gitea's blob dedup is DB-backed) and skips the upload — the corrupt blob stays authoritative. - BuildKit `--cache-from env:latest` is poisoned too: cache-hit layers must be materialized by pulling the corrupt blob (observed live in artpipe run 17905, which spun on pull-retry loops). ## Fix 1. `model-env-images.yml` gains a `no_cache` workflow_dispatch input: builds with `--no-cache` (and without `--cache-from`), so the weights layer is regenerated with new content -> new digest -> fresh blob upload; the `:latest` manifest then references the healthy blob. Corrupt blob becomes unreferenced garbage for the package prune (`gitea/cleanup/prune-container-packages.py`) to age out. 2. Dispatched with `images: safetycheck`, `no_cache: true`. 3. Mitigation for the hairpin while at it: the 4090's `/etc/hosts` now pins `git.spikersoft.com` to its LAN address, so multi-GB registry transfers on the build node no longer traverse the router (#538). ## Follow-ups - [ ] Verify remaining env images' big blobs are readable (prodstages 15.7 GB + 17.2 GB, photostack 289 MB) — check running as of this writing. - [ ] Consider a post-push read-back verification step in the env workflow (HEAD is not enough — needs a ranged GET of the blob tail) to catch truncated multipart uploads at push time. - [ ] Audit other buckets/objects written around the disk-full window for the same truncation.

spikerj commented

2026-07-23 23:57:14 +00:00

Follow-up verification done — the damage is wider than safetycheck. Read every >100 MB env-image layer end-to-end from MinIO (job pinned on SERVER, local to the data):

Layer	Image	Declared	Readable	Verdict
`f3d9300873ec`	env-safetycheck	5,184,057,166	4,614,782,976	CORRUPT
`f45c1c37c08f`	env-prodstages	15,754,496,814	8,115,978,240	CORRUPT
`5652590022e6`	env-prodstages	17,250,041,346	342,884,352	CORRUPT
`2d717c997605`	env-photostack	289,166,548	289,166,548	healthy

Every giant multipart object pushed by env run 17775 (2026-07-22) is truncated; the small one survived. All shared base layers (<3.3 GB) read fine during pulls.

Remediation in flight: workflow commit 6c4fade added the no_cache input; dispatched run 17908 (images: "safetycheck prodstages", no_cache: true) — full rebuild, fresh digests, fresh uploads. photostack is untouched (healthy). The cascade will rebuild the tier-3 worker images afterward.

Follow-up verification done — the damage is wider than safetycheck. Read every >100 MB env-image layer end-to-end from MinIO (job pinned on SERVER, local to the data): | Layer | Image | Declared | Readable | Verdict | |---|---|---|---|---| | `f3d9300873ec` | env-safetycheck | 5,184,057,166 | 4,614,782,976 | CORRUPT | | `f45c1c37c08f` | env-prodstages | 15,754,496,814 | 8,115,978,240 | CORRUPT | | `5652590022e6` | env-prodstages | 17,250,041,346 | 342,884,352 | CORRUPT | | `2d717c997605` | env-photostack | 289,166,548 | 289,166,548 | healthy | Every giant multipart object pushed by env run 17775 (2026-07-22) is truncated; the small one survived. All shared base layers (<3.3 GB) read fine during pulls. Remediation in flight: workflow commit 6c4fade added the `no_cache` input; dispatched run 17908 (`images: "safetycheck prodstages"`, `no_cache: true`) — full rebuild, fresh digests, fresh uploads. photostack is untouched (healthy). The cascade will rebuild the tier-3 worker images afterward.

spikerj referenced this issue

2026-07-29 16:06:40 +00:00

[Bug][Infra][Gitea/MinIO] Large registry push restarts single-node MinIO → Gitea storage.Init crash-loops → full git+CI 404 outage #700

spikerj referenced this issue

2026-07-29 16:56:50 +00:00

artpipe env images: two RUN loops produce 17 GB and 16 GB single layers — split per model #858

spikerj referenced this issue

2026-07-29 18:26:03 +00:00

[Epic][ArtPipe][Infra] Per-model baked Docker images — venvs + weights built at image build time, no /opt/art_pipe bind #515

Sign in to join this conversation.