[Infra][Data-integrity] Corrupt blob in MinIO-backed container registry — 17GB env-image layer unreadable at one part, blocked the #702 fix chain all night #720

New Issue

2026-07-19T03:50:03Z

spikerj commented

2026-07-19 03:50:03 +00:00

Found 2026-07-19 ~04:30Z while delivering the #702 modeling fix. The artpipe-model-env-prodstages:latest layer blob sha256:64da58ddeb39... (17.25 GB, multipart ETag ...-1029, uploaded 22:29Z during the heavily-loaded window) is corrupt in MinIO:

Reads consistently die at absolute offset 6,255,804,416 regardless of client/path (docker pull ×5, curl LAN-direct ×40, mc cat full + chunked ×8 — all identical failure offset).
Ranged probes (stderr-separated): offsets 6255804416 / 6255804417 → Unable to read; offsets 7GB/12GB/17GB → clean 1MB reads. Exactly one ~16MB part inside [6.25GB, 7GB) is unreadable; everything else is intact.
A gzip layer with a hole is unrecoverable → the pushed env image was never actually pullable, compounding #717 (cancelled builds) into a fully wedged fix chain.

Context/risk: MinIO is single-node with no erasure parity (accepted for now in #696), so there is no mc admin heal path — one bad part = permanent object loss. This is on SERVER (the #503 bad-RAM / storage-gremlin machine; data recently moved to its NVMe). The push at 22:29Z succeeded from the client's perspective — corruption happened at/after write.

Remediation done: re-dispatched model-env-images (images: prodstages) at ~04:30Z on idle runners — a fresh build re-pushes new blob digests, and the cascade auto-runs tier-3 + deploy.

Recommendations:

Periodic scrub job for the registry-critical buckets (gitea-packages): ranged sweep or mc support object verification; alert on unreadable parts — tonight this was only discovered because a deploy chain died.
Revisit single-node/no-parity for gitea-packages specifically (registry blobs are the supply chain for every deploy) — even 2-drive mirroring or scheduled replication to a second node would make this healable.
CI's manifest-pullable gate (#511) checks the MANIFEST only — consider a post-push ranged read of the largest blob (cheap: 3 ranged GETs) to catch this class at push time.

Related: #503 (SERVER hardware history), #696 (single-node decision), #717 (the build-cancellation half of tonight's wedge), #702 (the stranded fix).

**Found 2026-07-19 ~04:30Z** while delivering the #702 modeling fix. The `artpipe-model-env-prodstages:latest` layer blob `sha256:64da58ddeb39...` (17.25 GB, multipart ETag `...-1029`, uploaded 22:29Z during the heavily-loaded window) is **corrupt in MinIO**: - Reads consistently die at absolute offset **6,255,804,416** regardless of client/path (docker pull ×5, curl LAN-direct ×40, `mc cat` full + chunked ×8 — all identical failure offset). - Ranged probes (stderr-separated): offsets `6255804416` / `6255804417` → **Unable to read**; offsets 7GB/12GB/17GB → clean 1MB reads. Exactly one ~16MB part inside `[6.25GB, 7GB)` is unreadable; everything else is intact. - A gzip layer with a hole is unrecoverable → the pushed env image was never actually pullable, compounding #717 (cancelled builds) into a fully wedged fix chain. **Context/risk**: MinIO is single-node with no erasure parity (accepted for now in #696), so there is no `mc admin heal` path — one bad part = permanent object loss. This is on SERVER (the #503 bad-RAM / storage-gremlin machine; data recently moved to its NVMe). The push at 22:29Z *succeeded* from the client's perspective — corruption happened at/after write. **Remediation done**: re-dispatched `model-env-images` (`images: prodstages`) at ~04:30Z on idle runners — a fresh build re-pushes new blob digests, and the cascade auto-runs tier-3 + deploy. **Recommendations**: 1. Periodic scrub job for the registry-critical buckets (`gitea-packages`): ranged sweep or `mc support` object verification; alert on unreadable parts — tonight this was only discovered because a deploy chain died. 2. Revisit single-node/no-parity for `gitea-packages` specifically (registry blobs are the supply chain for every deploy) — even 2-drive mirroring or scheduled replication to a second node would make this healable. 3. CI's manifest-pullable gate (#511) checks the MANIFEST only — consider a post-push ranged read of the largest blob (cheap: 3 ranged GETs) to catch this class at push time. Related: #503 (SERVER hardware history), #696 (single-node decision), #717 (the build-cancellation half of tonight's wedge), #702 (the stranded fix).

spikerj referenced this issue

2026-07-19 03:50:21 +00:00

[ArtPipe][Bug] Modeling stage fails offline — TripoSR's DINO image encoder (facebook/dino-vitb16) never baked (same class as #699) #702

spikerj commented

2026-07-19 07:46:30 +00:00

Rebuild verified (2026-07-19 ~06:5xZ): the re-dispatched env build succeeded and pushed a NEW 17.3GB layer (sha256:f659ac2f3c6a...). Ranged-read scrub via internal MinIO at offsets 0/3G/6255804416(the old corpse's exact dead byte)/9G/12G/16G/17G — all clean 1MB reads. Recommendation 3 from this ticket demonstrated in practice: 7 ranged GETs took seconds and definitively separates a healthy push from tonight's corrupt one. The cascade tier-3 build is queued; #702's fix finally has a viable path to prod.

spikerj referenced this issue

2026-07-19 08:00:49 +00:00

[ArtPipe][Bug] Modeling stage fails offline — TripoSR's DINO image encoder (facebook/dino-vitb16) never baked (same class as #699) #702

spikerj commented

2026-07-21 02:51:41 +00:00

New evidence of the same failure class, 2026-07-20 (swarm-recon sweep from the 4090 node):

Docker daemon journal on 4090 shows a large-layer pull repeatedly failing in a 5-attempt retry loop, alternating unexpected EOF and expected HTTP 206 from byte range request, for layer sha256:24d8c17743a0bf2b9802862b4d3223290324cd6b1199eb18a27dec45af86e322 (a different blob than the original 64da58ddeb39...).

Two bursts: 15:41–16:02 UTC and 19:54–20:38 UTC on 2026-07-20; 15 terminal "failed after 5 attempts" errors total (~one every 4–5 min).
The loop stopped at 20:38:14 — immediately after the 20:30–20:55 fleet redeploy began (consistent with a fresh build pushing new blob digests so the bad one stops being requested, same pattern as the #720 remediation).
Not recurred in the ~6h since; the journal does not name the image, but the pattern (huge layer, byte-range reads from git.spikersoft.com's MinIO-backed registry) matches this ticket exactly.

This makes it two distinct blobs exhibiting mid-object read failure within ~36h, which strengthens the case for the periodic scrub job on gitea-packages recommended above — the second occurrence was only detected by accident during a log sweep.

New evidence of the same failure class, 2026-07-20 (swarm-recon sweep from the 4090 node): Docker daemon journal on **4090** shows a large-layer pull repeatedly failing in a 5-attempt retry loop, alternating `unexpected EOF` and `expected HTTP 206 from byte range request`, for layer `sha256:24d8c17743a0bf2b9802862b4d3223290324cd6b1199eb18a27dec45af86e322` (a **different** blob than the original `64da58ddeb39...`). - Two bursts: **15:41–16:02 UTC** and **19:54–20:38 UTC** on 2026-07-20; 15 terminal "failed after 5 attempts" errors total (~one every 4–5 min). - The loop stopped at **20:38:14** — immediately after the 20:30–20:55 fleet redeploy began (consistent with a fresh build pushing new blob digests so the bad one stops being requested, same pattern as the #720 remediation). - Not recurred in the ~6h since; the journal does not name the image, but the pattern (huge layer, byte-range reads from git.spikersoft.com's MinIO-backed registry) matches this ticket exactly. This makes it two distinct blobs exhibiting mid-object read failure within ~36h, which strengthens the case for the periodic scrub job on `gitea-packages` recommended above — the second occurrence was only detected by accident during a log sweep.

spikerj commented

2026-07-21 19:38:42 +00:00

Superseded/resolved: this blob (sha256:64da58…, 07-19) was replaced by the 07-19 13:58 ModelEnvImages rebuild — whose OWN 17GB layer then corrupted the same way (#768, 07-21). Today's rebuild produced layer c84d8ce…, which was full-stream SHA-256 read-back verified and is serving builds (deploy chain green). Systemic follow-ups (two corruption incidents in 3 days) consolidated into #776. Closing.