spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra] Corrupt 17GB registry blob in MinIO gitea-packages blocks all artpipe prodstages CI builds (unexpected EOF at fixed offset) #768

New Issue
Closed
opened 2026-07-21 05:37:44 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-21 05:37:44 +00:00
Owner
Copy Link
Copy Source

Symptom: spikersoft-backend ArtPipeProcessor build_and_publish (run 11507 / job 41850, triggered by the #767 messaging merge but unrelated to it) fails all 5 env-image pull attempts with unexpected EOF on layer f88a4842bed8 of artpipe-model-env-prodstages:latest. create_manifest failed downstream. Every future master push touching the artpipe paths will fail the same way.

Diagnosis (2026-07-21 ~05:30 UTC):

  • MinIO and Gitea are both healthy — no crash-loop, no restarts since 07-18 18:30, disk 11% used. This is NOT a repeat of #700's outage mode.
  • The failing layer blob is 17,250,041,544 bytes. Direct curl of the blob dies at byte ~534,770,000 (±2 KB) on every attempt regardless of elapsed time (43–65s) → deterministic offset, not a timeout.
  • Reading the object directly from MinIO (mc cat on SERVER against gitea-packages/packages/f8/8a/f88a4842…) dies at the same offset: 534,773,760 bytes = exactly inside part.32 of the 1029-part multipart object (16 MiB parts). mc stat reports the full 16 GiB; all 1029 part files exist on disk (16.8 GB). MinIO aborts the stream at part 32 → integrity/bitrot verification failure on that part. Object was pushed 2026-07-19 15:49 UTC while MinIO was stably up, so the part was either written bad or rotted at rest on the fioa drive.
  • Compounding Gitea defect: the container registry blob endpoint ignores Range headers (returns 200 from byte 0, not 206), so docker's layer-resume retries always restart from zero and hit the same wall — retries can never help.

Impact: artpipe-model-env-prodstages:latest is unpullable by any node that doesn't already have the layer (no build node has it cached — 4090/laptop-server/SERVER all checked). Tier-3 prodstages builds are blocked until the env image is republished.

Remediation:

  1. Re-run spikersoft-artpipe → ModelEnvImages via workflow_dispatch with images: prodstages (its designed rebuild path; it cascade-dispatches the backend ArtPipeProcessor workflow on success). Fresh layers get new digests, so the corrupt blob is simply bypassed; it becomes orphaned garbage for Gitea's package cleanup.
  2. Afterwards re-run the failed backend job (or let the next master push do it).
  3. Consider a sweep of other recent large gitea-packages blobs for the same silent corruption (read-verify, not just stat).

Systemic notes: third MinIO-single-node incident class (#700 Gitea registry hard-depends on MinIO; #696 MinIO SPOF discussion; #413 storage epic) — single-drive MinIO detects bitrot but cannot heal it. Erasure-coded multi-drive/node MinIO would have healed this transparently.

**Symptom:** spikersoft-backend ArtPipeProcessor `build_and_publish` (run 11507 / job 41850, triggered by the #767 messaging merge but unrelated to it) fails all 5 env-image pull attempts with `unexpected EOF` on layer `f88a4842bed8` of `artpipe-model-env-prodstages:latest`. `create_manifest` failed downstream. Every future master push touching the artpipe paths will fail the same way. **Diagnosis (2026-07-21 ~05:30 UTC):** - MinIO and Gitea are both healthy — no crash-loop, no restarts since 07-18 18:30, disk 11% used. This is NOT a repeat of #700's outage mode. - The failing layer blob is 17,250,041,544 bytes. Direct curl of the blob dies at byte ~534,770,000 (±2 KB) on every attempt regardless of elapsed time (43–65s) → deterministic offset, not a timeout. - Reading the object directly from MinIO (`mc cat` on SERVER against `gitea-packages/packages/f8/8a/f88a4842…`) dies at the same offset: 534,773,760 bytes = exactly inside part.32 of the 1029-part multipart object (16 MiB parts). `mc stat` reports the full 16 GiB; all 1029 part files exist on disk (16.8 GB). MinIO aborts the stream at part 32 → integrity/bitrot verification failure on that part. Object was pushed 2026-07-19 15:49 UTC while MinIO was stably up, so the part was either written bad or rotted at rest on the fioa drive. - **Compounding Gitea defect:** the container registry blob endpoint ignores `Range` headers (returns 200 from byte 0, not 206), so docker's layer-resume retries always restart from zero and hit the same wall — retries can never help. **Impact:** `artpipe-model-env-prodstages:latest` is unpullable by any node that doesn't already have the layer (no build node has it cached — 4090/laptop-server/SERVER all checked). Tier-3 prodstages builds are blocked until the env image is republished. **Remediation:** 1. Re-run `spikersoft-artpipe` → `ModelEnvImages` via workflow_dispatch with `images: prodstages` (its designed rebuild path; it cascade-dispatches the backend ArtPipeProcessor workflow on success). Fresh layers get new digests, so the corrupt blob is simply bypassed; it becomes orphaned garbage for Gitea's package cleanup. 2. Afterwards re-run the failed backend job (or let the next master push do it). 3. Consider a sweep of other recent large `gitea-packages` blobs for the same silent corruption (read-verify, not just stat). **Systemic notes:** third MinIO-single-node incident class (#700 Gitea registry hard-depends on MinIO; #696 MinIO SPOF discussion; #413 storage epic) — single-drive MinIO detects bitrot but cannot heal it. Erasure-coded multi-drive/node MinIO would have healed this transparently.
spikerj commented 2026-07-21 08:12:49 +00:00
Author
Owner
Copy Link
Copy Source

Resolved (2026-07-21 ~08:30 UTC). Recovery executed per remediation plan:

  1. Dispatched ModelEnvImages (images: prodstages) in spikersoft-artpipe → success. New 17.25 GB layer has fresh digest c84d8ce56a2d…, replacing corrupt f88a4842bed8… (which is now orphaned pending package cleanup). The 15.7 GB sibling layer 8ab5954… kept its digest — it was never damaged.
  2. Cascade auto-dispatched spikersoft-backend ArtPipeProcessor: build_and_publish → deploy → notify all green; rebuilt processor image deployed to the swarm.
  3. Read-verified the replacement blob: streamed all 17,250,040,045 bytes from the registry and SHA-256 of the stream matches the digest exactly — no silent corruption this time (the 07-19 push looked "successful" too, hence the explicit check).

Remaining open items from this ticket:

  • Sweep other recent large gitea-packages blobs with the same read-verify (a "successful push" is demonstrably not proof of a readable blob).
  • Gitea registry ignores Range on blob GETs (200-from-zero, never 206) — makes docker layer-resume useless against any future flaky/corrupt blob; consider upstream issue.
  • Systemic: single-drive MinIO detects bitrot but cannot heal (#413/#696/#700 context).
**Resolved (2026-07-21 ~08:30 UTC).** Recovery executed per remediation plan: 1. Dispatched `ModelEnvImages` (`images: prodstages`) in spikersoft-artpipe → success. New 17.25 GB layer has fresh digest `c84d8ce56a2d…`, replacing corrupt `f88a4842bed8…` (which is now orphaned pending package cleanup). The 15.7 GB sibling layer `8ab5954…` kept its digest — it was never damaged. 2. Cascade auto-dispatched spikersoft-backend ArtPipeProcessor: build_and_publish → deploy → notify all green; rebuilt processor image deployed to the swarm. 3. **Read-verified the replacement blob**: streamed all 17,250,040,045 bytes from the registry and SHA-256 of the stream matches the digest exactly — no silent corruption this time (the 07-19 push looked "successful" too, hence the explicit check). Remaining open items from this ticket: - Sweep other recent large `gitea-packages` blobs with the same read-verify (a "successful push" is demonstrably not proof of a readable blob). - Gitea registry ignores `Range` on blob GETs (200-from-zero, never 206) — makes docker layer-resume useless against any future flaky/corrupt blob; consider upstream issue. - Systemic: single-drive MinIO detects bitrot but cannot heal (#413/#696/#700 context).
spikerj commented 2026-07-21 19:38:42 +00:00
Author
Owner
Copy Link
Copy Source

Incident resolved same-day (rebuild + cascade + full read-back verification — see resolution comment above). Systemic follow-ups consolidated with #720's into #776 now that this is a RECURRING class (2 incidents in 3 days). Closing the incident ticket.

Incident resolved same-day (rebuild + cascade + full read-back verification — see resolution comment above). Systemic follow-ups consolidated with #720's into #776 now that this is a RECURRING class (2 incidents in 3 days). Closing the incident ticket.
spikerj commented 2026-07-21 19:38:55 +00:00
Author
Owner
Copy Link
Copy Source

Correction: the consolidated hardening ticket is #775 (not #776).

Correction: the consolidated hardening ticket is #775 (not #776).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#768