spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][Registry] Blob-integrity hardening: read-verify sweep + Gitea ignores Range on blob GETs + single-node MinIO cannot heal bitrot (consolidates #720/#768) #775

New Issue
Open
opened 2026-07-21 19:38:43 +00:00 by spikerj · 5 comments
spikerj commented 2026-07-21 19:38:43 +00:00
Owner
Copy Link
Copy Source

TWO large registry blobs corrupted in MinIO within 3 days (#720: 07-19, offset 6.25GB; #768: 07-21, part.32 ≈535MB — different blobs, same class: multipart artpipe env layers, integrity failure on read, deterministic offset). Both incidents are resolved (rebuilds, read-back-verified), but the class is clearly recurring. Consolidated follow-ups:

  1. Read-verify sweep of existing large gitea-packages blobs (stream + hash vs digest) — a successful push is demonstrably not proof of a readable blob. The #768 diagnosis recipe (3× curl same-offset death → mc cat) applies.
  2. Post-push read-back verification in the big-image CI workflows (ModelEnvImages, artpipe finals) — the #771/#773 data migrations already do this per-object; the image pipelines should too.
  3. Gitea defect: the container registry ignores Range headers on blob GETs (200-from-zero, never 206), so docker's layer-resume retries always restart and can never route around a bad byte. Consider an upstream issue/patch.
  4. Systemic: single-drive MinIO detects bitrot but cannot heal (#413/#696 context) — erasure-coded storage would have self-healed both incidents; the fioa drive's health is also now suspect (two corruptions, same volume) and worth a SMART/badblocks look.
TWO large registry blobs corrupted in MinIO within 3 days (#720: 07-19, offset 6.25GB; #768: 07-21, part.32 ≈535MB — different blobs, same class: multipart artpipe env layers, integrity failure on read, deterministic offset). Both incidents are resolved (rebuilds, read-back-verified), but the class is clearly recurring. Consolidated follow-ups: 1. **Read-verify sweep** of existing large `gitea-packages` blobs (stream + hash vs digest) — a successful push is demonstrably not proof of a readable blob. The #768 diagnosis recipe (3× curl same-offset death → mc cat) applies. 2. **Post-push read-back verification** in the big-image CI workflows (ModelEnvImages, artpipe finals) — the #771/#773 data migrations already do this per-object; the image pipelines should too. 3. **Gitea defect**: the container registry ignores `Range` headers on blob GETs (200-from-zero, never 206), so docker's layer-resume retries always restart and can never route around a bad byte. Consider an upstream issue/patch. 4. **Systemic**: single-drive MinIO detects bitrot but cannot heal (#413/#696 context) — erasure-coded storage would have self-healed both incidents; the fioa drive's health is also now suspect (two corruptions, same volume) and worth a SMART/badblocks look.
spikerj commented 2026-07-22 00:32:49 +00:00
Author
Owner
Copy Link
Copy Source

THIRD incident, and this one caused a full site outage: the frontend image digest 5538e2bd… (published 2026-07-21 ~22:01Z on the #510 merge) has a corrupt layer — pulls die with EOF deterministically (verified 2×). The swarm update rejected every task since 22:01 and PAUSED at 0/1 replicas → learn.spikersoft.com served Traefik 404s for ~10 HOURS, undetected because ops alerting is dead (#756). Recovered 2026-07-22 ~08:0xZ via docker service rollback (previous image cached on node; site 200 again, serving the pre-#510 build). Fresh image rebuilding via workflow_dispatch now; will redeploy on publish. This escalates both consolidation items: (a) the read-verify sweep is now urgent — 3 corrupt blobs in 4 days, and (b) deploy-convergence checking (#774) would have turned a silent 10-hour outage into a red pipeline at 22:0x. SERVER fioa drive integrity check should be treated as priority.

THIRD incident, and this one caused a full site outage: the frontend image digest 5538e2bd… (published 2026-07-21 ~22:01Z on the #510 merge) has a corrupt layer — pulls die with EOF deterministically (verified 2×). The swarm update rejected every task since 22:01 and PAUSED at 0/1 replicas → learn.spikersoft.com served Traefik 404s for ~10 HOURS, undetected because ops alerting is dead (#756). Recovered 2026-07-22 ~08:0xZ via docker service rollback (previous image cached on node; site 200 again, serving the pre-#510 build). Fresh image rebuilding via workflow_dispatch now; will redeploy on publish. This escalates both consolidation items: (a) the read-verify sweep is now urgent — 3 corrupt blobs in 4 days, and (b) deploy-convergence checking (#774) would have turned a silent 10-hour outage into a red pipeline at 22:0x. SERVER fioa drive integrity check should be treated as priority.
spikerj commented 2026-07-22 00:55:38 +00:00
Author
Owner
Copy Link
Copy Source

Frontend recovered fully: rebuilt image published (digest 73abed19…), pull-verified BEFORE deploy this time, and learn.spikersoft.com now serves the new stack-gallery build (bundle main-2HPZWHQU). Note: the dispatched pipeline's own deploy hit yet another Rejected-then-recovered on first pull (manifest propagation race) — that's now FOUR deploys in 24h showing the #774 pattern. Today's full e2e suite is green (incl. the axe a11y gate over the new gallery UI); last night's e2e-anonymous failure did not reproduce and coincided with the corrupt-publish window.

Frontend recovered fully: rebuilt image published (digest 73abed19…), pull-verified BEFORE deploy this time, and learn.spikersoft.com now serves the new stack-gallery build (bundle main-2HPZWHQU). Note: the dispatched pipeline's own deploy hit yet another Rejected-then-recovered on first pull (manifest propagation race) — that's now FOUR deploys in 24h showing the #774 pattern. Today's full e2e suite is green (incl. the axe a11y gate over the new gallery UI); last night's e2e-anonymous failure did not reproduce and coincided with the corrupt-publish window.
spikerj commented 2026-07-22 01:45:42 +00:00
Author
Owner
Copy Link
Copy Source

Timeline correction: the outage window was 2026-07-21 22:01Z → ~00:10Z 07-22 — about 2 HOURS, not the ~10 I stated earlier (I misjudged the current time when computing it). The severity points stand (undetected until a human noticed; alerting dead), but the record should be accurate.

Timeline correction: the outage window was 2026-07-21 22:01Z → ~00:10Z 07-22 — about 2 HOURS, not the ~10 I stated earlier (I misjudged the current time when computing it). The severity points stand (undetected until a human noticed; alerting dead), but the record should be accurate.
spikerj commented 2026-07-22 18:08:15 +00:00
Author
Owner
Copy Link
Copy Source

Fresh read-path data point (2026-07-22 16:36–18:03, backend run 13851): the tier-3 artpipe finals build attempted to pull artpipe-model-env-prodstages (multi-GB, freshly built at 06:45 same day) and died with unexpected EOF on all 5 retry attempts, each ~19 minutes into the transfer, ~87 minutes wasted total. This is the no-Range-resume failure this ticket names — an interrupted blob GET restarts from zero, so a blob whose transfer time exceeds the connection's stability window can never arrive. Note the retry-5x mitigation already in the workflow (the #538-era hardening) is structurally insufficient for blobs this size; only resumable reads (or a path fast enough to finish inside the stability window) fixes it. This currently hard-blocks the #460 OTel rollout's tier-3 leg.

Fresh read-path data point (2026-07-22 16:36–18:03, backend run 13851): the tier-3 artpipe finals build attempted to pull `artpipe-model-env-prodstages` (multi-GB, freshly built at 06:45 same day) and **died with `unexpected EOF` on all 5 retry attempts**, each ~19 minutes into the transfer, ~87 minutes wasted total. This is the no-Range-resume failure this ticket names — an interrupted blob GET restarts from zero, so a blob whose transfer time exceeds the connection's stability window can never arrive. Note the retry-5x mitigation already in the workflow (the #538-era hardening) is structurally insufficient for blobs this size; only resumable reads (or a path fast enough to finish inside the stability window) fixes it. This currently hard-blocks the #460 OTel rollout's tier-3 leg.
spikerj commented 2026-07-29 16:04:57 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — NOT DONE (items 1–3); item 4 unverifiable from git. Staying open, and flagging that this is now on the critical path for other work.

  • Item 1 — read-verify sweep of gitea-packages blobs: no artifact. git grep -rln "read-verify\|readverify\|blob.integrity\|verify-blob\|bitrot" origin/master across spikersoft-infrastructure → zero files.
  • Item 2 — post-push read-back verification in big-image CI: not implemented. The only post-push gate is the older #511 one at _build-deploy-service.yml:196-205 (copied into spikersoft-quiz-generation.yml:63-75), which does docker manifest inspect. Nothing streams a blob and hashes it against its digest.
  • Item 3 — Gitea ignoring Range on blob GETs: no patch, no vendored fix, no tracked upstream reference in any repo.
  • Item 4 — single-drive MinIO can't heal / fioa SMART check: git cannot settle this. Needs mc admin info / erasure-set config from the live cluster plus a SMART or badblocks report for the fioa volume.

Important, because it's easy to mistake for coverage: that docker manifest inspect gate runs on every deploy and is described in workflow comments as post-push verification. It verifies manifest queryability from the runner — not blob readability, and not pullability on the target node. So it structurally cannot catch this ticket's failure class or #774's, despite looking like it does. Both threads say this explicitly; worth not letting its presence read as protection.

This ticket is blocking measurable work now:

  • #460 (artpipe OpenTelemetry) is hard-blocked — four tier-3 dispatch attempts failed, the last on registry unexpected EOF pull failures. No trace has ever reached Jaeger as a direct result.
  • #521 (upstream image mirror) is arguably gated too: mirroring mongo/postgres/keycloak into a registry that has been observed truncating blobs would convert an upstream-availability risk into a local-corruption risk. I've noted on that ticket that #775 probably wants sequencing ahead of the critical-tier repoints.

Given item 4 (can a single-drive MinIO heal anything?) determines whether items 1–2 are detection-only or actually recoverable, the SMART/erasure check is probably the cheapest first move — it tells you whether you're building a repair path or just an alarm.

Audited against `origin/master` — **NOT DONE (items 1–3); item 4 unverifiable from git.** Staying open, and flagging that this is now on the critical path for other work. - **Item 1 — read-verify sweep of `gitea-packages` blobs:** no artifact. `git grep -rln "read-verify\|readverify\|blob.integrity\|verify-blob\|bitrot" origin/master` across spikersoft-infrastructure → **zero files**. - **Item 2 — post-push read-back verification in big-image CI:** not implemented. The only post-push gate is the older #511 one at `_build-deploy-service.yml:196-205` (copied into `spikersoft-quiz-generation.yml:63-75`), which does `docker manifest inspect`. Nothing streams a blob and hashes it against its digest. - **Item 3 — Gitea ignoring `Range` on blob GETs:** no patch, no vendored fix, no tracked upstream reference in any repo. - **Item 4 — single-drive MinIO can't heal / fioa SMART check:** git cannot settle this. Needs `mc admin info` / erasure-set config from the live cluster plus a SMART or badblocks report for the fioa volume. **Important, because it's easy to mistake for coverage:** that `docker manifest inspect` gate runs on every deploy and is described in workflow comments as post-push verification. It verifies **manifest queryability from the runner** — not blob readability, and not pullability on the target node. So it structurally cannot catch this ticket's failure class *or* #774's, despite looking like it does. Both threads say this explicitly; worth not letting its presence read as protection. **This ticket is blocking measurable work now:** - **#460** (artpipe OpenTelemetry) is hard-blocked — four tier-3 dispatch attempts failed, the last on registry `unexpected EOF` pull failures. No trace has ever reached Jaeger as a direct result. - **#521** (upstream image mirror) is arguably gated too: mirroring mongo/postgres/keycloak into a registry that has been observed truncating blobs would convert an upstream-availability risk into a local-corruption risk. I've noted on that ticket that #775 probably wants sequencing ahead of the critical-tier repoints. Given item 4 (can a single-drive MinIO heal anything?) determines whether items 1–2 are detection-only or actually recoverable, the SMART/erasure check is probably the cheapest first move — it tells you whether you're building a repair path or just an alarm.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#775