spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra][Gitea/MinIO] Large registry push restarts single-node MinIO → Gitea storage.Init crash-loops → full git+CI 404 outage #700

New Issue
Open
opened 2026-07-18 07:08:49 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-18 07:08:49 +00:00
Owner
Copy Link
Copy Source

Incident (2026-07-18 ~07:02–07:05, self-healed)

A large image push to the Gitea container registry took down all of Gitea (and therefore all CI + git) for ~3 minutes.

Trigger: the artpipe model-env-images build (run 9507) pushed the ~36 GiB prodstages image to the Gitea registry. Registry blobs are stored in MinIO (gitea-packages bucket). During the push:

  • container.go:395 PatchBlobsUpload [E] Package registry API internal error: 500 unexpected EOF
  • MinIO restarted (minio_minio: Exiting on signal: TERMINATED — swarm-initiated, consistent with a healthcheck timeout under push load).
  • With MinIO down, Gitea crash-looped on startup: routers/init.go:65 mustInit [F] storage.Init failed: Get "http://minio:9000/gitea-packages/?versioning=": dial tcp: lookup minio: no such host → Gitea served 404 for everything (web, API, and the act_runners' declare endpoint → runners crash-looped, misreporting "Gitea version too old").
  • MinIO recovered (~07:04) → Gitea recovered (~07:05). git.spikersoft.com back to 200; no lingering degradation.

Root cause / fragility

Single-node MinIO is a hard dependency of the Gitea package registry, and Gitea treats storage.Init failure as FATAL (crash-loop, not degraded). So any MinIO hiccup — including one caused by a large registry push — takes the whole dev/CI platform offline. This is adjacent to the single-node-MinIO SPOF flagged in #696.

Mitigations to consider

  • Harden MinIO under load: raise its swarm healthcheck timeout/interval + resource reservations so a heavy push doesn't trip a restart; confirm it isn't OOM/healthcheck-restarting.
  • Reduce blast radius of large pushes: throttle/limit the giant model-env images (the 36 GiB prodstages monolith is the worst offender — retiring the double-baked per-model redundancy would also shrink it), or route their registry storage off the shared MinIO.
  • Resilience: make Gitea storage.Init failure non-fatal / retry with backoff instead of crash-looping, so a transient MinIO blip degrades the registry only, not all of git+CI.
  • Longer-term: the #696 distributed/replicated-MinIO question directly affects this SPOF.

Note

Self-healed, no data loss observed. Filing for the fragility, not an active outage.

## Incident (2026-07-18 ~07:02–07:05, self-healed) A large image push to the Gitea container registry took down **all of Gitea (and therefore all CI + git)** for ~3 minutes. **Trigger:** the artpipe `model-env-images` build (run 9507) pushed the ~36 GiB `prodstages` image to the Gitea registry. Registry blobs are stored in **MinIO** (`gitea-packages` bucket). During the push: - `container.go:395 PatchBlobsUpload [E] Package registry API internal error: 500 unexpected EOF` - **MinIO restarted** (`minio_minio`: `Exiting on signal: TERMINATED` — swarm-initiated, consistent with a healthcheck timeout under push load). - With MinIO down, Gitea crash-looped on startup: `routers/init.go:65 mustInit [F] storage.Init failed: Get "http://minio:9000/gitea-packages/?versioning=": dial tcp: lookup minio: no such host` → Gitea served **404 for everything** (web, API, and the act_runners' declare endpoint → runners crash-looped, misreporting "Gitea version too old"). - MinIO recovered (~07:04) → Gitea recovered (~07:05). `git.spikersoft.com` back to 200; no lingering degradation. ## Root cause / fragility **Single-node MinIO is a hard dependency of the Gitea package registry, and Gitea treats `storage.Init` failure as FATAL** (crash-loop, not degraded). So any MinIO hiccup — including one *caused by* a large registry push — takes the whole dev/CI platform offline. This is adjacent to the single-node-MinIO SPOF flagged in #696. ## Mitigations to consider - Harden MinIO under load: raise its swarm healthcheck timeout/interval + resource reservations so a heavy push doesn't trip a restart; confirm it isn't OOM/healthcheck-restarting. - Reduce blast radius of large pushes: throttle/limit the giant model-env images (the 36 GiB `prodstages` monolith is the worst offender — retiring the double-baked per-model redundancy would also shrink it), or route their registry storage off the shared MinIO. - Resilience: make Gitea `storage.Init` failure non-fatal / retry with backoff instead of crash-looping, so a transient MinIO blip degrades the registry only, not all of git+CI. - Longer-term: the #696 distributed/replicated-MinIO question directly affects this SPOF. ## Note Self-healed, no data loss observed. Filing for the fragility, not an active outage.
spikerj commented 2026-07-18 18:01:44 +00:00
Author
Owner
Copy Link
Copy Source

Repo-side finding for the mitigation list (2026-07-18): the healthcheck-hardening suggestion is partly moot — minio/docker-stack.yml on master already carries the drive-aware probe with generous headroom (interval: 30s, timeout: 30s, retries: 5, start_period: 300s, restart: any — landed 07-12/07-13 for #504 and the clean-exit kill loop). The incident happened with that config, so a simple healthcheck-timeout bump is unlikely to be the fix.

What the stack does not have: any resources: block — no memory reservation or limit. Under a ~36 GiB registry push, MinIO's buffer/cache growth on a contended SERVER makes it a prime OOM-kill/scheduling-pressure candidate, and an OOM kill looks exactly like the observed swarm-initiated Exiting on signal: TERMINATED. Suggested next diagnostic before changing anything: check whether the 07:02 restart was OOM (docker service ps minio_minio --no-trunc / journalctl -k | grep -i oom around the window) vs healthcheck. If OOM → add a memory reservation (and consider MINIO_API_REQUESTS_MAX to bound concurrent push load); if healthcheck → the probe itself (mc ready under IO saturation) needs a longer timeout or a lighter probe.

The other two mitigations stand as written: Gitea treating storage.Init failure as fatal (crash-loop → full git+CI outage) is the blast-radius multiplier, and the 36 GiB prodstages monolith is tracked under the artpipe baked-images epic (#515/#520 cleanup)."

**Repo-side finding for the mitigation list (2026-07-18):** the healthcheck-hardening suggestion is partly moot — `minio/docker-stack.yml` on master *already* carries the drive-aware probe with generous headroom (`interval: 30s`, `timeout: 30s`, `retries: 5`, `start_period: 300s`, `restart: any` — landed 07-12/07-13 for #504 and the clean-exit kill loop). The incident happened **with** that config, so a simple healthcheck-timeout bump is unlikely to be the fix. What the stack does **not** have: any `resources:` block — no memory reservation or limit. Under a ~36 GiB registry push, MinIO's buffer/cache growth on a contended SERVER makes it a prime OOM-kill/scheduling-pressure candidate, and an OOM kill looks exactly like the observed swarm-initiated `Exiting on signal: TERMINATED`. Suggested next diagnostic before changing anything: check whether the 07:02 restart was OOM (`docker service ps minio_minio --no-trunc` / `journalctl -k | grep -i oom` around the window) vs healthcheck. If OOM → add a memory reservation (and consider `MINIO_API_REQUESTS_MAX` to bound concurrent push load); if healthcheck → the probe itself (`mc ready` under IO saturation) needs a longer timeout or a lighter probe. The other two mitigations stand as written: Gitea treating `storage.Init` failure as fatal (crash-loop → full git+CI outage) is the blast-radius multiplier, and the 36 GiB `prodstages` monolith is tracked under the artpipe baked-images epic (#515/#520 cleanup)."
spikerj commented 2026-07-22 11:02:05 +00:00
Author
Owner
Copy Link
Copy Source

Recurrence 2026-07-22 ~10:03–10:14Z (second data point for this ticket's hardening):

  • Trigger matches the documented signature: tier-3 artpipe finals build (backend run 12130) pushing multi-GB per-model worker images. Its push step failed at 10:03:40 and within the same minute every Gitea endpoint went 404 (web root, /explore, API version, Actions API) while MinIO itself answered health/live 200 throughout — i.e., a brief storage blip wedged Gitea's storage layer even though MinIO recovered instantly, exactly the storage.Init crash-loop pattern.
  • Duration ~10 minutes; recovered at ~10:14 after a restart. Ops email alerting is dead (#756), so detection was luck (an active session watching the run) — the two tickets compound each other.
  • Also observed on the same run: pulling the env base image took 83 minutes (06:56→08:19) over the registry data path — the #538/#775 slow-path making the outage window for large transfers much wider than it needs to be.

The mitigation this ticket already names (resource limits on the minio stack + Gitea storage-init resilience) remains unimplemented; tonight makes it two incidents from the same trigger class.

**Recurrence 2026-07-22 ~10:03–10:14Z** (second data point for this ticket's hardening): - Trigger matches the documented signature: tier-3 artpipe finals build (backend run 12130) pushing multi-GB per-model worker images. Its push step failed at 10:03:40 and within the same minute **every Gitea endpoint went 404** (web root, /explore, API version, Actions API) while **MinIO itself answered health/live 200 throughout** — i.e., a brief storage blip wedged Gitea's storage layer even though MinIO recovered instantly, exactly the storage.Init crash-loop pattern. - Duration ~10 minutes; recovered at ~10:14 after a restart. Ops email alerting is dead (#756), so detection was luck (an active session watching the run) — the two tickets compound each other. - Also observed on the same run: pulling the env base image took **83 minutes** (06:56→08:19) over the registry data path — the #538/#775 slow-path making the outage window for large transfers much wider than it needs to be. The mitigation this ticket already names (resource limits on the minio stack + Gitea storage-init resilience) remains unimplemented; tonight makes it two incidents from the same trigger class.
spikerj commented 2026-07-29 16:06:40 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — NOT DONE. Two incidents, zero implemented mitigations.

  • Resource limits: absent. git grep -n "resources" origin/master -- '*minio*' → zero matches. minio/docker-stack.yml still has no resources: block — which is exactly the mitigation the 2026-07-18 comment identified as untried.
  • MINIO_API_REQUESTS_MAX / push throttling: not present.
  • Gitea storage.Init resilience (retry/backoff instead of fatal): no change found. This is upstream Gitea behaviour and nothing in the repo works around it.

Worth being precise about what git proves here, since I've been careful to distinguish that on other infra tickets: the stack file is git-managed and deployed from the /mnt/infrastructure checkout, so git is authoritative for the stack config — the absence of resources: is a real finding, not an unverifiable gap. What git can't settle is whether the 2026-07-22 recurrence was OOM or healthcheck; that needs journalctl -k | grep -i oom on SERVER for the 10:03Z window.

Remaining: a memory reservation/limit on the minio stack, a request-concurrency bound, and a decision about the Gitea-fatal-init blast radius.

Two things make this more urgent than its age suggests. First, gitea/docker-stack.yml:7 still puts the Gitea Postgres volume on the same filesystem the runners build on (see #514), so disk pressure and this failure mode share a blast radius. Second, #521 proposes mirroring 37 upstream images — including mongo, postgres and keycloak — into this registry. Doing that before this ticket and #775 are settled would put the critical tier behind a registry that has twice taken git and CI down, and that has been observed truncating blobs (#821). I've noted the sequencing concern on #521 as well.

Audited against `origin/master` — **NOT DONE. Two incidents, zero implemented mitigations.** - **Resource limits: absent.** `git grep -n "resources" origin/master -- '*minio*'` → **zero matches**. `minio/docker-stack.yml` still has no `resources:` block — which is exactly the mitigation the 2026-07-18 comment identified as untried. - **`MINIO_API_REQUESTS_MAX` / push throttling: not present.** - **Gitea `storage.Init` resilience** (retry/backoff instead of fatal): no change found. This is upstream Gitea behaviour and nothing in the repo works around it. Worth being precise about what git proves here, since I've been careful to distinguish that on other infra tickets: the stack file **is** git-managed and deployed from the `/mnt/infrastructure` checkout, so git *is* authoritative for the stack config — the absence of `resources:` is a real finding, not an unverifiable gap. What git can't settle is whether the 2026-07-22 recurrence was OOM or healthcheck; that needs `journalctl -k | grep -i oom` on SERVER for the 10:03Z window. **Remaining:** a memory reservation/limit on the minio stack, a request-concurrency bound, and a decision about the Gitea-fatal-init blast radius. **Two things make this more urgent than its age suggests.** First, `gitea/docker-stack.yml:7` still puts the Gitea Postgres volume on the same filesystem the runners build on (see #514), so disk pressure and this failure mode share a blast radius. Second, **#521** proposes mirroring 37 upstream images — including mongo, postgres and keycloak — *into this registry*. Doing that before this ticket and #775 are settled would put the critical tier behind a registry that has twice taken git and CI down, and that has been observed truncating blobs (#821). I've noted the sequencing concern on #521 as well.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#700