spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][CI][Infra] gitea-runners arm_v8_2 crash-looping on dreamstream6 since the #539 outage — ARM CI capacity silently down 3→2 #579

New Issue
Closed
opened 2026-07-14 17:43:18 +00:00 by spikerj · 4 comments
spikerj commented 2026-07-14 17:43:18 +00:00
Owner
Copy Link
Copy Source

QA Team — found during the 2026-07-14 ~17:45Z sweep. Filed against closed #539 (dreamstream6 down since ~16:30Z 2026-07-13, closed 2026-07-14 09:40).

#539 was closed on the node coming back — and the node genuinely is back (docker node ls shows dreamstream6 Ready/Active, and it is serving swarm tasks). But the Gitea act_runner that lives on it never recovered, and nothing noticed, because a dead CI runner does not page anyone; jobs just quietly queue onto the survivors.

Evidence

$ docker service ls --filter name=gitea-runners
gitea-runners_amd64_4090_2_act_runner    1/1
gitea-runners_amd64_4090_act_runner      1/1
gitea-runners_amd64_laptop_act_runner    1/1
gitea-runners_arm_v8_1_act_runner        1/1
gitea-runners_arm_v8_2_act_runner        0/1     <-- dead
gitea-runners_arm_v8_3_act_runner        1/1

$ docker service ps gitea-runners_arm_v8_2_act_runner --no-trunc
... dreamstream6  Failed  "task: non-zero exit (1)"
... dreamstream6  Failed  "task: non-zero exit (1)"
... dreamstream6  Failed  "task: non-zero exit (1)"

It is in a tight crash loop — a fresh exit(1) roughly every 20 seconds, indefinitely.

Likely cause

arm_v8_2 is the only ARM runner whose state is bind-mounted off the ds6 host filesystem:

runner node /data
arm_v8_1 DreamStream5 bind /home/joey/Docker/gitea-act-runner/data
arm_v8_2 DreamStream6 bind /home/joey/Docker/gitea-act-runner/data + bind arm2-config.yaml
arm_v8_3 DreamStream7 named volume gitea-runners_arm3-runner-data

Image, env and registration token are byte-identical to the healthy arm_v8_1, so this is not a config-drift or bad-token problem — the spec is fine. What differs is that v8_2 depends on files that live on ds6's local disk, and ds6 is precisely the node that went down hard in #539. act_runner exits 1 at startup when /config.yaml is unreadable or its /data/.runner registration state is missing/corrupt, which is exactly what you would expect if that host path did not survive the outage.

I could not confirm the filesystem state directly — there is no SSH from the manager to ds6 (Permission denied (publickey)), and I did not want to create a probe service to poke at it. Someone with node access should check /home/joey/Docker/gitea-act-runner/ on ds6 — that will settle it in about ten seconds.

Why this matters

  • ARM CI capacity is 3 runners → 2, silently, since yesterday. Every ubuntu-arm job in every repo now contends for ds5 and ds7.
  • It makes ARM job queueing and flakiness worse, which muddies the water on unrelated CI bugs (see the IPv6 base-image failure filed alongside this).

Fix

Short term: restore the config + registration state on ds6, or just re-register the runner.
Better: move arm_v8_2 (and arm_v8_1) onto named volumes like arm_v8_3 already uses. Host bind-mounted runner state is the reason a node reboot turns into a permanently dead runner. arm_v8_3 survived on the same class of hardware because its state is in a volume; the other two are one bad reboot away from this exact ticket.

Also worth considering: an alert when any gitea-runners_* service sits at 0/1, since a dead runner is invisible by design.

**QA Team** — found during the 2026-07-14 ~17:45Z sweep. Filed against **closed #539** (*dreamstream6 down since ~16:30Z 2026-07-13*, closed 2026-07-14 09:40). #539 was closed on the node coming back — and the node genuinely is back (`docker node ls` shows dreamstream6 `Ready`/`Active`, and it is serving swarm tasks). But **the Gitea act_runner that lives on it never recovered**, and nothing noticed, because a dead CI runner does not page anyone; jobs just quietly queue onto the survivors. ## Evidence ``` $ docker service ls --filter name=gitea-runners gitea-runners_amd64_4090_2_act_runner 1/1 gitea-runners_amd64_4090_act_runner 1/1 gitea-runners_amd64_laptop_act_runner 1/1 gitea-runners_arm_v8_1_act_runner 1/1 gitea-runners_arm_v8_2_act_runner 0/1 <-- dead gitea-runners_arm_v8_3_act_runner 1/1 $ docker service ps gitea-runners_arm_v8_2_act_runner --no-trunc ... dreamstream6 Failed "task: non-zero exit (1)" ... dreamstream6 Failed "task: non-zero exit (1)" ... dreamstream6 Failed "task: non-zero exit (1)" ``` It is in a tight crash loop — a fresh exit(1) roughly every 20 seconds, indefinitely. ## Likely cause `arm_v8_2` is the only ARM runner whose **state is bind-mounted off the ds6 host filesystem**: | runner | node | `/data` | |---|---|---| | `arm_v8_1` | DreamStream5 | bind `/home/joey/Docker/gitea-act-runner/data` | | `arm_v8_2` | **DreamStream6** | bind `/home/joey/Docker/gitea-act-runner/data` + bind `arm2-config.yaml` | | `arm_v8_3` | DreamStream7 | **named volume** `gitea-runners_arm3-runner-data` | Image, env and registration token are byte-identical to the healthy `arm_v8_1`, so this is not a config-drift or bad-token problem — the spec is fine. What differs is that v8_2 depends on files that live on **ds6's local disk**, and ds6 is precisely the node that went down hard in #539. act_runner exits 1 at startup when `/config.yaml` is unreadable or its `/data/.runner` registration state is missing/corrupt, which is exactly what you would expect if that host path did not survive the outage. I could not confirm the filesystem state directly — there is no SSH from the manager to ds6 (`Permission denied (publickey)`), and I did not want to create a probe service to poke at it. Someone with node access should check `/home/joey/Docker/gitea-act-runner/` on ds6 — that will settle it in about ten seconds. ## Why this matters - ARM CI capacity is **3 runners → 2**, silently, since yesterday. Every `ubuntu-arm` job in every repo now contends for ds5 and ds7. - It makes ARM job queueing and flakiness worse, which muddies the water on unrelated CI bugs (see the IPv6 base-image failure filed alongside this). ## Fix Short term: restore the config + registration state on ds6, or just re-register the runner. Better: **move `arm_v8_2` (and `arm_v8_1`) onto named volumes like `arm_v8_3` already uses.** Host bind-mounted runner state is the reason a node reboot turns into a permanently dead runner. `arm_v8_3` survived on the same class of hardware because its state is in a volume; the other two are one bad reboot away from this exact ticket. Also worth considering: an alert when any `gitea-runners_*` service sits at 0/1, since a dead runner is invisible by design.
spikerj commented 2026-07-14 20:01:27 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — sweep 2026-07-14 ~20:15Z. Still broken, and it lies to the obvious check.

docker service ls showed this at 1/1 during my sweep and I briefly recorded it as recovered. That was wrong, and the way it was wrong is worth writing down:

$ docker service ls --filter name=gitea-runners
gitea-runners_arm_v8_2_act_runner   1/1     <-- looks healthy

$ docker service ps gitea-runners_arm_v8_2_act_runner
dreamstream6 | Starting less than a second ago |
dreamstream6 | Failed 5 seconds ago            | "task: non-zero exit (1)"
dreamstream6 | Failed 25 seconds ago           | "task: non-zero exit (1)"
dreamstream6 | Failed 45 seconds ago           | "task: non-zero exit (1)"

It is flapping, not running — a fresh exit(1) roughly every 20 seconds, indefinitely. Because swarm counts a task in Starting toward the replica count, a docker service ls sampled mid-restart reports 1/1. So the cheapest check anyone would reach for reports this runner as healthy at least some of the time.

Two consequences worth carrying:

  • Do not verify this ticket with docker service ls. Use docker service ps <svc> and look for the Failed/exit(1) cycle. Same applies to any alert we might add for dead runners — alerting on replica count alone will miss a crash-looping service that restarts fast enough.
  • ARM CI capacity remains 2 runners, not 3. Every ubuntu-arm job across every repo is still contending for ds5 and ds7, which is also making #580 (the IPv6 base-image flake) hit more often than it otherwise would.

Root cause from the original report is unchanged and still unverified for want of node access: arm_v8_2 is the only ARM runner whose config + registration state is bind-mounted off the ds6 host filesystem (/home/joey/Docker/gitea-act-runner/), and ds6 is the node that went down hard in #539. arm_v8_3 on ds7 uses a named volume and survived. Someone with a shell on ds6 can settle this in ten seconds by checking whether that directory still holds arm2-config.yaml and a valid data/.runner.

**QA Team** — sweep 2026-07-14 ~20:15Z. **Still broken, and it lies to the obvious check.** `docker service ls` showed this at **1/1** during my sweep and I briefly recorded it as recovered. That was wrong, and the way it was wrong is worth writing down: ``` $ docker service ls --filter name=gitea-runners gitea-runners_arm_v8_2_act_runner 1/1 <-- looks healthy $ docker service ps gitea-runners_arm_v8_2_act_runner dreamstream6 | Starting less than a second ago | dreamstream6 | Failed 5 seconds ago | "task: non-zero exit (1)" dreamstream6 | Failed 25 seconds ago | "task: non-zero exit (1)" dreamstream6 | Failed 45 seconds ago | "task: non-zero exit (1)" ``` It is **flapping**, not running — a fresh exit(1) roughly every 20 seconds, indefinitely. Because swarm counts a task in `Starting` toward the replica count, a `docker service ls` sampled mid-restart reports **1/1**. So the cheapest check anyone would reach for reports this runner as healthy at least some of the time. Two consequences worth carrying: - **Do not verify this ticket with `docker service ls`.** Use `docker service ps <svc>` and look for the Failed/exit(1) cycle. Same applies to any alert we might add for dead runners — alerting on replica count alone will miss a crash-looping service that restarts fast enough. - ARM CI capacity remains **2 runners, not 3**. Every `ubuntu-arm` job across every repo is still contending for ds5 and ds7, which is also making #580 (the IPv6 base-image flake) hit more often than it otherwise would. Root cause from the original report is unchanged and still unverified for want of node access: `arm_v8_2` is the only ARM runner whose config + registration state is bind-mounted off the ds6 host filesystem (`/home/joey/Docker/gitea-act-runner/`), and ds6 is the node that went down hard in #539. `arm_v8_3` on ds7 uses a **named volume** and survived. Someone with a shell on ds6 can settle this in ten seconds by checking whether that directory still holds `arm2-config.yaml` and a valid `data/.runner`.
spikerj commented 2026-07-14 20:49:35 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — still flapping. Re-verified 2026-07-14.

gitea-runners_arm_v8_2_act_runner on dreamstream6 is still crash-looping, ~3 days on:

$ docker service ps gitea-runners_arm_v8_2_act_runner
  dreamstream6  Running 11 seconds ago
  dreamstream6  Failed  19 seconds ago       "task: non-zero exit (1)"
  dreamstream6  Failed  40 seconds ago       "task: non-zero exit (1)"
  dreamstream6  Failed  about a minute ago   "task: non-zero exit (1)"
  dreamstream6  Failed  about a minute ago   "task: non-zero exit (1)"

It exits(1) roughly every 20 seconds and is immediately rescheduled.

⚠️ Do not verify this ticket with docker service ls — it will lie to you. It reports 1/1 for this service, because swarm counts a task in Starting toward the replica count. The service looks perfectly healthy in service ls while completing zero work. docker service ps is the only view that shows the truth.

I nearly logged this as recovered for exactly that reason. Flagging it here so the next person doesn't.

Two other runners show historical churn worth a glance, though both are stable now:

  • gitea-runners_amd64_laptop_act_runner — 4× exit (1) around 44h ago, running clean since.
  • gitea-runners_amd64_4090_act_runner — one exit (137) (OOM-kill) 2 days ago.

The exit (137) on the 4090 runner is worth keeping an eye on given that node now also hosts quiz-generation and image-description (both large-VRAM, and image-description's resident footprint is ~18.3 GB). If runner OOM-kills recur there, memory pressure from the co-located GPU workers is the first place to look.

**QA Team — still flapping. Re-verified 2026-07-14.** `gitea-runners_arm_v8_2_act_runner` on **dreamstream6** is still crash-looping, ~3 days on: ``` $ docker service ps gitea-runners_arm_v8_2_act_runner dreamstream6 Running 11 seconds ago dreamstream6 Failed 19 seconds ago "task: non-zero exit (1)" dreamstream6 Failed 40 seconds ago "task: non-zero exit (1)" dreamstream6 Failed about a minute ago "task: non-zero exit (1)" dreamstream6 Failed about a minute ago "task: non-zero exit (1)" ``` It exits(1) roughly every 20 seconds and is immediately rescheduled. ⚠️ **Do not verify this ticket with `docker service ls` — it will lie to you.** It reports `1/1` for this service, because swarm counts a task in `Starting` toward the replica count. The service looks perfectly healthy in `service ls` while completing zero work. `docker service ps` is the only view that shows the truth. I nearly logged this as recovered for exactly that reason. Flagging it here so the next person doesn't. Two other runners show historical churn worth a glance, though both are stable *now*: - `gitea-runners_amd64_laptop_act_runner` — 4× `exit (1)` around 44h ago, running clean since. - `gitea-runners_amd64_4090_act_runner` — one `exit (137)` (OOM-kill) 2 days ago. The `exit (137)` on the 4090 runner is worth keeping an eye on given that node now also hosts quiz-generation and image-description (both large-VRAM, and image-description's resident footprint is ~18.3 GB). If runner OOM-kills recur there, memory pressure from the co-located GPU workers is the first place to look.
spikerj commented 2026-07-14 22:37:34 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — confirming this is still live, and that it is worse than "capacity down 3→2".

gitea-runners_arm_v8_2_act_runner is crash-looping right now on dreamstream6:

task: non-zero exit (1)   Failed 14 seconds ago
task: non-zero exit (1)   Failed 35 seconds ago
task: non-zero exit (1)   Failed 55 seconds ago
task: non-zero exit (1)   Failed ~1m ago

Exit(1) roughly every 20 seconds. Note it reported 1/1 in docker service ls on our first check and 0/1 minutes later — it is flapping, not converged, so a green service ls does not clear this ticket.

We could not retrieve its logs: docker service logs fails with node smjf65zries0iyugd9h2pgrml is not available — the manager cannot stream logs from ds6 at all.

The bigger context we hit while investigating

dreamstream1, dreamstream4 and dreamstream6 all report Ready / Active in docker node ls, but are effectively degraded:

  • the manager cannot stream logs from any of the three
  • all three reject image pulls (No such image: ...:latest — see our note on #587)
  • ds6 additionally failed a digest-pinned Docker Hub pull 21 h ago (postgres:16@sha256:be01cf...), which is not explained by the internal-registry tag problem

That last point suggests a node-level pull/disk fault on ds6 on top of the registry issue. We did not have node shell access to confirm — suspect disk-full, which would match the existing build-node disk-cleanup runbook. Worth a node-level check on ds1/4/6 as a follow-up; the arm runner crash-loop may simply be a symptom of it.

Related: #587 (registry :latest), #562, #539 (ds6 previously down).

**QA Team** — confirming this is still live, and that it is worse than "capacity down 3→2". `gitea-runners_arm_v8_2_act_runner` is **crash-looping right now** on dreamstream6: ``` task: non-zero exit (1) Failed 14 seconds ago task: non-zero exit (1) Failed 35 seconds ago task: non-zero exit (1) Failed 55 seconds ago task: non-zero exit (1) Failed ~1m ago ``` Exit(1) roughly every **20 seconds**. Note it reported `1/1` in `docker service ls` on our first check and `0/1` minutes later — it is **flapping, not converged**, so a green `service ls` does not clear this ticket. We could not retrieve its logs: `docker service logs` fails with `node smjf65zries0iyugd9h2pgrml is not available` — the manager cannot stream logs from ds6 at all. ## The bigger context we hit while investigating dreamstream1, dreamstream4 and dreamstream6 all report `Ready` / `Active` in `docker node ls`, but are effectively **degraded**: - the manager cannot stream logs from any of the three - all three reject image pulls (`No such image: ...:latest` — see our note on #587) - ds6 additionally failed a **digest-pinned Docker Hub** pull 21 h ago (`postgres:16@sha256:be01cf...`), which is *not* explained by the internal-registry tag problem That last point suggests a **node-level pull/disk fault on ds6 on top of** the registry issue. We did not have node shell access to confirm — **suspect disk-full**, which would match the existing build-node disk-cleanup runbook. Worth a node-level check on ds1/4/6 as a follow-up; the arm runner crash-loop may simply be a symptom of it. Related: #587 (registry `:latest`), #562, #539 (ds6 previously down).
spikerj commented 2026-07-29 15:56:03 +00:00
Author
Owner
Copy Link
Copy Source

Noting for the record — this was closed today (2026-07-29 ~15:36Z) with no closing comment, and I'm leaving it closed on the assumption that was deliberate. But the durable fix is definitively absent from origin/master, so if the close was based on ds6 being hand-repaired, the fragility that caused this is unchanged and will recur on the next bad reboot.

What I verified:

  • The recommended fix was never applied. gitea-act-runner/docker-stack.yml:150-153 still bind-mounts the ds6 host paths for arm_v8_2 — /home/joey/Docker/gitea-act-runner/arm2-config.yaml:/config.yaml and /home/joey/Docker/gitea-act-runner/data:/data:rw. The named-volume migration this ticket recommended did not happen.
  • arm_v8_1 is on a host bind too (:175), so the "one bad reboot away" exposure applies to both remaining ARM runners, not just the one that broke.
  • The healthy comparison case was deleted. arm_v8_3 — the runner on a named volume that survived #539 — was removed by 2d519f7 (2026-07-23, "drop ARM runner #3 — ds7 is now the dedicated Keycloak node"). So ARM capacity is now structurally 2, and if arm_v8_2 is still flapping it's effectively 1.
  • No 0/1 runner alert was added. Which matters more here than usual: this ticket's own comments document that docker service ls reports 1/1 for a service crash-looping every ~20s, because swarm counts a task in Starting toward the replica count. Any alert built on replica count would have the same blind spot the humans did.
  • git log for gitea-act-runner/ since 07-13 shows only runner-count and image-version changes (6ba6f2a, 2d519f7, d968109) — none referencing this ticket.

If ds6 was repaired by hand, that's fine and this can stay closed — but the two follow-ups worth carrying somewhere are migrate both ARM runners to named volumes and add a runner-liveness alert that uses docker service ps, not service ls. Happy to file those as a separate ticket if you'd rather not reopen this one.

Noting for the record — this was closed today (2026-07-29 ~15:36Z) with no closing comment, and I'm leaving it closed on the assumption that was deliberate. But the **durable fix is definitively absent from `origin/master`**, so if the close was based on ds6 being hand-repaired, the fragility that caused this is unchanged and will recur on the next bad reboot. What I verified: - **The recommended fix was never applied.** `gitea-act-runner/docker-stack.yml:150-153` still bind-mounts the ds6 host paths for `arm_v8_2` — `/home/joey/Docker/gitea-act-runner/arm2-config.yaml:/config.yaml` and `/home/joey/Docker/gitea-act-runner/data:/data:rw`. The named-volume migration this ticket recommended did not happen. - **`arm_v8_1` is on a host bind too** (`:175`), so the "one bad reboot away" exposure applies to both remaining ARM runners, not just the one that broke. - **The healthy comparison case was deleted.** `arm_v8_3` — the runner on a named volume that survived #539 — was removed by `2d519f7` (2026-07-23, "drop ARM runner #3 — ds7 is now the dedicated Keycloak node"). So ARM capacity is now structurally **2**, and if arm_v8_2 is still flapping it's effectively **1**. - **No 0/1 runner alert was added.** Which matters more here than usual: this ticket's own comments document that `docker service ls` reports `1/1` for a service crash-looping every ~20s, because swarm counts a task in `Starting` toward the replica count. Any alert built on replica count would have the same blind spot the humans did. - `git log` for `gitea-act-runner/` since 07-13 shows only runner-count and image-version changes (`6ba6f2a`, `2d519f7`, `d968109`) — none referencing this ticket. If ds6 was repaired by hand, that's fine and this can stay closed — but the two follow-ups worth carrying somewhere are **migrate both ARM runners to named volumes** and **add a runner-liveness alert that uses `docker service ps`, not `service ls`**. Happy to file those as a separate ticket if you'd rather not reopen this one.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#579