[Bug][CI] arm64v8 image builds intermittently fail pulling mcr.microsoft.com base image over IPv6 — 'network is unreachable', ~1 in 6 arm jobs #580

New Issue

2026-07-14T17:43:50Z

spikerj commented

2026-07-14 17:43:50 +00:00

QA Team — found during the 2026-07-14 ~17:45Z sweep, on a live PR (backend #280).

ARM64 Docker builds intermittently die on the base image pull, before any project code is compiled. The runner tries to reach mcr.microsoft.com over IPv6 and has no working IPv6 route.

Evidence

Gitea Actions job 24196 — Blog Media Processor / build_and_publish (arm64v8, ubuntu-arm), run #5578, 2026-07-14 17:30:03Z:

#4 ERROR: failed to do request: Head "https://mcr.microsoft.com/v2/dotnet/aspnet/manifests/10.0":
   dial tcp [2603:1061:f:101::10]:443: connect: network is unreachable
ERROR: failed to build: failed to solve: mcr.microsoft.com/dotnet/aspnet:10.0:
   failed to do request: Head "https://mcr.microsoft.com/v2/dotnet/aspnet/manifests/10.0":
   dial tcp [2603:1061:f:101::10]:443: connect: network is unreachable
  ❌  Failure - Main Build Dockerfile
exitcode '1': failure

2603:1061:f:101::10 is an IPv6 address. The runner resolved mcr.microsoft.com to its AAAA record, tried to connect, and the host had no IPv6 path out.

Frequency

Not a one-off, and not chronic — 1 failure in the last 6 arm64v8 jobs (5 success, 1 failure). Classic dual-stack flake: it depends on whether the resolver hands back the AAAA record and whether Happy Eyeballs falls back to IPv4 in time.

Why it matters more than it looks

It fails the base image pull, so it can hit any arm64 build in any repo. Nothing about the changed code is relevant — it took down an arm64 leg of a docs-only PR.
It is pure noise: it makes green PRs look broken and trains everyone to re-run failed CI without reading it, which is how a real failure eventually gets waved through.
ARM runner capacity is currently 2, not 3 (see the companion ticket on arm_v8_2 crash-looping on ds6), so the surviving runners take more arm traffic and hit this more often.

Fix options

Disable IPv6 in the ARM runners' Docker daemon / job containers (--ipv6=false, or sysctl net.ipv6.conf.all.disable_ipv6=1 on the Jetsons). Blunt, effective, and matches the fact that nothing else here needs IPv6.
Give the Jetsons a real IPv6 route, if IPv6 is actually wanted on that segment.
Prefer IPv4 in resolution on the runner hosts (gai.conf precedence, or --dns pointing at a resolver that does not return AAAA for mcr).

(1) is almost certainly the right call for this cluster — the rest of the swarm is IPv4-only and this is the only place IPv6 shows up at all.

Retriggered run #5578 to unblock backend PR #280. Related: #549 (arm-runner build leg in the Angular pipeline) and #562 (separate image-pull failures on the dreamstream nodes — different mechanism, same symptom class of "CI/deploy pulls are unreliable on the Jetsons").

**QA Team** — found during the 2026-07-14 ~17:45Z sweep, on a live PR (backend #280). ARM64 Docker builds intermittently die on the **base image pull**, before any project code is compiled. The runner tries to reach `mcr.microsoft.com` over **IPv6** and has no working IPv6 route. ## Evidence Gitea Actions job 24196 — `Blog Media Processor / build_and_publish (arm64v8, ubuntu-arm)`, run #5578, 2026-07-14 17:30:03Z: ``` #4 ERROR: failed to do request: Head "https://mcr.microsoft.com/v2/dotnet/aspnet/manifests/10.0": dial tcp [2603:1061:f:101::10]:443: connect: network is unreachable ERROR: failed to build: failed to solve: mcr.microsoft.com/dotnet/aspnet:10.0: failed to do request: Head "https://mcr.microsoft.com/v2/dotnet/aspnet/manifests/10.0": dial tcp [2603:1061:f:101::10]:443: connect: network is unreachable ❌ Failure - Main Build Dockerfile exitcode '1': failure ``` `2603:1061:f:101::10` is an IPv6 address. The runner resolved `mcr.microsoft.com` to its AAAA record, tried to connect, and the host had no IPv6 path out. ## Frequency Not a one-off, and not chronic — **1 failure in the last 6 arm64v8 jobs** (5 success, 1 failure). Classic dual-stack flake: it depends on whether the resolver hands back the AAAA record and whether Happy Eyeballs falls back to IPv4 in time. ## Why it matters more than it looks - It fails the **base image pull**, so it can hit *any* arm64 build in *any* repo. Nothing about the changed code is relevant — it took down an arm64 leg of a **docs-only PR**. - It is pure noise: it makes green PRs look broken and trains everyone to re-run failed CI without reading it, which is how a real failure eventually gets waved through. - ARM runner capacity is currently **2, not 3** (see the companion ticket on `arm_v8_2` crash-looping on ds6), so the surviving runners take more arm traffic and hit this more often. ## Fix options 1. **Disable IPv6 in the ARM runners' Docker daemon / job containers** (`--ipv6=false`, or `sysctl net.ipv6.conf.all.disable_ipv6=1` on the Jetsons). Blunt, effective, and matches the fact that nothing else here needs IPv6. 2. **Give the Jetsons a real IPv6 route**, if IPv6 is actually wanted on that segment. 3. **Prefer IPv4 in resolution** on the runner hosts (`gai.conf` precedence, or `--dns` pointing at a resolver that does not return AAAA for mcr). (1) is almost certainly the right call for this cluster — the rest of the swarm is IPv4-only and this is the only place IPv6 shows up at all. Retriggered run #5578 to unblock backend PR #280. Related: #549 (arm-runner build leg in the Angular pipeline) and #562 (separate image-pull failures on the dreamstream nodes — different mechanism, same symptom class of "CI/deploy pulls are unreliable on the Jetsons").

spikerj referenced this issue

2026-07-14 20:01:27 +00:00

[Bug][CI][Infra] gitea-runners arm_v8_2 crash-looping on dreamstream6 since the #539 outage — ARM CI capacity silently down 3→2 #579

spikerj closed this issue

2026-07-17 03:52:39 +00:00

spikerj referenced this issue

2026-07-26 14:03:36 +00:00

hex-tower-defence tab crash: art_pipe export stage never decimates (137k tris / 109 MB assets) + the game loads every slot concurrently #855

Sign in to join this conversation.