[CI] Deploy step's stale-stack guard (#555) trips on parallel git pull of /mnt/infrastructure during fan-outs #1196

Open
opened 2026-10-09 16:04:46 +00:00 by spikerj · 0 comments
Owner

What happens

_build-deploy-service.yml (backend) and the equivalent deploy steps SSH to the deploy host and run sudo git -C /mnt/infrastructure pull --ff-only, falling back to git fetch --prune origin master and refusing to deploy when that fails too ("ERROR: cannot fetch origin/master — refusing to deploy a possibly stale stack (#555)"). During a master fan-out, 20-30 service pipelines reach that step within the same minute and all pull the same checkout; some of them fail the pull and the fetch and the deploy is refused even though the image was built and pushed fine.

Evidence (2026-10-09)

  • 08:01:04–08:01:42Z, fan-out 5806fb2a: 6 deploys refused in 40 s (runs 46022, 46025, 46027, 46035, 46040, 46042); 23 sibling runs passed the same step.
  • 08:1xZ: 46025 and 46042 re-run in parallel → refused again; re-run one at a time → both pass.
  • 15:5xZ, fan-out 79ee8bbf: 46146 (artstudio-metrics) refused the same way, re-run passes.
  • The job log shows only the WARNING/ERROR lines; git's own stderr from the sudo'd pull is not captured, so the exact git error (index.lock / FETCH_HEAD.lock collision vs. a Gitea timeout under the burst) is not visible.

Why it matters

Every merge burst leaves a handful of services on the previous image until someone notices and re-runs; the #555 guard is doing its job, but the failure it detects is self-inflicted contention, not staleness.

Proposed fix

  1. Serialize the checkout update on the deploy host: wrap the pull/fetch in flock /mnt/infrastructure/.git/deploy.lock (or a flock -w 120 with a retry loop), so concurrent deploy steps queue instead of colliding.
  2. Capture git's stderr into the job log (2>&1) so the next failure names its cause.
  3. Optionally: one git fetch per fan-out is enough — check rev-parse HEAD vs origin/master first and skip the pull when already current.

Operator workaround until then: re-run failed deploys serially (POST /actions/runs/{id}/rerun one at a time); parallel re-runs collide again.

## What happens `_build-deploy-service.yml` (backend) and the equivalent deploy steps SSH to the deploy host and run `sudo git -C /mnt/infrastructure pull --ff-only`, falling back to `git fetch --prune origin master` and refusing to deploy when that fails too ("ERROR: cannot fetch origin/master — refusing to deploy a possibly stale stack (#555)"). During a master fan-out, 20-30 service pipelines reach that step within the same minute and all pull the **same checkout**; some of them fail the pull *and* the fetch and the deploy is refused even though the image was built and pushed fine. ## Evidence (2026-10-09) - 08:01:04–08:01:42Z, fan-out 5806fb2a: 6 deploys refused in 40 s (runs 46022, 46025, 46027, 46035, 46040, 46042); 23 sibling runs passed the same step. - 08:1xZ: 46025 and 46042 re-run **in parallel** → refused again; re-run **one at a time** → both pass. - 15:5xZ, fan-out 79ee8bbf: 46146 (artstudio-metrics) refused the same way, re-run passes. - The job log shows only the WARNING/ERROR lines; git's own stderr from the sudo'd pull is not captured, so the exact git error (index.lock / FETCH_HEAD.lock collision vs. a Gitea timeout under the burst) is not visible. ## Why it matters Every merge burst leaves a handful of services on the previous image until someone notices and re-runs; the #555 guard is doing its job, but the failure it detects is self-inflicted contention, not staleness. ## Proposed fix 1. Serialize the checkout update on the deploy host: wrap the pull/fetch in `flock /mnt/infrastructure/.git/deploy.lock` (or a `flock -w 120` with a retry loop), so concurrent deploy steps queue instead of colliding. 2. Capture git's stderr into the job log (`2>&1`) so the next failure names its cause. 3. Optionally: one `git fetch` per fan-out is enough — check `rev-parse HEAD` vs `origin/master` first and skip the pull when already current. Operator workaround until then: re-run failed deploys **serially** (`POST /actions/runs/{id}/rerun` one at a time); parallel re-runs collide again.
Sign in to join this conversation.