[CI] Deploy step's stale-stack guard (#555) trips on parallel git pull of /mnt/infrastructure during fan-outs
#1196
Reference in new issue
Block a user
git pull of /mnt/infrastructure during fan-outs
#1196
What happens
_build-deploy-service.yml(backend) and the equivalent deploy steps SSH to the deploy host and runsudo git -C /mnt/infrastructure pull --ff-only, falling back togit fetch --prune origin masterand refusing to deploy when that fails too ("ERROR: cannot fetch origin/master — refusing to deploy a possibly stale stack (#555)"). During a master fan-out, 20-30 service pipelines reach that step within the same minute and all pull the same checkout; some of them fail the pull and the fetch and the deploy is refused even though the image was built and pushed fine.Evidence (2026-10-09)
Why it matters
Every merge burst leaves a handful of services on the previous image until someone notices and re-runs; the #555 guard is doing its job, but the failure it detects is self-inflicted contention, not staleness.
Proposed fix
flock /mnt/infrastructure/.git/deploy.lock(or aflock -w 120with a retry loop), so concurrent deploy steps queue instead of colliding.2>&1) so the next failure names its cause.git fetchper fan-out is enough — checkrev-parse HEADvsorigin/masterfirst and skip the pull when already current.Operator workaround until then: re-run failed deploys serially (
POST /actions/runs/{id}/rerunone at a time); parallel re-runs collide again.