spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][CI][Deploy] cancel-in-progress on master kills DEPLOY jobs — back-to-back merges cancel each other and fixes silently never ship #590

New Issue
Closed
opened 2026-07-14 20:38:46 +00:00 by spikerj · 5 comments
spikerj commented 2026-07-14 20:38:46 +00:00
Owner
Copy Link
Copy Source

QA Team — sweep 2026-07-14 ~22:00Z. Fifth distinct way a merged fix fails to reach production, and this one is actively firing right now.

The bug

22 workflows carry:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

On master, github.ref is a constant (refs/heads/master). So the concurrency group is constant, and every merge cancels the in-flight run of the previous merge — including its deploy job.

deploy is the last stage in the chain (build_and_publish → deploy → notify), which makes it the most likely job to be executing when the next merge lands, and therefore the most likely to be killed. The build usually survives (it finishes first); the deploy is what dies.

cancel-in-progress is the right call for PR validation — you do not want to burn runners revalidating superseded commits. It is the wrong call for a CD pipeline on master, where the whole point of the job is to make production match the branch.

Live evidence, from today

Three merges landed inside ~10 minutes (#283 → #284 → #285). Every job on the current master HEAD is now:

$ tea api .../commits/f3f51669/statuses
  {'failure': 50}
  failure descriptions: {'Has been cancelled': 50}

50 jobs. All cancelled. Nothing built, nothing deployed.

And the concrete casualty:

#564 (quiz validation fail-closed) merged at         20:21Z
spikersoft-quiz-generation service last updated at   20:12:26Z    <-- BEFORE the merge

The #564 fix is not in the running image. Its Quiz Generation deploy job was cancelled by the next merge. The ticket is closed, the PR is merged, CI shows a plausible-looking history — and production is still running the old code with the fail-open validation bug that #564 was written to fix.

Note the runners are healthy (up for days) and Gitea has not restarted (up 15 h) — this is not infrastructure, it is the concurrency policy doing exactly what it was told.

Why this is worse than it sounds

The failure is silent and self-concealing:

  • The merge is green in the PR.
  • The cancelled run is not obviously "broken" — "cancelled" reads as intentional.
  • The next merge's run looks healthy, so a casual glance at CI says fine.
  • Nothing anywhere says "the fix you just merged did not deploy."

And it is worst exactly when it matters most — during an incident, when fixes land back-to-back. That is precisely when merges come fast enough to cancel each other, and precisely when you most need the deploy to actually happen. Today's session is the demonstration: we merged five fixes in quick succession while firefighting, and at least one of them never shipped.

Fix

Pick one:

  1. Drop cancel-in-progress on master. Keep it for pull_request only:

    concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

    Simple, and correct: PR runs still get superseded; master runs always finish.

  2. Queue master deploys instead of cancelling them — a separate concurrency group for the deploy job with cancel-in-progress: false, so deploys serialize rather than kill each other.

Option 1 is the smaller change and fixes the observed damage.

This is now a pattern, not an incident

Five independent mechanisms this week by which a merged fix does not reach production:

  • #540 — Scheduler had no CI workflow at all.
  • #581 — lesson-video-processor's deploy gate rejects every run (missing Actions secret); it has never deployed once.
  • #583 — 28/30 workflows do not re-run when their own workflow file changes, so a CI fix cannot trigger itself.
  • #584 — 28/29 deploy jobs never git pull /mnt/infrastructure, so they deploy stale stack files and exit 0.
  • This — back-to-back merges cancel each other's deploys.

Every one of them is green-looking and silent. Individually they are papercuts; together they mean "merged" does not imply "deployed" in this repo, and there is currently no way to tell the difference from the board. That is the thing worth fixing — a post-deploy reconciliation check (does the running image/config match master?) would have caught all five.

Immediate action: re-run the Quiz Generation workflow so #564 actually ships. It is currently closed-but-not-deployed.

**QA Team** — sweep 2026-07-14 ~22:00Z. Fifth distinct way a merged fix fails to reach production, and this one is actively firing right now. ## The bug 22 workflows carry: ```yaml concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true ``` On **master**, `github.ref` is a constant (`refs/heads/master`). So the concurrency group is constant, and **every merge cancels the in-flight run of the previous merge** — including its `deploy` job. `deploy` is the *last* stage in the chain (`build_and_publish` → `deploy` → `notify`), which makes it the **most likely job to be executing when the next merge lands**, and therefore the most likely to be killed. The build usually survives (it finishes first); the deploy is what dies. `cancel-in-progress` is the right call for **PR validation** — you do not want to burn runners revalidating superseded commits. It is the wrong call for a **CD pipeline on master**, where the whole point of the job is to make production match the branch. ## Live evidence, from today Three merges landed inside ~10 minutes (#283 → #284 → #285). Every job on the current master HEAD is now: ``` $ tea api .../commits/f3f51669/statuses {'failure': 50} failure descriptions: {'Has been cancelled': 50} ``` **50 jobs. All cancelled. Nothing built, nothing deployed.** And the concrete casualty: ``` #564 (quiz validation fail-closed) merged at 20:21Z spikersoft-quiz-generation service last updated at 20:12:26Z <-- BEFORE the merge ``` **The #564 fix is not in the running image.** Its Quiz Generation `deploy` job was cancelled by the next merge. The ticket is closed, the PR is merged, CI shows a plausible-looking history — and production is still running the old code with the fail-open validation bug that #564 was written to fix. Note the runners are healthy (up for days) and Gitea has not restarted (up 15 h) — this is not infrastructure, it is the concurrency policy doing exactly what it was told. ## Why this is worse than it sounds The failure is **silent and self-concealing**: - The merge is green in the PR. - The cancelled run is not obviously "broken" — "cancelled" reads as intentional. - The *next* merge's run looks healthy, so a casual glance at CI says fine. - Nothing anywhere says "the fix you just merged did not deploy." And it is **worst exactly when it matters most** — during an incident, when fixes land back-to-back. That is precisely when merges come fast enough to cancel each other, and precisely when you most need the deploy to actually happen. Today's session is the demonstration: we merged five fixes in quick succession while firefighting, and at least one of them never shipped. ## Fix Pick one: 1. **Drop `cancel-in-progress` on master.** Keep it for `pull_request` only: ```yaml concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} ``` Simple, and correct: PR runs still get superseded; master runs always finish. 2. **Queue master deploys instead of cancelling them** — a separate concurrency group for the deploy job with `cancel-in-progress: false`, so deploys serialize rather than kill each other. Option 1 is the smaller change and fixes the observed damage. ## This is now a pattern, not an incident Five independent mechanisms this week by which a merged fix does not reach production: - **#540** — Scheduler had no CI workflow at all. - **#581** — lesson-video-processor's deploy gate rejects every run (missing Actions secret); it has never deployed once. - **#583** — 28/30 workflows do not re-run when their own workflow file changes, so a CI fix cannot trigger itself. - **#584** — 28/29 deploy jobs never `git pull` `/mnt/infrastructure`, so they deploy stale stack files and exit 0. - **This** — back-to-back merges cancel each other's deploys. Every one of them is green-looking and silent. Individually they are papercuts; together they mean **"merged" does not imply "deployed"** in this repo, and there is currently no way to tell the difference from the board. That is the thing worth fixing — a post-deploy reconciliation check (does the running image/config match master?) would have caught all five. **Immediate action:** re-run the Quiz Generation workflow so #564 actually ships. It is currently closed-but-not-deployed.
spikerj commented 2026-07-14 21:26:54 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — this ticket's blast radius is bigger than "deploy jobs". It is also silently destroying multi-arch image manifests.

cancel-in-progress: true does not just skip deploys. In the multi-arch workflows, the job graph is:

build_and_publish   ->  pushes :amd64 and :arm64v8
create_manifest     ->  needs: [build_and_publish]   # stitches :latest from those two tags
deploy              ->  needs: [create_manifest]

When a back-to-back push cancels an in-flight run after the arch tags have been pushed but before create_manifest runs, the result is a registry state where the architecture images are healthy but :latest does not exist at all.

That is worse than a skipped deploy. A skipped deploy is a no-op — you just keep running the old version. A skipped create_manifest leaves :latest missing, so every subsequent pull on every node fails, including reschedules of services that were running perfectly fine.

This is the confirmed root cause of #587. I verified it: 10 images have :amd64=ok, :arm64v8=ok, :latest=MISSING. Full evidence on that ticket. It is currently why spikersoft-coderunner cannot place its 4th replica.

Suggested fix (widened)

Do not simply remove cancel-in-progress — cancelling redundant builds is worth keeping. Instead, make cancellation safe:

  • Exclude create_manifest and deploy from the concurrency group, so a superseded run still finishes stitching and shipping what it already pushed; or
  • Move the arch push and the manifest stitch into a single job, so it is atomic — the tags and the index either both land or neither does.

The second is preferable: it also closes the mixed-commit window described in #548 (which, note, is closed but whose fix never landed — all 23 multi-arch workflows still --amend mutable arch tags).

**QA Team — this ticket's blast radius is bigger than "deploy jobs". It is also silently destroying multi-arch image manifests.** `cancel-in-progress: true` does not just skip deploys. In the multi-arch workflows, the job graph is: ``` build_and_publish -> pushes :amd64 and :arm64v8 create_manifest -> needs: [build_and_publish] # stitches :latest from those two tags deploy -> needs: [create_manifest] ``` When a back-to-back push cancels an in-flight run **after the arch tags have been pushed but before `create_manifest` runs**, the result is a registry state where the architecture images are healthy but **`:latest` does not exist at all**. That is worse than a skipped deploy. A skipped deploy is a no-op — you just keep running the old version. A skipped `create_manifest` leaves `:latest` **missing**, so *every subsequent pull on every node fails*, including reschedules of services that were running perfectly fine. **This is the confirmed root cause of #587.** I verified it: 10 images have `:amd64=ok`, `:arm64v8=ok`, `:latest=MISSING`. Full evidence on that ticket. It is currently why `spikersoft-coderunner` cannot place its 4th replica. ## Suggested fix (widened) Do not simply remove `cancel-in-progress` — cancelling redundant *builds* is worth keeping. Instead, make cancellation safe: - Exclude `create_manifest` and `deploy` from the concurrency group, so a superseded run still finishes stitching and shipping what it already pushed; **or** - Move the arch push and the manifest stitch into a **single job**, so it is atomic — the tags and the index either both land or neither does. The second is preferable: it also closes the mixed-commit window described in **#548** (which, note, is closed but whose fix never landed — all 23 multi-arch workflows still `--amend` mutable arch tags).
spikerj commented 2026-07-14 21:35:20 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — 🚨 ESCALATION. This is not an intermittent race. Master's CI has been in a total cancellation cascade all day. NOTHING merged today has shipped.

The measurement

Every one of the 25 most recent workflow runs on master is cancelled. Not failed — cancelled. Zero completed.

conclusions across last 25 runs: {'cancelled': 25}

The cause is arithmetic. Today saw 22 merges to master, and the gaps between consecutive merges are:

gaps between merges (minutes, newest first): [1, 17, 0, 0, 0, 13, 5, 9, 0, 1, 7]
median gap: 1 minute
merges landing <15 min apart: 10 of 11

Several merges landed in the same minute. A multi-arch .NET build (amd64 + arm64v8) takes 10–20+ minutes — longer right now, because the ARM runner on dreamstream6 is crash-looping (#579).

With concurrency.cancel-in-progress: true keyed on ${{ github.workflow }}-${{ github.ref }}, every push to master cancels all in-flight runs for that workflow on master. When merges land ~1 minute apart and builds need ~15, no run can ever survive to completion. The cascade is self-sustaining: each merge kills the previous merge's build.

Everything merged today is sitting in master, unbuilt and undeployed

These all merged today and none of them reached production:

21:21  #569  keycloak: bind audit queue to KK.EVENT.#, drop the catch-all
21:01  #588  telemetry: correct service.namespace, node identity, orphaned trace source
21:01  #591  test: curriculum python interpreter
21:00  #574  upload: route upload.received through shared bus, delete hardcoded creds
20:31  #575  scheduler: propagate W3C trace context
20:31  #586  test(quiz): stale front-matter-skip expectation
20:21  #564  quiz: validation gate fails CLOSED
20:03  #573  delete dead autoAck code-execution result consumer
19:59  #553  ci: quiz-generation deploy must pass METADATA_S3_SECRET_KEY

I verified one end-to-end. #569 merged at 21:21:18Z. The spikersoft-keycloak-events service was last updated at 21:14:23Z — seven minutes before the merge. And the live broker confirms the fix is absent: the audit queue is still bound amq.topic → #, the exact catch-all binding #569 removes. The code is correct (KeycloakEventMessaging.cs:86 does call QueueUnbindAsync). It simply never built.

This is the root cause of the whole "merged but not deployed" family

  • #587 — 10 images with healthy :amd64/:arm64v8 and no :latest. Now fully explained: the arch builds pushed, then the run was cancelled before create_manifest could stitch the index. Exactly this mechanism.
  • #582 — "#553 is CLOSED but the fix never reached production." Same cause.
  • #584 — CI never git pulls /mnt/infrastructure. Compounding, but secondary: the deploy job never even runs.
  • #564, #569, #574, #575, #588, #591 — all merged today, all cancelled, none live.

The trap that hides it

The commit status API reports these as failure, while the runs are actually cancelled. Master HEAD shows 30 failing checks — which reads as "the build is broken" to anyone glancing at it. It is not broken. It is being cancelled. That mislabelling is very likely why this has gone unnoticed for so long, and it is worth calling out in whatever fix lands.

Fix — urgent

The concurrency guard is doing the exact opposite of its intent on a busy master.

  1. Immediate: remove cancel-in-progress: true on the master ref (keep it for PR branches, where cancelling superseded builds is genuinely useful). On master, every merge is a distinct artifact that must build, stitch and deploy.
  2. Better: make the publish + create_manifest + deploy jobs non-cancellable — either exclude them from the concurrency group or fold the arch push and manifest stitch into one atomic job. Cancelling a build is safe; cancelling after a partial push is what leaves :latest missing (#587) and a mixed-commit index reachable (#596).
  3. Then re-run the workflows for everything merged today, or master and production stay silently divergent.

Immediate consequence to be aware of

Production is not running today's code. Any ticket closed today on the strength of "the PR merged" should be treated as unverified until its workflow actually completes. That includes several I have been reasoning about — I will re-check the ones I have made claims on.

**QA Team — 🚨 ESCALATION. This is not an intermittent race. Master's CI has been in a total cancellation cascade all day. NOTHING merged today has shipped.** ## The measurement Every one of the **25 most recent workflow runs on master is `cancelled`**. Not failed — **cancelled**. Zero completed. ``` conclusions across last 25 runs: {'cancelled': 25} ``` The cause is arithmetic. Today saw **22 merges to master**, and the gaps between consecutive merges are: ``` gaps between merges (minutes, newest first): [1, 17, 0, 0, 0, 13, 5, 9, 0, 1, 7] median gap: 1 minute merges landing <15 min apart: 10 of 11 ``` Several merges landed in the **same minute**. A multi-arch .NET build (amd64 + arm64v8) takes **10–20+ minutes** — longer right now, because the ARM runner on dreamstream6 is crash-looping (**#579**). With `concurrency.cancel-in-progress: true` keyed on `${{ github.workflow }}-${{ github.ref }}`, **every push to master cancels all in-flight runs for that workflow on master.** When merges land ~1 minute apart and builds need ~15, no run can ever survive to completion. The cascade is self-sustaining: each merge kills the previous merge's build. ## Everything merged today is sitting in master, unbuilt and undeployed These all merged today and **none of them reached production**: ``` 21:21 #569 keycloak: bind audit queue to KK.EVENT.#, drop the catch-all 21:01 #588 telemetry: correct service.namespace, node identity, orphaned trace source 21:01 #591 test: curriculum python interpreter 21:00 #574 upload: route upload.received through shared bus, delete hardcoded creds 20:31 #575 scheduler: propagate W3C trace context 20:31 #586 test(quiz): stale front-matter-skip expectation 20:21 #564 quiz: validation gate fails CLOSED 20:03 #573 delete dead autoAck code-execution result consumer 19:59 #553 ci: quiz-generation deploy must pass METADATA_S3_SECRET_KEY ``` I verified one end-to-end. **#569** merged at **21:21:18Z**. The `spikersoft-keycloak-events` service was last updated at **21:14:23Z** — *seven minutes before the merge*. And the live broker confirms the fix is absent: the audit queue is **still bound `amq.topic` → `#`**, the exact catch-all binding #569 removes. The code is correct (`KeycloakEventMessaging.cs:86` does call `QueueUnbindAsync`). It simply never built. ## This is the root cause of the whole "merged but not deployed" family - **#587** — 10 images with healthy `:amd64`/`:arm64v8` and **no `:latest`**. Now fully explained: the arch builds pushed, then the run was cancelled before `create_manifest` could stitch the index. Exactly this mechanism. - **#582** — "#553 is CLOSED but the fix never reached production." Same cause. - **#584** — CI never `git pull`s `/mnt/infrastructure`. Compounding, but secondary: the deploy job never even runs. - **#564, #569, #574, #575, #588, #591** — all merged today, all cancelled, none live. ## The trap that hides it **The commit status API reports these as `failure`, while the runs are actually `cancelled`.** Master HEAD shows *30 failing checks* — which reads as "the build is broken" to anyone glancing at it. It is not broken. It is being cancelled. That mislabelling is very likely why this has gone unnoticed for so long, and it is worth calling out in whatever fix lands. ## Fix — urgent The concurrency guard is doing the exact opposite of its intent on a busy master. 1. **Immediate:** remove `cancel-in-progress: true` on the `master` ref (keep it for PR branches, where cancelling superseded builds is genuinely useful). On master, every merge is a distinct artifact that must build, stitch and deploy. 2. **Better:** make the publish + `create_manifest` + `deploy` jobs **non-cancellable** — either exclude them from the concurrency group or fold the arch push and manifest stitch into one atomic job. Cancelling a *build* is safe; cancelling *after a partial push* is what leaves `:latest` missing (#587) and a mixed-commit index reachable (#596). 3. **Then re-run** the workflows for everything merged today, or master and production stay silently divergent. ## Immediate consequence to be aware of **Production is not running today's code.** Any ticket closed today on the strength of "the PR merged" should be treated as **unverified** until its workflow actually completes. That includes several I have been reasoning about — I will re-check the ones I have made claims on.
spikerj changed title from [Bug][CI][Deploy] cancel-in-progress on master kills DEPLOY jobs — back-to-back merges cancel each other and fixes silently never ship to [Bug][CI][Deploy][P0] Master CI is in a TOTAL cancellation cascade — 22 merges/day at a 1-min median gap vs 15-min builds means ALL 25 recent runs cancelled; NOTHING merged today has shipped 2026-07-14 21:35:20 +00:00
spikerj commented 2026-07-14 21:36:03 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — RETRACTION of my escalation immediately above. I got the cause wrong.

The repo owner has confirmed: he cancelled those runs manually, because the build system was overwhelmed. They were not killed by concurrency.cancel-in-progress.

So my "total cancellation cascade" reading is withdrawn. {'cancelled': 25} is exactly what a human clicking cancel looks like via the API — identical to what the concurrency guard produces — and I inferred a mechanism from a signal that could not distinguish the two. I should have asked before escalating. I have reverted this ticket's title.

#590's original scope stands as written — cancel-in-progress can kill deploy jobs on back-to-back merges — but I have no evidence that is what happened today, and the "P0 / nothing ever ships" framing was mine and was unfounded. Treat the arithmetic in my previous comment (22 merges, 1-minute median gap) as context for build-system load, not as proof of an automatic cascade.

What is still true, independent of the cause

These were verified against live production, not inferred from CI:

  • #569 is merged but not live. The audit queue on the broker is still bound amq.topic → # — the exact catch-all binding that ticket removes. The code is correct (KeycloakEventMessaging.cs:86 does call QueueUnbindAsync); it just has not been built and deployed. Verified directly against the RabbitMQ management API.
  • Production is not running today's merges. Whatever cancelled them, the effect is the same: master and production have diverged, and the work merged today needs its workflows re-run deliberately.
  • #587's missing :latest manifests are real — 10 images with healthy :amd64/:arm64v8 and no index. A cancelled run after the arch push and before create_manifest produces exactly that, and manual cancellation does it just as effectively as automatic cancellation does. That finding does not depend on which one it was.

The real question this raises

Why is the build system getting overwhelmed? That is worth its own ticket rather than being buried here. 22 merges today, each fanning out across ~23 multi-arch workflows, onto a runner pool where:

  • gitea-runners_arm_v8_2_act_runner (dreamstream6) is crash-looping, exit(1) every ~20s — #579, still open;
  • gitea-runners_amd64_4090_act_runner took an exit(137) OOM-kill 2 days ago;
  • gitea-runners_amd64_laptop_act_runner logged 4× exit(1) ~44h ago.

So the effective runner capacity is materially below the nominal pool, while the fan-out per merge is large. Being forced to cancel by hand is a symptom of that, not a process failure.

I will scope the capacity question separately rather than speculate further here.

**QA Team — RETRACTION of my escalation immediately above. I got the cause wrong.** The repo owner has confirmed: **he cancelled those runs manually**, because the **build system was overwhelmed**. They were not killed by `concurrency.cancel-in-progress`. So my "total cancellation cascade" reading is **withdrawn**. `{'cancelled': 25}` is exactly what a human clicking cancel looks like via the API — identical to what the concurrency guard produces — and I inferred a mechanism from a signal that could not distinguish the two. I should have asked before escalating. I have reverted this ticket's title. **#590's original scope stands as written** — `cancel-in-progress` *can* kill deploy jobs on back-to-back merges — but I have no evidence that is what happened today, and the "P0 / nothing ever ships" framing was mine and was unfounded. Treat the arithmetic in my previous comment (22 merges, 1-minute median gap) as **context for build-system load**, not as proof of an automatic cascade. ## What is still true, independent of the cause These were verified against live production, not inferred from CI: - **#569 is merged but not live.** The audit queue on the broker is **still bound `amq.topic` → `#`** — the exact catch-all binding that ticket removes. The code is correct (`KeycloakEventMessaging.cs:86` does call `QueueUnbindAsync`); it just has not been built and deployed. Verified directly against the RabbitMQ management API. - **Production is not running today's merges.** Whatever cancelled them, the effect is the same: master and production have diverged, and the work merged today needs its workflows re-run deliberately. - **#587's missing `:latest` manifests are real** — 10 images with healthy `:amd64`/`:arm64v8` and no index. A cancelled run *after* the arch push and *before* `create_manifest` produces exactly that, and manual cancellation does it just as effectively as automatic cancellation does. That finding does not depend on which one it was. ## The real question this raises **Why is the build system getting overwhelmed?** That is worth its own ticket rather than being buried here. 22 merges today, each fanning out across ~23 multi-arch workflows, onto a runner pool where: - `gitea-runners_arm_v8_2_act_runner` (dreamstream6) is **crash-looping**, exit(1) every ~20s — **#579**, still open; - `gitea-runners_amd64_4090_act_runner` took an **exit(137) OOM-kill** 2 days ago; - `gitea-runners_amd64_laptop_act_runner` logged 4× exit(1) ~44h ago. So the effective runner capacity is materially below the nominal pool, while the fan-out per merge is large. Being forced to cancel by hand is a symptom of that, not a process failure. I will scope the capacity question separately rather than speculate further here.
spikerj changed title from [Bug][CI][Deploy][P0] Master CI is in a TOTAL cancellation cascade — 22 merges/day at a 1-min median gap vs 15-min builds means ALL 25 recent runs cancelled; NOTHING merged today has shipped to [Bug][CI][Deploy] cancel-in-progress on master kills DEPLOY jobs — back-to-back merges cancel each other and fixes silently never ship 2026-07-14 21:36:03 +00:00
spikerj commented 2026-07-15 18:07:26 +00:00
Author
Owner
Copy Link
Copy Source

Fleet fix up as spikersoft-backend PR #305. Gates cancel-in-progress to ${{ github.ref != 'refs/heads/master' }} on the 14 remaining workflows that still cancelled master deploys (the batch-1 template + notifications pilot); api/blog-media/scheduler already got it in #304. After #305 merges the repo has zero cancel-in-progress: true on master — will close then.

Fleet fix up as spikersoft-backend PR #305. Gates `cancel-in-progress` to `${{ github.ref != 'refs/heads/master' }}` on the 14 remaining workflows that still cancelled master deploys (the batch-1 template + notifications pilot); api/blog-media/scheduler already got it in #304. After #305 merges the repo has **zero** `cancel-in-progress: true` on master — will close then.
spikerj commented 2026-07-15 18:08:34 +00:00
Author
Owner
Copy Link
Copy Source

Resolved in spikersoft-backend PR #305 (merged to master, be410487). Gated cancel-in-progress to ${{ github.ref != 'refs/heads/master' }} on the 14 remaining workflows; api/blog-media/scheduler already carried it from #304. The repo now has zero cancel-in-progress: true on master, so a back-to-back master merge can no longer abort an in-flight deploy between the image push and docker stack deploy. PR runs still cancel their own superseded runs. Closing.

Resolved in spikersoft-backend PR #305 (merged to `master`, `be410487`). Gated `cancel-in-progress` to `${{ github.ref != 'refs/heads/master' }}` on the 14 remaining workflows; api/blog-media/scheduler already carried it from #304. The repo now has zero `cancel-in-progress: true` on master, so a back-to-back master merge can no longer abort an in-flight deploy between the image push and `docker stack deploy`. PR runs still cancel their own superseded runs. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#590