spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[CI][Backend] Tier-3 artpipe finals builds keep getting superseded/cancelled before pushing — dino fix (#702) never reached the registry #717

New Issue
Open
opened 2026-07-19 01:29:52 +00:00 by spikerj · 1 comment
spikerj commented 2026-07-19 01:29:52 +00:00
Owner
Copy Link
Copy Source

Observed 2026-07-18 22:44Z → 2026-07-19 01:25Z: after artpipe PR #26 (the #702 dino fix) merged and the tier-2 env image published (22:29Z, 1h47m build), the cascade-dispatched tier-3 spikersoft-artpipe-processor build started at 22:44Z — and then two more master merges (backend PRs #420/#421) triggered fresh runs. Both long-running build containers (tasks 19064/19094) were killed simultaneously at ~01:25Z with nothing pushed: the registry's newest artpipe-model-prodstages pin tag is still 9f33587-9984a25 (created 20:03Z, pre-dino env). Verified via the packages API — no *-e668b00 tag exists.

The trap: the finals build takes ~1-2h (22-36 GB pulls/pushes over the #538 hairpin), and spikersoft-artpipe-processor.yml triggers on most backend merges (Common/Data/Contracts/Infrastructure paths). On an active merge day, ref-scoped concurrency (#548) cancels the in-flight run before it can push — so the tier-3 image can lag the env image INDEFINITELY. Tonight that means the #702 modeling fix has been merged for 5 hours and still isn't deployable.

Fix options (pick one):

  1. Exempt this workflow from cancel-in-progress (cancel-in-progress: false in its concurrency stanza) — newest-wins matters for cheap builds, but for a 2h build a completed stale image + immediate re-run beats an eternal cancel loop.
  2. Narrow its paths triggers (most backend merges don't change what the finals bake — the thin code layer is ArtPipeProcessor-only) so it fires far less often.
  3. Split the finals build into its own dispatch-only workflow with a queue-don't-cancel policy, cascaded from tier 2 as today.

Interim: building tier-3 manually on the 4090 (same daemon CI uses) to unblock the modeling lane tonight.

Related: #702 (the stranded fix), #534 (runner routing), #536/#548 (the concurrency guards — working as designed, wrong policy for this workflow), #538 (hairpin).

**Observed 2026-07-18 22:44Z → 2026-07-19 01:25Z:** after artpipe PR #26 (the #702 dino fix) merged and the tier-2 env image published (22:29Z, 1h47m build), the cascade-dispatched tier-3 `spikersoft-artpipe-processor` build started at 22:44Z — and then two more master merges (backend PRs #420/#421) triggered fresh runs. Both long-running build containers (tasks 19064/19094) were killed simultaneously at ~01:25Z with **nothing pushed**: the registry's newest `artpipe-model-prodstages` pin tag is still `9f33587-9984a25` (created 20:03Z, pre-dino env). Verified via the packages API — no `*-e668b00` tag exists. **The trap:** the finals build takes ~1-2h (22-36 GB pulls/pushes over the #538 hairpin), and `spikersoft-artpipe-processor.yml` triggers on most backend merges (Common/Data/Contracts/Infrastructure paths). On an active merge day, ref-scoped concurrency (#548) cancels the in-flight run before it can push — so the tier-3 image can lag the env image INDEFINITELY. Tonight that means the #702 modeling fix has been merged for 5 hours and still isn't deployable. **Fix options (pick one):** 1. Exempt this workflow from cancel-in-progress (`cancel-in-progress: false` in its concurrency stanza) — newest-wins matters for cheap builds, but for a 2h build a completed stale image + immediate re-run beats an eternal cancel loop. 2. Narrow its `paths` triggers (most backend merges don't change what the finals bake — the thin code layer is ArtPipeProcessor-only) so it fires far less often. 3. Split the finals build into its own dispatch-only workflow with a queue-don't-cancel policy, cascaded from tier 2 as today. **Interim:** building tier-3 manually on the 4090 (same daemon CI uses) to unblock the modeling lane tonight. Related: #702 (the stranded fix), #534 (runner routing), #536/#548 (the concurrency guards — working as designed, wrong policy for this workflow), #538 (hairpin).
spikerj commented 2026-07-29 16:04:22 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — NOT DONE, none of the three fix options. And the audit turned up something sharper than "nobody got to it":

This is the one backend workflow with no concurrency policy at all. spikersoft-artpipe-processor.yml contains zero occurrences of concurrency, and git log -S"concurrency" on that path returns no commits — the stanza was never added or removed. Meanwhile every other backend workflow carries group: ${{ github.workflow }}-${{ github.ref }} with cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}, landed fleet-wide on 2026-07-15 by bd5f63c7 ("gate cancel-in-progress to non-master refs, fleet-wide (#590)").

So the fleet-wide fix for exactly this problem shipped two weeks ago and skipped the one workflow where it matters most — the multi-hour tier-3 build. Option 1 isn't new work; it's applying the existing pattern to the file that was missed.

Option 2 went backwards. Diffing the file at this ticket's filing date (7311772a, 07-19) against master shows SpikerSoft.Storage/** was added to both the push and pull_request path lists. SpikerSoft.Common/**, SpikerSoft.Data/**, SpikerSoft.Contracts.Workers/** and SpikerSoft.EventHandlers.Infrastructure/** all remain — so the trigger surface got wider, and it still fires on most backend merges.

Option 3 (dispatch-only split) not done; push: and pull_request: triggers remain alongside workflow_dispatch.

Unrelated hardening did land on this file since — loopback direct-registry (a33db6a2/d4254cfb, for #856) and pin-tag pruning (cd3113da) — which shortens pulls but does nothing about cancel/supersede.

Not verifiable from git: whether the #702 dino fix ever reached the registry. That's a packages-API question — look for an artpipe-model-prodstages pin tag newer than 9f33587-9984a25. Worth noting I did confirm on #519 that artpipe-model-prodstages now has four versions with the newest built 2026-07-29T15:43Z, so tier-3 finals are building successfully today. Whether that specific dino fix is in them still needs checking.

Remaining: add the concurrency stanza matching the fleet pattern (smallest, highest-value change), then decide whether to narrow the paths or split to dispatch-only.

Audited against `origin/master` — **NOT DONE, none of the three fix options.** And the audit turned up something sharper than "nobody got to it": **This is the one backend workflow with no concurrency policy at all.** `spikersoft-artpipe-processor.yml` contains **zero** occurrences of `concurrency`, and `git log -S"concurrency"` on that path returns **no commits** — the stanza was never added or removed. Meanwhile every *other* backend workflow carries `group: ${{ github.workflow }}-${{ github.ref }}` with `cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}`, landed fleet-wide on 2026-07-15 by `bd5f63c7` ("gate cancel-in-progress to non-master refs, fleet-wide (#590)"). So the fleet-wide fix for exactly this problem shipped two weeks ago and **skipped the one workflow where it matters most** — the multi-hour tier-3 build. Option 1 isn't new work; it's applying the existing pattern to the file that was missed. **Option 2 went backwards.** Diffing the file at this ticket's filing date (`7311772a`, 07-19) against master shows `SpikerSoft.Storage/**` was **added** to both the push and pull_request path lists. `SpikerSoft.Common/**`, `SpikerSoft.Data/**`, `SpikerSoft.Contracts.Workers/**` and `SpikerSoft.EventHandlers.Infrastructure/**` all remain — so the trigger surface got *wider*, and it still fires on most backend merges. **Option 3** (dispatch-only split) not done; `push:` and `pull_request:` triggers remain alongside `workflow_dispatch`. Unrelated hardening did land on this file since — loopback direct-registry (`a33db6a2`/`d4254cfb`, for #856) and pin-tag pruning (`cd3113da`) — which shortens pulls but does nothing about cancel/supersede. **Not verifiable from git:** whether the #702 dino fix ever reached the registry. That's a packages-API question — look for an `artpipe-model-prodstages` pin tag newer than `9f33587-9984a25`. Worth noting I *did* confirm on #519 that `artpipe-model-prodstages` now has four versions with the newest built 2026-07-29T15:43Z, so tier-3 finals are building successfully today. Whether that specific dino fix is in them still needs checking. **Remaining:** add the concurrency stanza matching the fleet pattern (smallest, highest-value change), then decide whether to narrow the paths or split to dispatch-only.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#717