spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[CI][ArtPipe] ModelEnvImages has no concurrency guard — concurrent runs race the same :latest tags (observed live) #536

New Issue
Closed
opened 2026-07-13 02:25:04 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-13 02:25:04 +00:00
Owner
Copy Link
Copy Source

Observed 2026-07-13 ~02:25Z: runs 10775 (Phase-2 merge f439de2, on the laptop runner, mid-batch) and 10777 (caching fix b76f743, on the 4090 runner) are building concurrently, both pushing artpipe-base:* and artpipe-model-env-{safetycheck,blender,sdxllightning,triposr}:latest.

Consequences:

  • :latest and the ARTPIPE_SHA provenance labels (d1f6ba7) end up from whichever run pushes LAST per image — an older commit can clobber a newer one's tags (mostly-benign now since the model Dockerfiles are identical across the two commits, but silently wrong provenance).
  • Duplicate 60 GB-class builds burn both amd64 runners + disk (#514) for ~an hour.
  • Recurs any time two artpipe commits merge within a build window (~1h) — the team is moving fast, so that's often.

Fixes, cheapest first:

  1. concurrency: { group: model-env-images, cancel-in-progress: true } at workflow level, if the Gitea version honors it (verify — support landed in recent Gitea; if unsupported it's silently ignored, so test with two dummy pushes).
  2. Landing #534 (runs-on: [ubuntu-amd64-4090]) serializes runs as a side effect — a single act_runner executes one job at a time, so later runs queue instead of racing.
  3. Guard step: query GET /api/v1/repos/spikerj/spikersoft-artpipe/actions/tasks for an in-flight ModelEnvImages run and exit early.

Related: #514, #515, #516, #534.

Observed 2026-07-13 ~02:25Z: runs **10775** (Phase-2 merge f439de2, on the laptop runner, mid-batch) and **10777** (caching fix b76f743, on the 4090 runner) are building **concurrently**, both pushing `artpipe-base:*` and `artpipe-model-env-{safetycheck,blender,sdxllightning,triposr}:latest`. Consequences: - `:latest` and the ARTPIPE_SHA provenance labels (d1f6ba7) end up from whichever run pushes LAST per image — an older commit can clobber a newer one's tags (mostly-benign now since the model Dockerfiles are identical across the two commits, but silently wrong provenance). - Duplicate 60 GB-class builds burn both amd64 runners + disk (#514) for ~an hour. - Recurs any time two artpipe commits merge within a build window (~1h) — the team is moving fast, so that's often. Fixes, cheapest first: 1. `concurrency: { group: model-env-images, cancel-in-progress: true }` at workflow level, if the Gitea version honors it (verify — support landed in recent Gitea; if unsupported it's silently ignored, so test with two dummy pushes). 2. Landing #534 (`runs-on: [ubuntu-amd64-4090]`) serializes runs as a side effect — a single act_runner executes one job at a time, so later runs queue instead of racing. 3. Guard step: query `GET /api/v1/repos/spikerj/spikersoft-artpipe/actions/tasks` for an in-flight ModelEnvImages run and exit early. Related: #514, #515, #516, #534.
spikerj commented 2026-07-13 20:39:18 +00:00
Author
Owner
Copy Link
Copy Source

artpipe PR #18: workflow-level concurrency group with cancel-in-progress (ticket fix option 1) — newest commit owns :latest + provenance; superseded runs cancel instead of clobbering. Option 2 side-effect already live since artpipe PR #16 (4090 exclusive runner serializes); the guard covers the documented urgent-flip escape hatch back to the two-runner pool. Verification (two dummy pushes, first shows Cancelled) best done after the current push batch. Note for #534: the ModelEnvImages half is done via PR #16 — remaining is routing the backend tier-3 finals loop.

artpipe PR #18: workflow-level concurrency group with cancel-in-progress (ticket fix option 1) — newest commit owns :latest + provenance; superseded runs cancel instead of clobbering. Option 2 side-effect already live since artpipe PR #16 (4090 exclusive runner serializes); the guard covers the documented urgent-flip escape hatch back to the two-runner pool. Verification (two dummy pushes, first shows Cancelled) best done after the current push batch. Note for #534: the ModelEnvImages half is done via PR #16 — remaining is routing the backend tier-3 finals loop.
spikerj commented 2026-07-13 20:52:32 +00:00
Author
Owner
Copy Link
Copy Source

QA watch 2026-07-13 ~20:53Z — guard merged (a3fba37) and version-compatible. Implementation: workflow-level concurrency group 'model-env-images' with cancel-in-progress: true — newest commit owns :latest, superseded runs cancelled. The commit's own caveat ('silently ignored when unsupported') checks out OK here: this instance is Gitea 1.26.4, and workflow concurrency support landed in the 1.24 line — so the key should be honored, not ignored.

Close condition: observe one real supersede — next time two ModelEnvImages runs would overlap (rapid merge pair or manual double-dispatch), confirm the older run flips to cancelled instead of racing. I'll watch for a natural occurrence; a deliberate two-dummy-push test works too if someone wants it closed sooner. Also note the 4090 single-runner routing (#534) currently serializes these runs anyway, so the guard is the backstop for future pool changes rather than a live-fire mechanism today.

QA watch 2026-07-13 ~20:53Z — guard merged (a3fba37) and version-compatible. Implementation: workflow-level concurrency group 'model-env-images' with cancel-in-progress: true — newest commit owns :latest, superseded runs cancelled. The commit's own caveat ('silently ignored when unsupported') checks out OK here: this instance is Gitea 1.26.4, and workflow concurrency support landed in the 1.24 line — so the key should be honored, not ignored. Close condition: observe one real supersede — next time two ModelEnvImages runs would overlap (rapid merge pair or manual double-dispatch), confirm the older run flips to cancelled instead of racing. I'll watch for a natural occurrence; a deliberate two-dummy-push test works too if someone wants it closed sooner. Also note the 4090 single-runner routing (#534) currently serializes these runs anyway, so the guard is the backstop for future pool changes rather than a live-fire mechanism today.
spikerj commented 2026-07-14 03:04:03 +00:00
Author
Owner
Copy Link
Copy Source

Tonight's runs add the live evidence on top of the version check: build_env_images run 6612 (the #537 weight-patterns merge) ran to success under the guard — built + pushed env-sdxllightning AND env-triposr, cascade dispatched, no interference. And the cancel semantics of the exact same stanza were exercised for real on the backend fleet tonight (see #548: superseded arm64v8 runs cancelled at 20:44Z rather than racing :latest). Between Gitea 1.26.4 support confirmed, a green run under the guard, and the stanza's cancel path proven fleet-wide, I'd call the two-dummy-push ceremony redundant — recommend closing.

— macbook-claude-session

Tonight's runs add the live evidence on top of the version check: `build_env_images` run [6612](https://git.spikersoft.com/spikerj/spikersoft-artpipe/actions/runs/6612) (the #537 weight-patterns merge) ran to **success under the guard** — built + pushed env-sdxllightning AND env-triposr, cascade dispatched, no interference. And the cancel semantics of the exact same stanza were exercised for real on the backend fleet tonight (see #548: superseded arm64v8 runs cancelled at 20:44Z rather than racing :latest). Between Gitea 1.26.4 support confirmed, a green run under the guard, and the stanza's cancel path proven fleet-wide, I'd call the two-dummy-push ceremony redundant — recommend closing. — macbook-claude-session
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#536