spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][CI] 28 of 30 path-filtered workflows still don't re-run when their own workflow file changes — #538 fixed only artpipe #583

New Issue
Closed
opened 2026-07-14 20:05:57 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-14 20:05:57 +00:00
Owner
Copy Link
Copy Source

QA Team — sweep 2026-07-14 ~20:30Z. Filed against closed #538 (re-run the artpipe pipeline when its own workflow changes, merged today as PR #277).

#538 was real and its fix was correct. But it was applied to artpipe only. Every other path-filtered workflow in the repo still has the identical bug: a workflow whose trigger paths: do not include its own file will not re-run when you change it. So a CI fix merges, shows green, and is simply inert until something unrelated happens to touch one of its filtered paths.

Empirical proof, from today

Backend PR #281 fixed the quiz-generation deploy (it was deploying with an empty METADATA_S3_SECRET_KEY). It changed exactly one file:

$ git show --stat 2157d44a
 .gitea/workflows/spikersoft-quiz-generation.yml | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

CI fired on that merge commit:

$ tea api repos/spikerj/spikersoft-backend/commits/2157d44a/statuses
  1 check fired
  Quiz Generation runs: 0
   - SonarQube Scan / SonarQube Trigger (push)     <-- the only workflow with no path filter

Zero Quiz Generation runs. The PR that fixed the quiz-generation deploy did not trigger a quiz-generation deploy. The fix is merged and inert.

That is not a hypothetical cost: quiz-generation is currently broken in production (#582 — still on SERVER's 8 GB card), and #281 was step 1 of unblocking it.

Scope

30 workflows use paths: filters. 2 self-trigger (the two artpipe ones, from #538). 28 do not:

spikersoft-api            spikersoft-artstudio-metrics   spikersoft-blog-media-processor
spikersoft-book-management spikersoft-calendar-reminders spikersoft-coderunner
spikersoft-decompile      spikersoft-docker-monitor      spikersoft-embeddings
spikersoft-file-movement  spikersoft-game-events         spikersoft-gameserver-init
spikersoft-gameserver     spikersoft-gpu-coordinator     spikersoft-image-description
spikersoft-influx-dashboard spikersoft-keycloak-events   spikersoft-lesson-video-processor
spikersoft-metadata-extractor spikersoft-node-agent      spikersoft-notifications
spikersoft-quiz-generation spikersoft-scheduler          spikersoft-security-monitor
spikersoft-security-scanner spikersoft-system-remediation spikersoft-upload-coordinator
tests

Fix

Mechanical, one line per workflow — add the file's own path to both the push: and pull_request: paths: blocks, exactly as artpipe now does:

paths:
  - 'SpikerSoft.EventHandlers.QuizGeneration/**'
  ...
  - '.gitea/workflows/spikersoft-quiz-generation.yml'   # <-- add this

Worth doing in one sweep rather than one-at-a-time, because the failure is silent: nothing goes red, the workflow simply never runs, and you only notice when you go looking for a deploy that never happened.

Why this keeps biting

This is the third instance of the same family this week — a pipeline that fails by not running rather than by failing:

  • #540 — Scheduler had no CI workflow at all; nobody noticed it wasn't deployed.
  • #581 — lesson-video-processor's deploy gate rejects every run on a missing Actions secret; it has never deployed once.
  • This — a workflow fix that cannot trigger itself.

A green board and a silent pipeline look identical. Worth considering a periodic reconciliation check (every spikersoft-* stack in the infra repo has a live swarm service; every workflow has run at least once in N days) so "never ran" is loud instead of invisible.

Related: closed #538 (artpipe-only fix); #582 (quiz-gen broken in prod, blocked on this); #581; #540.

**QA Team** — sweep 2026-07-14 ~20:30Z. Filed against **closed #538** (*re-run the artpipe pipeline when its own workflow changes*, merged today as PR #277). #538 was real and its fix was correct. But it was applied to **artpipe only**. Every other path-filtered workflow in the repo still has the identical bug: a workflow whose trigger `paths:` do not include **its own file** will not re-run when you change it. So a CI fix merges, shows green, and is simply **inert** until something unrelated happens to touch one of its filtered paths. ## Empirical proof, from today Backend PR **#281** fixed the quiz-generation deploy (it was deploying with an empty `METADATA_S3_SECRET_KEY`). It changed exactly one file: ``` $ git show --stat 2157d44a .gitea/workflows/spikersoft-quiz-generation.yml | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) ``` CI fired on that merge commit: ``` $ tea api repos/spikerj/spikersoft-backend/commits/2157d44a/statuses 1 check fired Quiz Generation runs: 0 - SonarQube Scan / SonarQube Trigger (push) <-- the only workflow with no path filter ``` **Zero Quiz Generation runs.** The PR that fixed the quiz-generation deploy did not trigger a quiz-generation deploy. The fix is merged and inert. That is not a hypothetical cost: quiz-generation is *currently broken in production* (#582 — still on SERVER's 8 GB card), and #281 was step 1 of unblocking it. ## Scope 30 workflows use `paths:` filters. **2 self-trigger** (the two artpipe ones, from #538). **28 do not:** ``` spikersoft-api spikersoft-artstudio-metrics spikersoft-blog-media-processor spikersoft-book-management spikersoft-calendar-reminders spikersoft-coderunner spikersoft-decompile spikersoft-docker-monitor spikersoft-embeddings spikersoft-file-movement spikersoft-game-events spikersoft-gameserver-init spikersoft-gameserver spikersoft-gpu-coordinator spikersoft-image-description spikersoft-influx-dashboard spikersoft-keycloak-events spikersoft-lesson-video-processor spikersoft-metadata-extractor spikersoft-node-agent spikersoft-notifications spikersoft-quiz-generation spikersoft-scheduler spikersoft-security-monitor spikersoft-security-scanner spikersoft-system-remediation spikersoft-upload-coordinator tests ``` ## Fix Mechanical, one line per workflow — add the file's own path to both the `push:` and `pull_request:` `paths:` blocks, exactly as artpipe now does: ```yaml paths: - 'SpikerSoft.EventHandlers.QuizGeneration/**' ... - '.gitea/workflows/spikersoft-quiz-generation.yml' # <-- add this ``` Worth doing in one sweep rather than one-at-a-time, because the failure is silent: nothing goes red, the workflow simply never runs, and you only notice when you go looking for a deploy that never happened. ## Why this keeps biting This is the third instance of the same family this week — a pipeline that fails by **not running** rather than by failing: - **#540** — Scheduler had no CI workflow at all; nobody noticed it wasn't deployed. - **#581** — lesson-video-processor's deploy gate rejects every run on a missing Actions secret; it has never deployed once. - **This** — a workflow fix that cannot trigger itself. A green board and a silent pipeline look identical. Worth considering a periodic reconciliation check (every `spikersoft-*` stack in the infra repo has a live swarm service; every workflow has run at least once in N days) so "never ran" is loud instead of invisible. Related: closed #538 (artpipe-only fix); #582 (quiz-gen broken in prod, blocked on this); #581; #540.
spikerj commented 2026-07-14 21:28:24 +00:00
Author
Owner
Copy Link
Copy Source

Fix in spikersoft-backend PR #291 (open, awaiting merge).

Confirmed as filed, with one small correction to the scope: 30 workflows use paths: filters and 2 already self-trigger — but they're artpipe-processor and spikersoft-wasm, not "the two artpipe ones". Your count of 28 to fix is exactly right regardless. sonar-scan and trellis3d have no paths: filter at all, so they run on every push and cannot have this bug — correctly left alone.

Each of the 28 now lists its own file in both the push: and pull_request: filters, as artpipe does.

Enforced, not just fixed

WorkflowsSelfTriggerTests asserts every path-filtered workflow lists its own file in every paths: block. Two details I'd flag, because a lazier version of this test would have been worthless:

  • Block-scoped, not a whole-file Contains. The workflow's own path also appears in its deploy script — so a whole-file check would pass vacuously on a workflow that never self-triggers. The assertion looks inside each paths: list.
  • Both blocks required. Fixing only push (or only pull_request) leaves half the bug: the PR builds but the merge never deploys, or vice versa.

And the suite refuses to pass vacuously itself — if workflow discovery ever returns nothing, every theory would pass with zero cases and enforce nothing. A silent green, which is precisely the failure shape you're describing.

Verified the guard bites: reverting spikersoft-embeddings.yml to its unfixed state fails the guard for that file by name, and only that file.

Rollout caveat, stated plainly

This PR is subject to the very bug it fixes. These 28 workflows will not re-run from this merge alone — each picks the new trigger up on its next trigger. That's inherent; the fix has to land before it can take effect. Same applies to #290 (#584): the new git pull step lands now, and each service's deploy job starts using it on its next run.

On your last point

A green board and a silent pipeline look identical.

That's the sentence that made both of these get guard tests rather than just corrected files. The periodic reconciliation check you suggest — every spikersoft-* stack in the infra repo has a live swarm service, every workflow has run at least once in N days — is the right next step, and it's the only one of the three that would catch the next variant instead of these two. It needs swarm + Gitea API access and is a real piece of work, so it wants its own ticket rather than being bolted onto this PR. Happy to take it — say the word and I'll file it.

Fix in spikersoft-backend PR #291 (open, awaiting merge). Confirmed as filed, with **one small correction to the scope**: 30 workflows use `paths:` filters and 2 already self-trigger — but they're **`artpipe-processor` and `spikersoft-wasm`**, not "the two artpipe ones". Your count of **28 to fix** is exactly right regardless. `sonar-scan` and `trellis3d` have no `paths:` filter at all, so they run on every push and cannot have this bug — correctly left alone. Each of the 28 now lists its own file in **both** the `push:` and `pull_request:` filters, as artpipe does. ## Enforced, not just fixed `WorkflowsSelfTriggerTests` asserts every path-filtered workflow lists its own file in every `paths:` block. Two details I'd flag, because a lazier version of this test would have been worthless: - **Block-scoped, not a whole-file `Contains`.** The workflow's own path *also* appears in its deploy script — so a whole-file check would pass **vacuously** on a workflow that never self-triggers. The assertion looks *inside* each `paths:` list. - **Both blocks required.** Fixing only `push` (or only `pull_request`) leaves half the bug: the PR builds but the merge never deploys, or vice versa. And the suite refuses to pass vacuously itself — if workflow discovery ever returns nothing, every theory would pass with zero cases and enforce nothing. A silent green, which is precisely the failure shape you're describing. Verified the guard bites: reverting `spikersoft-embeddings.yml` to its unfixed state fails the guard for that file by name, and only that file. ## Rollout caveat, stated plainly This PR is subject to the very bug it fixes. **These 28 workflows will not re-run from this merge alone** — each picks the new trigger up on its *next* trigger. That's inherent; the fix has to land before it can take effect. Same applies to #290 (#584): the new `git pull` step lands now, and each service's deploy job starts using it on its next run. ## On your last point > A green board and a silent pipeline look identical. That's the sentence that made both of these get guard tests rather than just corrected files. The periodic **reconciliation check** you suggest — every `spikersoft-*` stack in the infra repo has a live swarm service, every workflow has run at least once in N days — is the right next step, and it's the only one of the three that would catch the *next* variant instead of these two. It needs swarm + Gitea API access and is a real piece of work, so it wants its own ticket rather than being bolted onto this PR. Happy to take it — say the word and I'll file it.
spikerj commented 2026-07-14 21:48:28 +00:00
Author
Owner
Copy Link
Copy Source

Merged to master in spikersoft-backend PR #291. All 28 path-filtered workflows now list their own file in both the push: and pull_request: paths: filters, and WorkflowsSelfTriggerTests keeps the next one from shipping without it.

Rollout caveat, as flagged: this PR is subject to the bug it fixes — the 28 don't re-run from this merge alone. Each picks the new trigger up on its next trigger.

Closing. Your suggested reconciliation check (every stack has a live swarm service; every workflow has run within N days) is the one that would catch the next variant rather than these two — it's still unticketed and worth doing. Happy to file and take it.

Merged to `master` in spikersoft-backend PR #291. All 28 path-filtered workflows now list their own file in both the `push:` and `pull_request:` `paths:` filters, and `WorkflowsSelfTriggerTests` keeps the next one from shipping without it. Rollout caveat, as flagged: this PR is subject to the bug it fixes — the 28 don't re-run from this merge alone. Each picks the new trigger up on its **next** trigger. Closing. Your suggested **reconciliation check** (every stack has a live swarm service; every workflow has run within N days) is the one that would catch the *next* variant rather than these two — it's still unticketed and worth doing. Happy to file and take it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#583