spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

ArtPipeProcessor deploy: /mnt/infrastructure pull fails auth (stale checkout deployed, only a WARNING) + one stack's 'update out of sequence' aborts all later deploys #841

New Issue
Open
opened 2026-07-25 08:17:49 +00:00 by spikerj · 1 comment
spikerj commented 2026-07-25 08:17:49 +00:00
Owner
Copy Link
Copy Source

The ArtPipeProcessor deploy job has two independent faults. Together they mean infrastructure stack changes silently do not reach production, and a single unrelated service can abort every deploy after it.

1. The /mnt/infrastructure pull fails, and the job carries on regardless

fatal: could not read Username for 'https://git.spikersoft.com': No such device or address
WARNING: git pull failed — deploying the EXISTING /mnt/infrastructure checkout as-is
deploying from: 49cf637 Merge pull request 'feat(artpipe): Pixal3D per-model modeling stack (#838)' (#149)

The checkout was one commit stale — missing spikersoft-artpipe-model-textto3d (infra #150, merged) — so the new stack could not have deployed even if the rest of the job had succeeded.

This is #584's failure mode wearing a different hat. #584 added the pull precisely because "28 of 29 deploy jobs applied whatever stale revision sat on the node and still exited 0". The pull is now present — and InfrastructureCheckoutIsPulledTests enforces that it is present — but it fails on authentication and the job only logs a WARNING. So the guard passes while the behaviour it guards against continues.

Two things to fix:

  • Credentials. The SSH deploy user cannot authenticate to git.spikersoft.com over HTTPS. Either give that checkout a credential helper / deploy token, or switch the remote to SSH with a key the deploy user holds.
  • Failing the job. A pull failure should be fatal, not a warning. Deploying a knowingly-stale checkout is worse than not deploying: it reports success while shipping nothing. If there is a reason to tolerate it, the job should at least compare git rev-parse HEAD against the triggering commit and fail on mismatch.

The enforcement test should assert the pull is checked, not merely that the line exists.

2. One service's update out of sequence aborts every later stack

Updating service spikersoft-artpipe-model-sdxl_artpipe-model-sdxl
failed to update service ...: rpc error: code = Unknown desc = update out of sequence
Process exited with status 1

spikersoft-artpipe-model-sdxl is a stood-down 0/0 stack — nothing runs on it. Its stale spec version killed the deploy of every stack ordered after it, including the ones that actually matter.

Same disease as spikersoft-issues#839 on the build side: an all-or-nothing loop where one irrelevant casualty takes out the rest. The fix is the same shape — deploy each stack independently, collect failures, report a summary, and fail the job at the END naming which stacks did not deploy. update out of sequence specifically is retryable (re-read the spec and re-apply), so it deserves a retry before being counted as a failure.

Impact observed

Working around both by hand: docker stack deploy run directly for the one new stack. That is not a fix — every infra stack change is affected, and the failure is silent in exactly the way #584 set out to prevent.

Found while landing spikersoft-issues#832 (Hunyuan3D modeling lane).

The ArtPipeProcessor `deploy` job has two independent faults. Together they mean **infrastructure stack changes silently do not reach production**, and a single unrelated service can abort every deploy after it. ## 1. The `/mnt/infrastructure` pull fails, and the job carries on regardless ``` fatal: could not read Username for 'https://git.spikersoft.com': No such device or address WARNING: git pull failed — deploying the EXISTING /mnt/infrastructure checkout as-is deploying from: 49cf637 Merge pull request 'feat(artpipe): Pixal3D per-model modeling stack (#838)' (#149) ``` The checkout was one commit stale — missing `spikersoft-artpipe-model-textto3d` (infra #150, merged) — so the new stack could not have deployed even if the rest of the job had succeeded. This is #584's failure mode wearing a different hat. #584 added the pull precisely because "28 of 29 deploy jobs applied whatever stale revision sat on the node and still exited 0". The pull is now present — and `InfrastructureCheckoutIsPulledTests` enforces that it is present — but it **fails on authentication** and the job only logs a WARNING. So the guard passes while the behaviour it guards against continues. Two things to fix: - **Credentials.** The SSH deploy user cannot authenticate to `git.spikersoft.com` over HTTPS. Either give that checkout a credential helper / deploy token, or switch the remote to SSH with a key the deploy user holds. - **Failing the job.** A pull failure should be fatal, not a warning. Deploying a knowingly-stale checkout is worse than not deploying: it reports success while shipping nothing. If there is a reason to tolerate it, the job should at least compare `git rev-parse HEAD` against the triggering commit and fail on mismatch. The enforcement test should assert the pull is *checked*, not merely that the line exists. ## 2. One service's `update out of sequence` aborts every later stack ``` Updating service spikersoft-artpipe-model-sdxl_artpipe-model-sdxl failed to update service ...: rpc error: code = Unknown desc = update out of sequence Process exited with status 1 ``` `spikersoft-artpipe-model-sdxl` is a **stood-down 0/0 stack** — nothing runs on it. Its stale spec version killed the deploy of every stack ordered after it, including the ones that actually matter. Same disease as spikersoft-issues#839 on the build side: an all-or-nothing loop where one irrelevant casualty takes out the rest. The fix is the same shape — deploy each stack independently, collect failures, report a summary, and fail the job at the END naming which stacks did not deploy. `update out of sequence` specifically is retryable (re-read the spec and re-apply), so it deserves a retry before being counted as a failure. ## Impact observed Working around both by hand: `docker stack deploy` run directly for the one new stack. That is not a fix — every infra stack change is affected, and the failure is silent in exactly the way #584 set out to prevent. Found while landing spikersoft-issues#832 (Hunyuan3D modeling lane).
spikerj commented 2026-07-29 16:56:07 +00:00
Author
Owner
Copy Link
Copy Source

Re-verified at file level against today's tip (bba34a2e, i.e. after the #894 deploy-list changes) — NOT DONE, both halves.

Half 1 — the pull is still warn-and-continue. .gitea/workflows/spikersoft-artpipe-processor.yml:546-548, verbatim unchanged:

sudo git -C /mnt/infrastructure pull --ff-only \
  || echo "WARNING: git pull failed — deploying the EXISTING /mnt/infrastructure checkout as-is"
echo "deploying from: $(git -C /mnt/infrastructure log -1 --oneline)"

And SpikerSoft.Common.Tests/Deployment/InfrastructureCheckoutIsPulledTests.cs still asserts only presence and ordering — content.Should().Contain(PullCommand) followed by pullAt.Should().BeLessThan(deployAt), plus the "deploying from:" echo. No exit gate, no rev-parse comparison.

The test's own closing comment is worth quoting, because it makes this a decision rather than an oversight: it describes the --ff-only failure as "deliberately warn-and-continue, so an emergency deploy is never blocked." That is precisely the behaviour this ticket asks to be made fatal. So the ticket's phrasing — "the guard passes while the behaviour it guards against continues" — is still exactly true, and someone will need to overrule that stated intent (or reconcile it, e.g. fatal by default with an explicit ALLOW_STALE=1 escape hatch for emergencies).

Half 2 — still all-or-nothing. The deploy shell opens with set -e (:539), and the stack loop at :552-562 iterates 12 GPU stacks plus two more (:563, :568) with no per-stack failure collection, no summary, and no retry for the retryable update out of sequence. One failure aborts every later stack.

The fix shape already exists in this same file. The build job does it correctly: a FAILED_IMAGES accumulator (:201, :236, :289, :304, :327, :344), a summary at :366, and a deferred exit 1 at :399-401 — added for #839. So half 2 is a matter of copying an established pattern from ~200 lines above, not designing one. #894 edited the stack list (textto3d, qrmonster, florence2 are all present now) without touching the error handling.

Remaining: credential material for root's pull plus a fatal/rev-compare gate, with the test strengthened to assert the check rather than the string; and per-stack failure collection with an update out of sequence retry.

Related and mutually reinforcing: #615 (same pull, other workflows — 8 files carry it), #774 (no post-deploy convergence check anywhere), #820 (this exact deploy job never fires on infra-repo changes, so an infra-only env change is inert). Those four together mean a deploy can report success having pulled nothing, deployed a stale file, rolled back, and told no one.

Re-verified at file level against today's tip (`bba34a2e`, i.e. **after** the #894 deploy-list changes) — **NOT DONE, both halves.** **Half 1 — the pull is still warn-and-continue.** `.gitea/workflows/spikersoft-artpipe-processor.yml:546-548`, verbatim unchanged: ``` sudo git -C /mnt/infrastructure pull --ff-only \ || echo "WARNING: git pull failed — deploying the EXISTING /mnt/infrastructure checkout as-is" echo "deploying from: $(git -C /mnt/infrastructure log -1 --oneline)" ``` And `SpikerSoft.Common.Tests/Deployment/InfrastructureCheckoutIsPulledTests.cs` still asserts only **presence and ordering** — `content.Should().Contain(PullCommand)` followed by `pullAt.Should().BeLessThan(deployAt)`, plus the `"deploying from:"` echo. No exit gate, no `rev-parse` comparison. The test's own closing comment is worth quoting, because it makes this a *decision* rather than an oversight: it describes the `--ff-only` failure as **"deliberately warn-and-continue, so an emergency deploy is never blocked."** That is precisely the behaviour this ticket asks to be made fatal. So the ticket's phrasing — "the guard passes while the behaviour it guards against continues" — is still exactly true, and someone will need to overrule that stated intent (or reconcile it, e.g. fatal by default with an explicit `ALLOW_STALE=1` escape hatch for emergencies). **Half 2 — still all-or-nothing.** The deploy shell opens with `set -e` (`:539`), and the stack loop at `:552-562` iterates 12 GPU stacks plus two more (`:563`, `:568`) with no per-stack failure collection, no summary, and no retry for the retryable `update out of sequence`. One failure aborts every later stack. **The fix shape already exists in this same file.** The *build* job does it correctly: a `FAILED_IMAGES` accumulator (`:201`, `:236`, `:289`, `:304`, `:327`, `:344`), a summary at `:366`, and a deferred `exit 1` at `:399-401` — added for #839. So half 2 is a matter of copying an established pattern from ~200 lines above, not designing one. #894 edited the stack list (`textto3d`, `qrmonster`, `florence2` are all present now) without touching the error handling. **Remaining:** credential material for root's pull plus a fatal/rev-compare gate, with the test strengthened to assert the *check* rather than the string; and per-stack failure collection with an `update out of sequence` retry. Related and mutually reinforcing: **#615** (same pull, other workflows — 8 files carry it), **#774** (no post-deploy convergence check anywhere), **#820** (this exact deploy job never fires on infra-repo changes, so an infra-only env change is inert). Those four together mean a deploy can report success having pulled nothing, deployed a stale file, rolled back, and told no one.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#841