spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][CI][Infra] #584 not fixed — deploy job's git pull runs unprivileged on a root-owned /mnt/infrastructure, so every infra deploy silently ships a stale stack #615

New Issue
Open
opened 2026-07-16 08:05:35 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-16 08:05:35 +00:00
Owner
Copy Link
Copy Source

Follow-on to #584 (which is NOT actually preventing stale deploys).

Problem

#584 made every deploy job git -C /mnt/infrastructure pull --ff-only before deploying. But that pull runs as the unprivileged SSH deploy user, while /mnt/infrastructure is root-owned (humans refresh it with sudo git pull). So the CI pull fails:

error: cannot open .git/FETCH_HEAD: Permission denied

The || echo "WARNING…" fallback swallows it (exit 0), and the job deploys whatever revision root last hand-pulled — silently, green.

Live evidence (2026-07-16)

spikersoft-embeddings #527 (UseS3 + drop the /app/ai bind) merged to both repos. The Embeddings deploy ran green 5× and the live service kept the old bound stack. Deploy job log (run 7896):

error: cannot open .git/FETCH_HEAD: Permission denied
WARNING: git pull failed — deploying the EXISTING /mnt/infrastructure checkout as-is
deploying from: 88df147 … (#118)   # pre-#527

It only deployed correctly after a human ran sudo git pull on dreamstream1 by hand.

Impact

Every infra stack-file change is silently not deployed until someone hand-pulls the deploy host. This is the same class as #582/#553 (fix merged + closed, never ran in prod).

Fix

spikersoft-backend PR: prepend sudo to the pull in all 29 deploy workflows (matches the sudo docker stack deploy right below it), and tighten InfrastructureCheckoutIsPulledTests to require the sudo form. Strictly safe: fixes the pull where sudo-git is permitted; where it isn't, same warn-and-continue as today.

Host prerequisite to verify: the deploy SSH user's sudoers must permit sudo git (it already permits sudo docker). If it's command-restricted to docker, either broaden it or chown the checkout to the deploy user. Until then deploys are no worse than today (warn-and-continue).

Follow-on to #584 (which is NOT actually preventing stale deploys). ## Problem #584 made every deploy job `git -C /mnt/infrastructure pull --ff-only` before deploying. But that pull runs as the **unprivileged SSH deploy user**, while `/mnt/infrastructure` is **root-owned** (humans refresh it with `sudo git pull`). So the CI pull fails: ``` error: cannot open .git/FETCH_HEAD: Permission denied ``` The `|| echo "WARNING…"` fallback swallows it (exit 0), and the job deploys **whatever revision root last hand-pulled** — silently, green. ## Live evidence (2026-07-16) `spikersoft-embeddings` #527 (UseS3 + drop the /app/ai bind) merged to both repos. The Embeddings deploy ran green **5×** and the live service kept the old bound stack. Deploy job log (run 7896): ``` error: cannot open .git/FETCH_HEAD: Permission denied WARNING: git pull failed — deploying the EXISTING /mnt/infrastructure checkout as-is deploying from: 88df147 … (#118) # pre-#527 ``` It only deployed correctly after a human ran `sudo git pull` on dreamstream1 by hand. ## Impact **Every** infra stack-file change is silently not deployed until someone hand-pulls the deploy host. This is the same class as #582/#553 (fix merged + closed, never ran in prod). ## Fix spikersoft-backend PR: prepend `sudo` to the pull in all 29 deploy workflows (matches the `sudo docker stack deploy` right below it), and tighten `InfrastructureCheckoutIsPulledTests` to require the sudo form. Strictly safe: fixes the pull where sudo-git is permitted; where it isn't, same warn-and-continue as today. **Host prerequisite to verify:** the deploy SSH user's sudoers must permit `sudo git` (it already permits `sudo docker`). If it's command-restricted to docker, either broaden it or `chown` the checkout to the deploy user. Until then deploys are no worse than today (warn-and-continue).
spikerj commented 2026-07-17 01:59:35 +00:00
Author
Owner
Copy Link
Copy Source

Updated diagnosis (2026-07-17, from #625 — closing that as a duplicate of this): the pull now runs with sudo in the deploy jobs, which fixed the root-owned-checkout permission error described here, but it fails differently — root has no git credentials for git.spikersoft.com and no TTY:

fatal: could not read Username for 'https://git.spikersoft.com': No such device or address
WARNING: git pull failed — deploying the EXISTING /mnt/infrastructure checkout as-is

Every deploy job (green ones included) hits this, so the #584 guard still never actually pulls. /mnt/infrastructure currently happens to be AT master (a730f69… lineage), so nothing stale has shipped yet — but infrastructure PR #123 (lesson-video-processor service-name fix, #624) is merged and cannot roll out until this is fixed or the mount is pulled manually.

Fix options (from #625): repo-scoped deploy token in the mount's remote URL, an SSH deploy key for root, running the pull as the credentialed runner user, or at minimum hard-failing when the pull fails AND the mount differs from origin/master. Credential material goes to OpenBao per policy.

Updated diagnosis (2026-07-17, from #625 — closing that as a duplicate of this): the pull now runs **with sudo** in the deploy jobs, which fixed the root-owned-checkout permission error described here, but it fails differently — root has no git credentials for git.spikersoft.com and no TTY: ``` fatal: could not read Username for 'https://git.spikersoft.com': No such device or address WARNING: git pull failed — deploying the EXISTING /mnt/infrastructure checkout as-is ``` Every deploy job (green ones included) hits this, so the #584 guard still never actually pulls. `/mnt/infrastructure` currently happens to be AT master (a730f69… lineage), so nothing stale has shipped yet — but infrastructure PR #123 (lesson-video-processor service-name fix, #624) is merged and cannot roll out until this is fixed or the mount is pulled manually. Fix options (from #625): repo-scoped deploy token in the mount's remote URL, an SSH deploy key for root, running the pull as the credentialed runner user, or at minimum hard-failing when the pull fails AND the mount differs from origin/master. Credential material goes to OpenBao per policy.
spikerj commented 2026-07-29 15:58:01 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — the literal complaint was fixed; the impact persists. Staying open.

Fixed: every workflow carrying the pull now uses the sudo form. git grep -l "sudo git -C /mnt/infrastructure pull" returns 8 files, and git grep -n "^\s*git -C /mnt/infrastructure pull" (the bare unprivileged form) returns nothing. The enforcement test was tightened as promised: SpikerSoft.Common.Tests/Deployment/InfrastructureCheckoutIsPulledTests.cs:37 now pins PullCommand = "sudo git -C /mnt/infrastructure pull", with the rationale at :32.

Still broken, per comment 7657's updated diagnosis: root has no git credentials, so the pull now fails with could not read Username — and the || echo "WARNING…" still swallows it. Every infra deploy still ships whatever is already on disk. Confirming nothing was done about the credentials: git grep -rn "GIT_ASKPASS\|credential.helper\|extraheader\|git remote set-url" origin/master -- '.gitea/workflows/*.yml' → no matches. No deploy token in the remote URL, no SSH key for root, no hard-fail on divergence.

So the ticket's title is now inaccurate (it isn't unprivileged any more) but its substance is unchanged: infra deploys silently ship stale stacks.

This is currently blocking at least one other ticket. #582 (quiz-generation placement) cannot be verified or resolved because spikersoft-backend/.gitea/workflows/spikersoft-quiz-generation.yml:109-110 carries this exact swallow-the-failure pattern, so its deploys go green against a possibly-stale checkout. #841 tracks the same pattern from the ArtPipeProcessor side. Worth treating this as infrastructure-wide rather than one workflow's problem.

Remaining:

  1. Credential material for root's pull — per the secrets rule, sourced from OpenBao rather than a committed token.
  2. At minimum, hard-fail when the pull fails and the checkout differs from origin/master. Right now a failed pull is indistinguishable from a successful no-op, which is the property that makes this invisible.

One unresolved gap in my own evidence, flagged rather than glossed: I found the pull in 8 workflow files, not the 29 this ticket describes. I didn't determine whether the others dropped the line, moved it into a shared/composite action, or never had it — worth a look, because if some deploy paths have no pull at all that's a different (and quieter) staleness bug.

Audited against `origin/master` — **the literal complaint was fixed; the impact persists.** Staying open. **Fixed:** every workflow carrying the pull now uses the sudo form. `git grep -l "sudo git -C /mnt/infrastructure pull"` returns 8 files, and `git grep -n "^\s*git -C /mnt/infrastructure pull"` (the bare unprivileged form) returns **nothing**. The enforcement test was tightened as promised: `SpikerSoft.Common.Tests/Deployment/InfrastructureCheckoutIsPulledTests.cs:37` now pins `PullCommand = "sudo git -C /mnt/infrastructure pull"`, with the rationale at `:32`. **Still broken, per comment 7657's updated diagnosis:** root has no git credentials, so the pull now fails with `could not read Username` — and the `|| echo "WARNING…"` still swallows it. Every infra deploy still ships whatever is already on disk. Confirming nothing was done about the credentials: `git grep -rn "GIT_ASKPASS\|credential.helper\|extraheader\|git remote set-url" origin/master -- '.gitea/workflows/*.yml'` → **no matches**. No deploy token in the remote URL, no SSH key for root, no hard-fail on divergence. So the ticket's title is now inaccurate (it isn't unprivileged any more) but its *substance* is unchanged: infra deploys silently ship stale stacks. **This is currently blocking at least one other ticket.** #582 (quiz-generation placement) cannot be verified or resolved because `spikersoft-backend/.gitea/workflows/spikersoft-quiz-generation.yml:109-110` carries this exact swallow-the-failure pattern, so its deploys go green against a possibly-stale checkout. #841 tracks the same pattern from the ArtPipeProcessor side. Worth treating this as infrastructure-wide rather than one workflow's problem. **Remaining:** 1. Credential material for root's pull — per the secrets rule, sourced from OpenBao rather than a committed token. 2. At minimum, **hard-fail when the pull fails and the checkout differs from `origin/master`**. Right now a failed pull is indistinguishable from a successful no-op, which is the property that makes this invisible. One unresolved gap in my own evidence, flagged rather than glossed: I found the pull in **8** workflow files, not the 29 this ticket describes. I didn't determine whether the others dropped the line, moved it into a shared/composite action, or never had it — worth a look, because if some deploy paths have no pull at all that's a different (and quieter) staleness bug.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#615