[Infra][CI] every deploy job's /mnt/infrastructure git pull fails under sudo (no credentials) — #584 stale-checkout guard is decorative #625

New Issue

2026-07-17T00:50:38Z

spikerj commented

2026-07-17 00:50:38 +00:00

Found while verifying post-merge CI for the #618 part-2 merges. Affects EVERY backend deploy job, including green ones.

Symptom

Every deploy job logs:

fatal: could not read Username for 'https://git.spikersoft.com': No such device or address
WARNING: git pull failed — deploying the EXISTING /mnt/infrastructure checkout as-is

and then deploys from whatever revision sits on the shared mount. The #584 fix added the git pull --ff-only so deploys can't ship stale stack files — but the pull has (apparently always) failed under sudo: root has no credential helper for git.spikersoft.com and no TTY to prompt, so the || echo WARNING fallback fires on every single deploy.

Current exposure

Right now /mnt/infrastructure is at a730f69 which IS current master — nothing stale has shipped yet. But the guard is decorative: the next infrastructure change will not reach deploys until someone manually pulls the mount, which is exactly the #584 incident mode (28/29 deploys shipping stale revisions while exiting 0).

Fix options

Give the pull working non-interactive auth: repo-scoped deploy token in the mount's remote URL (git remote set-url origin https://<token>@git.spikersoft.com/...), or an SSH deploy key for root; or
Run the pull as the runner user (who has credentials) instead of via sudo; or
At minimum make the job HARD-FAIL when the pull fails and the mount revision differs from origin/master (the warning-and-continue defeats #584's purpose).

Token/credential material goes to OpenBao per the secrets policy — no new plaintext Actions secrets.

Found while verifying post-merge CI for the #618 part-2 merges. Affects EVERY backend deploy job, including green ones. ## Symptom Every `deploy` job logs: ``` fatal: could not read Username for 'https://git.spikersoft.com': No such device or address WARNING: git pull failed — deploying the EXISTING /mnt/infrastructure checkout as-is ``` and then deploys from whatever revision sits on the shared mount. The #584 fix added the `git pull --ff-only` so deploys can't ship stale stack files — but the pull has (apparently always) failed under `sudo`: root has no credential helper for git.spikersoft.com and no TTY to prompt, so the `|| echo WARNING` fallback fires on every single deploy. ## Current exposure Right now `/mnt/infrastructure` is at a730f69 which IS current master — nothing stale has shipped yet. But the guard is decorative: the next infrastructure change will not reach deploys until someone manually pulls the mount, which is exactly the #584 incident mode (28/29 deploys shipping stale revisions while exiting 0). ## Fix options - Give the pull working non-interactive auth: repo-scoped deploy token in the mount's remote URL (`git remote set-url origin https://<token>@git.spikersoft.com/...`), or an SSH deploy key for root; or - Run the pull as the runner user (who has credentials) instead of via `sudo`; or - At minimum make the job HARD-FAIL when the pull fails and the mount revision differs from origin/master (the warning-and-continue defeats #584's purpose). Token/credential material goes to OpenBao per the secrets policy — no new plaintext Actions secrets.

spikerj added the bug label 2026-07-17 00:50:38 +00:00

spikerj referenced this issue

2026-07-17 00:53:19 +00:00

[Infra][CI] lesson-video-processor never deploys: swarm service name 67 chars exceeds 63-char limit (stack+service name doubled) #624

spikerj referenced this issue

2026-07-17 01:04:59 +00:00

[Security][Backend][Books] Private books are listable and downloadable by any authenticated user — /api/book/all and /api/book/{id}/file have no visibility/ownership check #618

spikerj referenced this issue

2026-07-17 01:59:35 +00:00

[Bug][CI][Infra] #584 not fixed — deploy job's git pull runs unprivileged on a root-owned /mnt/infrastructure, so every infra deploy silently ships a stale stack #615

spikerj commented

2026-07-17 01:59:35 +00:00

Duplicate of #615 (same underlying problem: the deploy jobs' /mnt/infrastructure pull has never worked; the sudo added after #615 traded the permission error for a credentials error). Consolidated the current diagnosis and fix options into #615 — follow there. Closing.

spikerj closed this issue

2026-07-17 01:59:35 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#625