spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra] sonarqube_backups stuck in reject-loop — no Sonar backups running (missing placement constraint) #479

New Issue
Closed
opened 2026-07-12 00:00:59 +00:00 by spikerj · 4 comments
spikerj commented 2026-07-12 00:00:59 +00:00
Owner
Copy Link
Copy Source

Summary

sonarqube_backups is stuck in an infinite scheduling reject-loop and no SonarQube backups are being produced (silent data-loss risk).

Symptoms

docker service ps sonarqube_backups shows tasks rejected every ~5 seconds across all nodes:

Rejected  "invalid mount config for type "bind": bind source path does not exist:
           /home/joey/docker/sonarqube/application/data"
Rejected  "... /home/joey/docker/sonarqube/postgres/backups"

Service sits at 0/1 replicas indefinitely and floods the swarm scheduler with rejected tasks on every node.

Root cause

In sonarqube/docker-stack.yml, the backups service has no placement constraint, while the postgresql and sonarqube services are both pinned to node.hostname == SERVER:

docker service inspect sonarqube_backups   -> constraints=[]
docker service inspect sonarqube_sonarqube -> constraints=[node.hostname == SERVER]

The bind-mount source paths (/home/joey/docker/sonarqube/...) only exist on the SERVER node. Without the constraint, swarm keeps scheduling backups onto the other nodes (4090, dreamstream1-7), each of which rejects it because the bind path doesn't exist locally -> endless reject loop, and the backup job never runs.

Impact

  • No PostgreSQL dump (pg_dump) and no application-data tarball are being written -> no recovery point for SonarQube.
  • Continuous rejected-task churn across all 10 swarm nodes.

Proposed fix

Add the same placement constraint to the backups service in sonarqube/docker-stack.yml:

  backups:
    ...
    deploy:
      placement:
        constraints:
          - node.hostname == SERVER

Secondary bug (fix while here)

The backups service's PGPASSWORD value does not match the password on the postgresql/sonarqube services: it has a leading you_ where the others have your_ (a one-character typo — missing r). Even after the placement fix, pg_dump auth will fail until these match. (Password values redacted here — see the stack file.)

Note: the DB password is currently committed in plaintext in sonarqube/docker-stack.yml. Consider moving it to a Docker secret / env file as a follow-up.

Verify after fix

docker stack deploy -c sonarqube/docker-stack.yml sonarqube
docker service ps sonarqube_backups          # should reach Running on SERVER, no Rejected churn
ls -la /home/joey/docker/sonarqube/postgres/backups/   # new .gz appears after BACKUP_INIT_SLEEP (30m)

Filed proactively by automated swarm health check (docker service audit).

## Summary `sonarqube_backups` is stuck in an infinite scheduling reject-loop and **no SonarQube backups are being produced** (silent data-loss risk). ## Symptoms `docker service ps sonarqube_backups` shows tasks rejected every ~5 seconds across all nodes: ``` Rejected "invalid mount config for type "bind": bind source path does not exist: /home/joey/docker/sonarqube/application/data" Rejected "... /home/joey/docker/sonarqube/postgres/backups" ``` Service sits at `0/1` replicas indefinitely and floods the swarm scheduler with rejected tasks on every node. ## Root cause In `sonarqube/docker-stack.yml`, the `backups` service has **no placement constraint**, while the `postgresql` and `sonarqube` services are both pinned to `node.hostname == SERVER`: ``` docker service inspect sonarqube_backups -> constraints=[] docker service inspect sonarqube_sonarqube -> constraints=[node.hostname == SERVER] ``` The bind-mount source paths (`/home/joey/docker/sonarqube/...`) only exist on the **SERVER** node. Without the constraint, swarm keeps scheduling `backups` onto the other nodes (4090, dreamstream1-7), each of which rejects it because the bind path doesn't exist locally -> endless reject loop, and the backup job never runs. ## Impact - No PostgreSQL dump (`pg_dump`) and no application-data tarball are being written -> **no recovery point for SonarQube**. - Continuous rejected-task churn across all 10 swarm nodes. ## Proposed fix Add the same placement constraint to the `backups` service in `sonarqube/docker-stack.yml`: ```yaml backups: ... deploy: placement: constraints: - node.hostname == SERVER ``` ## Secondary bug (fix while here) The `backups` service's `PGPASSWORD` value does not match the password on the `postgresql`/`sonarqube` services: it has a leading `you_` where the others have `your_` (a one-character typo — missing `r`). Even after the placement fix, `pg_dump` auth will fail until these match. (Password values redacted here — see the stack file.) Note: the DB password is currently committed in plaintext in `sonarqube/docker-stack.yml`. Consider moving it to a Docker secret / env file as a follow-up. ## Verify after fix ``` docker stack deploy -c sonarqube/docker-stack.yml sonarqube docker service ps sonarqube_backups # should reach Running on SERVER, no Rejected churn ls -la /home/joey/docker/sonarqube/postgres/backups/ # new .gz appears after BACKUP_INIT_SLEEP (30m) ``` --- _Filed proactively by automated swarm health check (docker service audit)._
spikerj commented 2026-07-12 00:08:31 +00:00
Author
Owner
Copy Link
Copy Source

Resolved in spikersoft-infrastructure PR #28 (merged to master). Four defects fixed, not one: (1) the missing node.hostname == SERVER pin from this ticket; (2) pg_dump targeted -h postgres but the service is named postgresql; (3) PGPASSWORD typo (you_ vs your_); (4) every VAR in command: was interpolated at deploy time from the deploying shell — all empty — so the command rendered as 'pg_dump -d -U | gzip > /-….gz'; now $-escaped so the container env expands them (docker compose config renders clean, 13 warnings → 0). Net: this job has never produced a backup since the stack existed. Ops: sync /mnt/infrastructure, redeploy sonarqube, verify a Running task on SERVER and a dump in postgres/backups/ after the 30m init sleep. Closing.

Resolved in spikersoft-infrastructure PR #28 (merged to master). Four defects fixed, not one: (1) the missing node.hostname == SERVER pin from this ticket; (2) pg_dump targeted -h postgres but the service is named postgresql; (3) PGPASSWORD typo (you_ vs your_); (4) every $VAR in command: was interpolated at deploy time from the deploying shell — all empty — so the command rendered as 'pg_dump -d -U | gzip > /-….gz'; now $$-escaped so the container env expands them (docker compose config renders clean, 13 warnings → 0). Net: this job has never produced a backup since the stack existed. Ops: sync /mnt/infrastructure, redeploy sonarqube, verify a Running task on SERVER and a dump in postgres/backups/ after the 30m init sleep. Closing.
spikerj commented 2026-07-13 00:52:05 +00:00
Author
Owner
Copy Link
Copy Source

Reopening — the fix (198ad66, pin backups to SERVER) is merged to master but the sonarqube stack was NOT redeployed. Evidence right now (2026-07-13):

  • Deployed service UpdatedAt: 2026-07-12 07:39:37 UTC (predates the fix commit)
  • docker service inspect sonarqube_backups → Placement.Constraints: [] (no SERVER pin in the live spec)
  • Task reject-loop is active: Rejected ... invalid mount config for type "bind": bind source path does not exist: /home/joey/docker/sonarqube/application/data, cycling every ~5s across laptop-server/dreamstream1/dreamstream5.

Needs: docker stack deploy of the sonarqube stack from current master (from /mnt/infrastructure). Will confirm + re-close once the deployed spec shows the constraint and the reject-loop stops.

Reopening — the fix (198ad66, pin backups to SERVER) is merged to master but the sonarqube stack was NOT redeployed. Evidence right now (2026-07-13): - Deployed service UpdatedAt: 2026-07-12 07:39:37 UTC (predates the fix commit) - `docker service inspect sonarqube_backups` → `Placement.Constraints: []` (no SERVER pin in the live spec) - Task reject-loop is active: `Rejected ... invalid mount config for type "bind": bind source path does not exist: /home/joey/docker/sonarqube/application/data`, cycling every ~5s across laptop-server/dreamstream1/dreamstream5. Needs: `docker stack deploy` of the sonarqube stack from current master (from /mnt/infrastructure). Will confirm + re-close once the deployed spec shows the constraint and the reject-loop stops.
spikerj reopened this issue 2026-07-13 00:52:05 +00:00
spikerj commented 2026-07-14 02:54:59 +00:00
Author
Owner
Copy Link
Copy Source

Code-side this is fixed on master (198ad66 — backups pinned node.hostname == SERVER like its siblings, plus the $$ escaping so pg_dump's vars survive the compose parser). What remains is ops:

  1. cd /mnt/infrastructure/sonarqube && sudo docker stack deploy -c docker-stack.yml sonarqube (manager)
  2. Verify convergence: docker service ps sonarqube_backups → one task Running on SERVER, reject-loop gone.
  3. Backups prove themselves after BACKUP_INIT_SLEEP (30 m): expect a fresh sonarqube-postgres-backup-*.gz under /home/joey/docker/sonarqube/postgres/backups on SERVER, then daily.

Worth noting while here: PGPASSWORD sits in plaintext in the stack file — prime OpenBao #545/#546 migration candidate once the vault is initialized.

— macbook-claude-session

Code-side this is **fixed on master** (`198ad66` — backups pinned `node.hostname == SERVER` like its siblings, plus the `$$` escaping so pg_dump's vars survive the compose parser). What remains is ops: 1. `cd /mnt/infrastructure/sonarqube && sudo docker stack deploy -c docker-stack.yml sonarqube` (manager) 2. Verify convergence: `docker service ps sonarqube_backups` → one task **Running on SERVER**, reject-loop gone. 3. Backups prove themselves after `BACKUP_INIT_SLEEP` (30 m): expect a fresh `sonarqube-postgres-backup-*.gz` under `/home/joey/docker/sonarqube/postgres/backups` on SERVER, then daily. Worth noting while here: `PGPASSWORD` sits in plaintext in the stack file — prime **OpenBao #545/#546** migration candidate once the vault is initialized. — macbook-claude-session
spikerj commented 2026-07-22 02:57:50 +00:00
Author
Owner
Copy Link
Copy Source

Board-sweep closing on merged evidence: both fixes in (infra #28 SERVER pin + pg_dump host/password/$$-escaping; #75 correct backups bind path). Caveat: whether a backup ARTIFACT now lands is an ops check we can't do from the repos — if the next backup window produces nothing, reopen with the cron log.

Board-sweep closing on merged evidence: both fixes in (infra #28 SERVER pin + pg_dump host/password/$$-escaping; #75 correct backups bind path). Caveat: whether a backup ARTIFACT now lands is an ops check we can't do from the repos — if the next backup window produces nothing, reopen with the cron log.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#479