[Bug][Infra] spikersoft-node-agent running on 1/10 nodes — rejected on missing /var/log/journal and /etc/timezone binds #480

New Issue

2026-07-12T00:00:59Z

spikerj commented

2026-07-12 00:00:59 +00:00

Summary

spikersoft-node-agent is a global service but is running on only 1 of 10 swarm nodes. On the other nodes its tasks are rejected because required read-only host bind-mount paths don't exist. The node agent (host journal / metrics collector) is therefore absent across most of the fleet — an observability blind spot.

Symptoms

docker service ps spikersoft-node-agent_spikersoft-node-agent:

laptop-server   Rejected   "bind source path does not exist: /etc/timezone"
dreamstream2    Rejected   "bind source path does not exist: /var/log/journal"
dreamstream4    Rejected   "bind source path does not exist: /var/log/journal"
dreamstream5    Rejected   "bind source path does not exist: /var/log/journal"
4090            Running

Only 4090 is up. Rejected tasks recur continuously (global service keeps retrying).

Root cause

spikersoft-node-agent/docker-stack.yml (mode: global) hard-binds host paths read-only:

- /var/log/journal:/var/log/journal:ro
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro

/var/log/journal only exists when persistent journald is enabled. The repo already ships spikersoft-node-agent/scripts/enable-persistent-journald.sh for this, but it has not been run on dreamstream2, dreamstream4, dreamstream5.
/etc/timezone is missing on laptop-server (not created by the journald script; some distros/minimal installs lack it).

Impact

Node agent not running on ≥4/10 nodes -> no host journal tailing / node metrics from those hosts. Continuous rejected-task churn in the scheduler.

Proposed fix

Run the existing helper on the affected nodes (documented as "run on EVERY node"):

for h in dreamstream2 dreamstream4 dreamstream5; do
  scp spikersoft-node-agent/scripts/enable-persistent-journald.sh $h:/tmp/ \
    && ssh $h 'sudo bash /tmp/enable-persistent-journald.sh'
done

Ensure /etc/timezone on laptop-server (e.g. sudo sh -c 'echo Etc/UTC > /etc/timezone', or sudo dpkg-reconfigure tzdata). Consider extending enable-persistent-journald.sh to also create /etc/timezone if absent so bootstrap is one step.
Hardening (optional): make the agent resilient to missing host paths so one absent bind can't keep it off a node — e.g. drop the /etc/timezone bind in favor of a TZ env var, and treat /var/log/journal as best-effort.

Verify after fix

docker service ps spikersoft-node-agent_spikersoft-node-agent   # 10/10 Running, no Rejected

Filed proactively by automated swarm health check (docker service audit).

## Summary `spikersoft-node-agent` is a **global** service but is running on only **1 of 10** swarm nodes. On the other nodes its tasks are rejected because required read-only host bind-mount paths don't exist. The node agent (host journal / metrics collector) is therefore absent across most of the fleet — an observability blind spot. ## Symptoms `docker service ps spikersoft-node-agent_spikersoft-node-agent`: ``` laptop-server Rejected "bind source path does not exist: /etc/timezone" dreamstream2 Rejected "bind source path does not exist: /var/log/journal" dreamstream4 Rejected "bind source path does not exist: /var/log/journal" dreamstream5 Rejected "bind source path does not exist: /var/log/journal" 4090 Running ``` Only `4090` is up. Rejected tasks recur continuously (global service keeps retrying). ## Root cause `spikersoft-node-agent/docker-stack.yml` (mode: global) hard-binds host paths read-only: ``` - /var/log/journal:/var/log/journal:ro - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro ``` - `/var/log/journal` only exists when persistent journald is enabled. The repo already ships `spikersoft-node-agent/scripts/enable-persistent-journald.sh` for this, but it has not been run on **dreamstream2, dreamstream4, dreamstream5**. - `/etc/timezone` is missing on **laptop-server** (not created by the journald script; some distros/minimal installs lack it). ## Impact Node agent not running on ≥4/10 nodes -> no host journal tailing / node metrics from those hosts. Continuous rejected-task churn in the scheduler. ## Proposed fix 1. Run the existing helper on the affected nodes (documented as "run on EVERY node"): ``` for h in dreamstream2 dreamstream4 dreamstream5; do scp spikersoft-node-agent/scripts/enable-persistent-journald.sh $h:/tmp/ \ && ssh $h 'sudo bash /tmp/enable-persistent-journald.sh' done ``` 2. Ensure `/etc/timezone` on `laptop-server` (e.g. `sudo sh -c 'echo Etc/UTC > /etc/timezone'`, or `sudo dpkg-reconfigure tzdata`). Consider extending `enable-persistent-journald.sh` to also create `/etc/timezone` if absent so bootstrap is one step. 3. Hardening (optional): make the agent resilient to missing host paths so one absent bind can't keep it off a node — e.g. drop the `/etc/timezone` bind in favor of a `TZ` env var, and treat `/var/log/journal` as best-effort. ## Verify after fix ``` docker service ps spikersoft-node-agent_spikersoft-node-agent # 10/10 Running, no Rejected ``` --- _Filed proactively by automated swarm health check (docker service audit)._

spikerj commented

2026-07-12 00:10:29 +00:00

Repo-side fix in spikersoft-infrastructure PR #29: dropped the redundant /etc/timezone bind (Debian-ism missing on laptop-server; /etc/localtime already provides the zone) and documented the /var/log/journal prerequisite inline. That bind stays deliberately — persistent journal is the agent's fallback read path given swarm ignores privileged/pid. Remaining OPS before this closes (ticket step 1): on dreamstream2, dreamstream4, dreamstream5 run spikersoft-node-agent/scripts/enable-persistent-journald.sh, then sync /mnt/infrastructure and redeploy the stack — the global service schedules onto each node by itself once the path exists. Expected end state: Running on all 10 nodes.

spikerj referenced this issue

2026-07-12 07:05:39 +00:00

[Bug][Infra][GPU] artpipe lane down on 4090 — workers Rejected on missing /mnt/fusionio/spikersoft/art_pipe bind after artpipe-gpu label move (collateral of #500) #501

spikerj commented

2026-07-12 18:44:33 +00:00

Priority escalation — tonight made this ticket expensive. Three resource-exhaustion incidents in one session, all in the disk/exhaustion class the #381 probe already detects: (1) runner Docker network pool exhaustion broke builds, (2) MinIO drive fault under write load, (3) laptop-server disk hit 100% and Gitea's Postgres started failing writes ('could not extend file: No space left on device') — CI/registry/issues backbone degraded until a manual prune.

The bitter part: the disk-space pipeline is FULLY BUILT (#381 — HostHealthProbe → df parser → DiskUsageReport → SystemRemediation threshold classifier) — but the agent runs on 1/10 nodes because of this ticket's bind rejections. Had the agent been on laptop-server, tonight's Gitea outage would have been a warning hours earlier instead of a failure.

Remaining runbook (all ops, ~5 min):

enable-persistent-journald.sh on dreamstream2/4/5 (this ticket's step 1).
Redeploy the node-agent stack (the /etc/timezone bind fix is already on master) → docker service ps spikersoft-node-agent_… should show 10/10 (or 9/9 + 4090).
Verify in Seq: HostHealth/DiskUsage reports from every node; then set a Seq alert (or SystemRemediation notification) on DiskUsageLevel=Warning/Critical if not already wired.

Also add to the probe wishlist from tonight (#505): MinIO Drives: N/N OK, Docker network-pool headroom, drive temperature (fio-status).

**Priority escalation — tonight made this ticket expensive.** Three resource-exhaustion incidents in one session, all in the disk/exhaustion class the #381 probe already detects: (1) runner Docker network pool exhaustion broke builds, (2) MinIO drive fault under write load, (3) **laptop-server disk hit 100% and Gitea's Postgres started failing writes** ('could not extend file: No space left on device') — CI/registry/issues backbone degraded until a manual prune. The bitter part: the disk-space pipeline is FULLY BUILT (#381 — HostHealthProbe → df parser → DiskUsageReport → SystemRemediation threshold classifier) — but the agent runs on 1/10 nodes because of this ticket's bind rejections. **Had the agent been on laptop-server, tonight's Gitea outage would have been a warning hours earlier instead of a failure.** Remaining runbook (all ops, ~5 min): 1. `enable-persistent-journald.sh` on dreamstream2/4/5 (this ticket's step 1). 2. Redeploy the node-agent stack (the /etc/timezone bind fix is already on master) → `docker service ps spikersoft-node-agent_…` should show 10/10 (or 9/9 + 4090). 3. Verify in Seq: HostHealth/DiskUsage reports from every node; then set a Seq alert (or SystemRemediation notification) on DiskUsageLevel=Warning/Critical if not already wired. Also add to the probe wishlist from tonight (#505): MinIO `Drives: N/N OK`, Docker network-pool headroom, drive temperature (fio-status).

spikerj commented

2026-07-12 19:34:13 +00:00

Architectural fix up — the runbook is no longer a deploy precondition. Two paired PRs:

backend #242 (fix/480-journal-dir-resolver): the agent now resolves the journal store per tail attempt — persistent /var/log/journal when present, else volatile /run/log/journal (which is what dreamstream2/4/5 actually have). Availability probe uses the same resolution, so the agent comes up working on volatile-journal nodes instead of idling. A store switch drops the saved journald cursor (cross-store cursors can wedge --after-cursor) and resumes from the tail. 7 new xUnit tests; NodeAgent suite 137/137.

infra #47 (fix/480-node-agent-parent-binds): the stack now binds the parents /var/log + /run/log (FHS + systemd-tmpfiles — exist on every host) instead of the journal dirs themselves, so no bind source can be missing → swarm can never reject the task again. Container paths unchanged.

Net effect after both merge + agent stack redeploy: 10/10 nodes running with zero host prerequisites. enable-persistent-journald.sh is demoted to a recommendation (reboot-surviving history) — running it later on a node is picked up live through the parent mount, no redeploy.

Verification once deployed: docker service ps spikersoft-node-agent_spikersoft-node-agent → 10/10 Running, no Rejected churn; Seq should show "Host journal access available (mode=MountedDirectory, dir=/run/log/journal [volatile fallback], ...)" from the Jetsons.

Architectural fix up — the runbook is no longer a deploy precondition. Two paired PRs: **backend #242** (`fix/480-journal-dir-resolver`): the agent now resolves the journal store **per tail attempt** — persistent `/var/log/journal` when present, else volatile `/run/log/journal` (which is what dreamstream2/4/5 actually have). Availability probe uses the same resolution, so the agent comes up *working* on volatile-journal nodes instead of idling. A store switch drops the saved journald cursor (cross-store cursors can wedge `--after-cursor`) and resumes from the tail. 7 new xUnit tests; NodeAgent suite 137/137. **infra #47** (`fix/480-node-agent-parent-binds`): the stack now binds the parents `/var/log` + `/run/log` (FHS + systemd-tmpfiles — exist on every host) instead of the journal dirs themselves, so **no bind source can be missing → swarm can never reject the task again**. Container paths unchanged. Net effect after both merge + agent stack redeploy: 10/10 nodes running with zero host prerequisites. `enable-persistent-journald.sh` is demoted to a recommendation (reboot-surviving history) — running it later on a node is picked up **live** through the parent mount, no redeploy. Verification once deployed: `docker service ps spikersoft-node-agent_spikersoft-node-agent` → 10/10 Running, no Rejected churn; Seq should show "Host journal access available (mode=MountedDirectory, dir=/run/log/journal [volatile fallback], ...)" from the Jetsons.

spikerj commented

2026-07-12 19:43:32 +00:00

Both PRs merged: backend #242 (journal-store resolver + volatile fallback; CI is shipping the new agent image now) and infra #47 (parent-dir binds /var/log + /run/log).

Remaining to close (needs the swarm side): pull /mnt/infrastructure, then cd /mnt/infrastructure/spikersoft-node-agent && sudo docker stack deploy --with-registry-auth -c docker-stack.yml spikersoft-node-agent. Expected: docker service ps spikersoft-node-agent_spikersoft-node-agent → 10/10 Running, zero Rejected; Seq shows "Host journal access available ... [volatile fallback]" from dreamstream2/4/5. Will close on that evidence.

Both PRs merged: backend #242 (journal-store resolver + volatile fallback; CI is shipping the new agent image now) and infra #47 (parent-dir binds `/var/log` + `/run/log`). **Remaining to close** (needs the swarm side): pull /mnt/infrastructure, then `cd /mnt/infrastructure/spikersoft-node-agent && sudo docker stack deploy --with-registry-auth -c docker-stack.yml spikersoft-node-agent`. Expected: `docker service ps spikersoft-node-agent_spikersoft-node-agent` → **10/10 Running, zero Rejected**; Seq shows "Host journal access available ... [volatile fallback]" from dreamstream2/4/5. Will close on that evidence.

spikerj commented

2026-07-12 20:55:30 +00:00

Deployed and verified in production. Seq, 2026-07-12 19:39–19:42 UTC: 12 × "Host journal access available (mode=MountedDirectory, ...)" startup events across the fleet — 2 tasks reading the persistent store (/var/log/journal) and 10 tasks up via the volatile fallback (/run/log/journal) — i.e. the exact nodes that previously hard-rejected on the missing bind are now running and tailing their journals, with zero host-side preparation. The parent-dir binds (infra #47) + per-attempt store resolver (backend #242) behave exactly as designed.

Bonus: enabling persistent journald on any Jetson later (scripts/enable-persistent-journald.sh) upgrades that node live — the resolver prefers the persistent store on the next tail attempt, no redeploy.

Closing. The #381 disk-space probe now has fleet-wide eyes — which is what tonight's laptop-server 93% would have paged on.

**Deployed and verified in production.** Seq, 2026-07-12 19:39–19:42 UTC: 12 × "Host journal access available (mode=MountedDirectory, ...)" startup events across the fleet — 2 tasks reading the persistent store (`/var/log/journal`) and **10 tasks up via the volatile fallback (`/run/log/journal`)** — i.e. the exact nodes that previously hard-rejected on the missing bind are now running and tailing their journals, with zero host-side preparation. The parent-dir binds (infra #47) + per-attempt store resolver (backend #242) behave exactly as designed. Bonus: enabling persistent journald on any Jetson later (`scripts/enable-persistent-journald.sh`) upgrades that node **live** — the resolver prefers the persistent store on the next tail attempt, no redeploy. Closing. The #381 disk-space probe now has fleet-wide eyes — which is what tonight's laptop-server 93% would have paged on.

spikerj closed this issue

2026-07-12 20:55:31 +00:00

spikerj referenced this issue

2026-07-20 22:41:07 +00:00

feature(sponsor): 'Sponsor a Journey' — family cards with shared destinations, destination chips, family detail page #749

spikerj referenced this issue

2026-07-29 15:38:58 +00:00

EPIC: Art Studio — record and replay generation parameters (+ text+image concept) #842

Sign in to join this conversation.