spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra][Deploy] docker-monitor: new correctly-named stack exits 0 on startup (stuck 0/1) + duplicate misspelled 'spiekersoft-docker-monitor' stack running in parallel #513

New Issue
Closed
opened 2026-07-12 19:33:27 +00:00 by spikerj · 9 comments
spikerj commented 2026-07-12 19:33:27 +00:00
Owner
Copy Link
Copy Source

Summary

Two problems on docker-monitor, surfaced 2026-07-12 ~19:23–19:32 UTC:

  1. A newly-deployed, correctly-named docker-monitor stack exits immediately (exit 0) on startup and can't stay up.
  2. There is a duplicate stack deployed under a misspelled name (spiekersoft-…) that has been the one actually running.

The two stacks

spiekersoft-docker-monitor_spikersoft-docker-monitor   1/1   img ...docker-monitor:latest@sha256:dd9d2988...   created 17:20:14
spikersoft-docker-monitor_spikersoft-docker-monitor    0/1   img ...docker-monitor:latest@sha256:c14863a1...   created 19:23:40
  • The misspelled stack spiekersoft-docker-monitor (note spie) has been running 1/1 since 17:20 on digest dd9d2988 — this is the instance currently doing the monitoring.
  • The correctly-spelled stack spikersoft-docker-monitor was deployed at 19:23 on a different digest c14863a1, and its task cycles Complete (exit 0) within ~1s of starting, repeatedly, on SERVER. RestartPolicy: on-failure/3 does not restart a clean exit-0, so it parks at 0/1; UpdateStatus: paused.

Evidence

  • docker service ps spikersoft-docker-monitor_spikersoft-docker-monitor → repeated Complete on node SERVER (exit 0, not Failed/Shutdown → the container's PID 1 exits cleanly, not a kill).
  • Container emits no log output before exiting (docker service logs empty).
  • A manual docker service update --force started a task that again went Complete in <1s → confirms the image/entrypoint exits immediately, not a transient.
  • (Image + container are on SERVER and couldn't be inspected from the other manager; SERVER's log endpoint was also intermittently unavailable during the window — a separate #503 symptom, not the cause of the exit-0.)

Likely cause

The c14863a1 image (the 19:23 deploy) appears to run an entrypoint/CMD that completes and exits 0 instead of running the long-lived monitor loop — e.g. a one-shot command baked as CMD, a missing exec/foreground process, or a changed entrypoint vs the working dd9d2988 image. Needs the team to diff the two images' entrypoint/CMD.

Requested actions

  1. Fix the exit-0: inspect the c14863a1 image's ENTRYPOINT/CMD; ensure the monitor runs as a foreground, long-lived process. (Compare against the working dd9d2988 image.)
  2. Resolve the duplicate: decide which stack name is canonical (spikersoft-docker-monitor is the correct spelling) and remove the misspelled spiekersoft-docker-monitor stack once the correct one runs — otherwise two monitors run in parallel (double alerting / wasted resources), or right now the only working one is the typo'd stack that's presumably not what deploys target.
  3. Consider a startup smoke-test in CI (container stays up >N seconds) so a monitor image that exits immediately can't ship — mirrors the import smoke-test added for image-description (#508).

Impact

Low user impact (monitoring only, and the misspelled instance is currently covering). But: the intended (correctly-named) monitor is down, deploys targeting it will keep failing, and the naming split is a latent confusion/duplication hazard. Also note docker-monitor is exactly the service that would help auto-detect the swarm issues in #510 (item B) — worth having healthy.

Related

  • #510 — resilience exploration; item C (registry/deploy) and item B (auto-reconcile / this is a monitor).
  • #508 — image-description startup smoke-test pattern (apply the same guard here).
  • #503 — SERVER log-endpoint flaps that made this harder to inspect (not the cause).
## Summary Two problems on `docker-monitor`, surfaced 2026-07-12 ~19:23–19:32 UTC: 1. **A newly-deployed, correctly-named `docker-monitor` stack exits immediately (exit 0) on startup** and can't stay up. 2. **There is a duplicate stack deployed under a misspelled name** (`spiekersoft-…`) that has been the one actually running. ### The two stacks ``` spiekersoft-docker-monitor_spikersoft-docker-monitor 1/1 img ...docker-monitor:latest@sha256:dd9d2988... created 17:20:14 spikersoft-docker-monitor_spikersoft-docker-monitor 0/1 img ...docker-monitor:latest@sha256:c14863a1... created 19:23:40 ``` - The **misspelled** stack `spiekersoft-docker-monitor` (note `spie`) has been running **1/1 since 17:20** on digest `dd9d2988` — this is the instance currently doing the monitoring. - The **correctly-spelled** stack `spikersoft-docker-monitor` was deployed at **19:23** on a *different* digest `c14863a1`, and its task cycles `Complete` (exit 0) within ~1s of starting, repeatedly, on SERVER. `RestartPolicy: on-failure/3` does not restart a clean exit-0, so it parks at **0/1**; `UpdateStatus: paused`. ### Evidence - `docker service ps spikersoft-docker-monitor_spikersoft-docker-monitor` → repeated `Complete` on node SERVER (exit 0, not Failed/Shutdown → the container's PID 1 exits cleanly, not a kill). - Container emits **no log output** before exiting (`docker service logs` empty). - A manual `docker service update --force` started a task that again went `Complete` in <1s → confirms the image/entrypoint exits immediately, not a transient. - (Image + container are on SERVER and couldn't be inspected from the other manager; SERVER's log endpoint was also intermittently unavailable during the window — a separate #503 symptom, not the cause of the exit-0.) ## Likely cause The `c14863a1` image (the 19:23 deploy) appears to run an entrypoint/CMD that completes and exits 0 instead of running the long-lived monitor loop — e.g. a one-shot command baked as CMD, a missing `exec`/foreground process, or a changed entrypoint vs the working `dd9d2988` image. Needs the team to diff the two images' entrypoint/CMD. ## Requested actions 1. **Fix the exit-0**: inspect the `c14863a1` image's `ENTRYPOINT`/`CMD`; ensure the monitor runs as a foreground, long-lived process. (Compare against the working `dd9d2988` image.) 2. **Resolve the duplicate**: decide which stack name is canonical (`spikersoft-docker-monitor` is the correct spelling) and **remove the misspelled `spiekersoft-docker-monitor` stack** once the correct one runs — otherwise two monitors run in parallel (double alerting / wasted resources), or right now the *only* working one is the typo'd stack that's presumably not what deploys target. 3. Consider a startup smoke-test in CI (container stays up >N seconds) so a monitor image that exits immediately can't ship — mirrors the import smoke-test added for image-description (#508). ## Impact Low user impact (monitoring only, and the misspelled instance is currently covering). But: the intended (correctly-named) monitor is down, deploys targeting it will keep failing, and the naming split is a latent confusion/duplication hazard. Also note `docker-monitor` is exactly the service that would help auto-detect the swarm issues in #510 (item B) — worth having healthy. ## Related - #510 — resilience exploration; item C (registry/deploy) and item B (auto-reconcile / this is a monitor). - #508 — image-description startup smoke-test pattern (apply the same guard here). - #503 — SERVER log-endpoint flaps that made this harder to inspect (not the cause).
spikerj commented 2026-07-12 21:27:00 +00:00
Author
Owner
Copy Link
Copy Source

Root-caused and fixed — backend PR #245. I pulled the exact c14863a1 amd64 image and ran it locally: the container is NOT silent — it logs a loud Serilog fatal and exits 0:

Unable to resolve service for type 'Docker.DotNet.IDockerClient'
while attempting to activate 'ServiceAutoReconciler'.

My #240 miss: ServiceAutoReconciler takes IDockerClient from DI but nothing registered it (the older services each construct their own client internally). Host.StartAsync throws during activation → host builder logs fatal, returns cleanly → exit 0 in <1s → swarm records Complete, on-failure never restarts → 0/1 parked, exactly as you observed. The "no log output" on SERVER was the #503 log-endpoint flakiness masking it. The misspelled spiekersoft- stack survives because it's pinned to the pre-#240 digest (dd9d2988).

PR #245: registers IDockerClient + adds the test class that would have caught this (composition-root mirror built with ValidateOnBuild — validates every constructor graph with no sockets; plus an every-IHostedService-is-registered sweep). 27/27.

After merge + CI deploy: confirm spikersoft-docker-monitor 1/1 (Seq will show "ServiceAutoReconciler active" + "MinIO cluster-health watcher" startup lines), then remove the duplicate: sudo docker stack rm spiekersoft-docker-monitor — two parallel monitors double-publish cluster events and would double-fire the reconciler once the fixed image is live.

Root-caused and fixed — **backend PR #245**. I pulled the exact `c14863a1` amd64 image and ran it locally: the container is NOT silent — it logs a loud Serilog fatal and exits 0: ``` Unable to resolve service for type 'Docker.DotNet.IDockerClient' while attempting to activate 'ServiceAutoReconciler'. ``` My #240 miss: `ServiceAutoReconciler` takes `IDockerClient` from DI but nothing registered it (the older services each construct their own client internally). `Host.StartAsync` throws during activation → host builder logs fatal, returns cleanly → **exit 0 in <1s** → swarm records `Complete`, `on-failure` never restarts → 0/1 parked, exactly as you observed. The "no log output" on SERVER was the #503 log-endpoint flakiness masking it. The misspelled `spiekersoft-` stack survives because it's pinned to the pre-#240 digest (`dd9d2988`). **PR #245**: registers `IDockerClient` + adds the test class that would have caught this (composition-root mirror built with `ValidateOnBuild` — validates every constructor graph with no sockets; plus an every-IHostedService-is-registered sweep). 27/27. **After merge + CI deploy**: confirm `spikersoft-docker-monitor` 1/1 (Seq will show "ServiceAutoReconciler active" + "MinIO cluster-health watcher" startup lines), then remove the duplicate: `sudo docker stack rm spiekersoft-docker-monitor` — two parallel monitors double-publish cluster events and would double-fire the reconciler once the fixed image is live.
spikerj commented 2026-07-12 21:40:14 +00:00
Author
Owner
Copy Link
Copy Source

Update: the exit-0 half is FIXED — docker-monitor was redeployed at 21:35 on a new digest 8afdb709 (was the crashing c14863a1) and the correctly-named spikersoft-docker-monitor is now Running 1/1. (The redeploy also carried the #510-B auto-reconcile sweep.) STILL OPEN: the duplicate — both spiekersoft-docker-monitor (typo, 1/1) AND spikersoft-docker-monitor (correct, 1/1) are now running in parallel, so two monitors + potential double-alerting. Please remove the misspelled spiekersoft- stack now that the correctly-named one is healthy. Keeping open for that dedup.

Update: the exit-0 half is FIXED — docker-monitor was redeployed at 21:35 on a new digest 8afdb709 (was the crashing c14863a1) and the correctly-named `spikersoft-docker-monitor` is now Running 1/1. (The redeploy also carried the #510-B auto-reconcile sweep.) STILL OPEN: the duplicate — both `spiekersoft-docker-monitor` (typo, 1/1) AND `spikersoft-docker-monitor` (correct, 1/1) are now running in parallel, so two monitors + potential double-alerting. Please remove the misspelled `spiekersoft-` stack now that the correctly-named one is healthy. Keeping open for that dedup.
spikerj commented 2026-07-12 23:20:49 +00:00
Author
Owner
Copy Link
Copy Source

Fix verified live in production. Seq, 2026-07-12 21:34:25 UTC — the deployed docker-monitor booted past the old fatal and both #240/#241 features announced themselves:

ServiceAutoReconciler active: sweeping every 300s for stranded services (#510-B)
MinioHealthWatcher active: probing <cluster-health> every 60s (alarm after 3 failures)

The exit-0 startup is dead (PR #245's IDockerClient registration). Remaining to close this ticket — one command on a manager once you confirm spikersoft-docker-monitor is 1/1:

sudo docker stack rm spiekersoft-docker-monitor

(the misspelled duplicate; with the fixed image live, two parallel monitors would double-publish cluster events and double-fire the auto-reconciler sweep).

**Fix verified live in production.** Seq, 2026-07-12 21:34:25 UTC — the deployed docker-monitor booted past the old fatal and both #240/#241 features announced themselves: ``` ServiceAutoReconciler active: sweeping every 300s for stranded services (#510-B) MinioHealthWatcher active: probing <cluster-health> every 60s (alarm after 3 failures) ``` The exit-0 startup is dead (PR #245's `IDockerClient` registration). Remaining to close this ticket — one command on a manager once you confirm `spikersoft-docker-monitor` is 1/1: ``` sudo docker stack rm spiekersoft-docker-monitor ``` (the misspelled duplicate; with the fixed image live, two parallel monitors would double-publish cluster events and double-fire the auto-reconciler sweep).
spikerj commented 2026-07-13 18:42:10 +00:00
Author
Owner
Copy Link
Copy Source

QA watch 2026-07-13 ~18:40Z — removal is now zero-risk. The typo'd stack's task (spiekersoft-docker-monitor, on SERVER) exited cleanly ~16:40Z and sits 0/1 Complete (on-failure policy won't restart a clean exit). The correct-spelled spikersoft-docker-monitor is 1/1 and stable (running the #510-B auto-reconciler build). Only remaining action for this ticket: docker stack rm spiekersoft-docker-monitor.

QA watch 2026-07-13 ~18:40Z — removal is now zero-risk. The typo'd stack's task (spiekersoft-docker-monitor, on SERVER) exited cleanly ~16:40Z and sits 0/1 Complete (on-failure policy won't restart a clean exit). The correct-spelled spikersoft-docker-monitor is 1/1 and stable (running the #510-B auto-reconciler build). Only remaining action for this ticket: docker stack rm spiekersoft-docker-monitor.
spikerj commented 2026-07-13 19:18:38 +00:00
Author
Owner
Copy Link
Copy Source

Status: (1) exit-0 root cause fixed on master in 28c62fb (missing IDockerClient DI registration; ValidateOnBuild regression tests added). (3) CI stays-up smoke test: backend PR #255 — 15s no-deps survival gate before push, premise verified by running the host 18s with no dependencies (it retries, doesn't exit). (2) still needs a hands-on-swarm action: once the correct stack runs 1/1, 'sudo docker stack rm spiekersoft-docker-monitor'; if the correct stack is still parked 0/1, 'docker service update --force' it after the fixed image deploys.

Status: (1) exit-0 root cause fixed on master in 28c62fb (missing IDockerClient DI registration; ValidateOnBuild regression tests added). (3) CI stays-up smoke test: backend PR #255 — 15s no-deps survival gate before push, premise verified by running the host 18s with no dependencies (it retries, doesn't exit). (2) still needs a hands-on-swarm action: once the correct stack runs 1/1, 'sudo docker stack rm spiekersoft-docker-monitor'; if the correct stack is still parked 0/1, 'docker service update --force' it after the fixed image deploys.
spikerj commented 2026-07-14 02:24:09 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — state update 2026-07-14 ~02:20Z:

  • The correctly-named stack is now healthy: spikersoft-docker-monitor_spikersoft-docker-monitor shows 1/1 Running (no more exit-0-on-startup loop in current task history).
  • The misspelled spiekersoft-docker-monitor stack still exists and is now 0/1 — its last task hit error while removing network: unknown network jaeger ~10h ago, then Complete; before that a Rejected with cannot create a swarm scoped network when swarm is not active (transient dockerd restart on the node). It's dead weight now that the real stack runs — the remaining action here is just docker stack rm spiekersoft-docker-monitor so it stops polluting docker service ls with a permanent 0/1.
**QA Team** — state update 2026-07-14 ~02:20Z: - The **correctly-named stack is now healthy**: `spikersoft-docker-monitor_spikersoft-docker-monitor` shows **1/1 Running** (no more exit-0-on-startup loop in current task history). - The **misspelled `spiekersoft-docker-monitor` stack still exists** and is now 0/1 — its last task hit `error while removing network: unknown network jaeger` ~10h ago, then Complete; before that a Rejected with `cannot create a swarm scoped network when swarm is not active` (transient dockerd restart on the node). It's dead weight now that the real stack runs — the remaining action here is just `docker stack rm spiekersoft-docker-monitor` so it stops polluting `docker service ls` with a permanent 0/1.
spikerj commented 2026-07-14 02:55:41 +00:00
Author
Owner
Copy Link
Copy Source

Problem 1 (exit-0) is fixed on master — Program.cs now registers IDockerClient with the annotation explaining exactly this failure: ServiceAutoReconciler (#510-B) resolved IDockerClient that no one registered → Host.StartAsync threw during activation → EventHandlerHostBuilder logged the fatal and returned → exit 0 in <1 s → swarm records Complete and on-failure never restarts it. The fix is in current :latest (rebuilt several times tonight, incl. the #541 node-events merge).

Remaining ops to close:

  1. Redeploy the correctly-named stack so it pulls the fixed image:
    cd /mnt/infrastructure/<docker-monitor stack dir> && sudo docker stack deploy --with-registry-auth -c docker-stack.yml spikersoft-docker-monitor
  2. Verify it converges 1/1 Running on SERVER (docker service ps spikersoft-docker-monitor_spikersoft-docker-monitor) and events flow (Seq: docker.node.* / docker.service.* messages).
  3. Only then remove the misspelled duplicate: sudo docker stack rm spiekersoft-docker-monitor — it is the only instance actually monitoring right now, so keep it up until step 2 proves the replacement.

Also worth a code follow-up someday: an activation failure exiting 0 is what let this hide — the builder should exit non-zero on a fatal so on-failure restart policies actually engage (same "clean exit on fatal" class as #554's jetson churn).

— macbook-claude-session

**Problem 1 (exit-0) is fixed on master** — `Program.cs` now registers `IDockerClient` with the annotation explaining exactly this failure: `ServiceAutoReconciler` (#510-B) resolved `IDockerClient` that no one registered → `Host.StartAsync` threw during activation → `EventHandlerHostBuilder` logged the fatal and returned → exit **0** in <1 s → swarm records `Complete` and `on-failure` never restarts it. The fix is in current `:latest` (rebuilt several times tonight, incl. the #541 node-events merge). **Remaining ops to close:** 1. Redeploy the correctly-named stack so it pulls the fixed image: `cd /mnt/infrastructure/<docker-monitor stack dir> && sudo docker stack deploy --with-registry-auth -c docker-stack.yml spikersoft-docker-monitor` 2. Verify it converges **1/1 Running** on SERVER (`docker service ps spikersoft-docker-monitor_spikersoft-docker-monitor`) and events flow (Seq: docker.node.* / docker.service.* messages). 3. Only then remove the misspelled duplicate: `sudo docker stack rm spiekersoft-docker-monitor` — it is the only instance actually monitoring right now, so keep it up until step 2 proves the replacement. Also worth a code follow-up someday: an activation failure exiting **0** is what let this hide — the builder should exit non-zero on a fatal so `on-failure` restart policies actually engage (same "clean exit on fatal" class as #554's jetson churn). — macbook-claude-session
spikerj commented 2026-07-15 19:12:56 +00:00
Author
Owner
Copy Link
Copy Source

Heads-up on action item (3), the CI startup smoke test: it was failing on teardown (not on the assertion). The container passed the 15s survival check, then docker rm -f dm-smoke failed with container PID <n> is zombie and can not be killed. Use the --init option ..., which red the whole job.

Opened spikersoft-backend PR #307 — adds --init to the smoke-test docker run so docker-init (tini) is PID 1 and the container is always killable/reapable (tini forwards the child's exit code, so the survival check is unaffected), plus a DockerMonitorSmokeTestUsesInitTests guard.

Scoped to the CI harness on purpose — reproduced locally that the app spawns no children and the exec-form healthcheck curls are reaped by the containerd-shim (a dotnet-as-PID-1 container accumulates zero zombies across healthcheck cycles), so the shipped image and swarm stack need no change. This is a runner-side SIGKILL/reaping artifact, not a prod defect.

Doesn't touch this ticket's remaining open item — removing the misspelled spiekersoft-docker-monitor stack (sudo docker stack rm spiekersoft-docker-monitor), still an ops action on a manager.

Heads-up on action item (3), the CI startup smoke test: it was **failing on teardown** (not on the assertion). The container passed the 15s survival check, then `docker rm -f dm-smoke` failed with `container PID <n> is zombie and can not be killed. Use the --init option ...`, which red the whole job. Opened **spikersoft-backend PR #307** — adds `--init` to the smoke-test `docker run` so docker-init (tini) is PID 1 and the container is always killable/reapable (tini forwards the child's exit code, so the survival check is unaffected), plus a `DockerMonitorSmokeTestUsesInitTests` guard. Scoped to the CI harness on purpose — reproduced locally that the app spawns no children and the exec-form healthcheck curls are reaped by the containerd-shim (a dotnet-as-PID-1 container accumulates zero zombies across healthcheck cycles), so the shipped image and swarm stack need no change. This is a runner-side SIGKILL/reaping artifact, not a prod defect. Doesn't touch this ticket's remaining open item — removing the misspelled `spiekersoft-docker-monitor` stack (`sudo docker stack rm spiekersoft-docker-monitor`), still an ops action on a manager.
spikerj commented 2026-07-22 02:57:51 +00:00
Author
Owner
Copy Link
Copy Source

Board-sweep closing on merged evidence: root cause fixed (IDockerClient DI, #245), guarded by CI smoke (#255/#307), canonical stack name pinned (infra). Residual: deleting the misspelled duplicate stack from the swarm is a 30-second ops action — reopen only if the ghost stack causes real confusion.

Board-sweep closing on merged evidence: root cause fixed (IDockerClient DI, #245), guarded by CI smoke (#255/#307), canonical stack name pinned (infra). Residual: deleting the misspelled duplicate stack from the swarm is a 30-second ops action — reopen only if the ghost stack causes real confusion.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#513