spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Exploration][Infra] Resilience to SERVER node flaps — restart-policy windows, auto-reconcile sweep, de-SPOF the SERVER-pinned stateful tier #510

New Issue
Closed
opened 2026-07-12 18:29:39 +00:00 by spikerj · 7 comments
spikerj commented 2026-07-12 18:29:39 +00:00
Owner
Copy Link
Copy Source

Goal

Explore concrete ways to make the swarm survive a SERVER node flap with little/no manual intervention. On 2026-07-12 SERVER flapped/crashed several times (see #503). Each event caused the same painful pattern, and recovery repeatedly required a human running docker service update --force. This is a spike to scope the fixes, roughly ordered cheapest-first.

What actually happens during a flap (observed today)

  1. SERVER (a manager) briefly drops → swarm control plane momentarily "not active".
  2. Every SERVER-pinned task that tries to (re)start in that window fails with transient errors:
    • cannot create a swarm scoped network when swarm is not active
    • error while removing network: unknown network jaeger/seq-attachable <id>
  3. Those tasks burn their restart budget (RestartPolicy: on-failure, maxAttempts=3, no window) on the transient errors.
  4. Once the budget is exhausted, swarm stops reconciling the slot (UpdateStatus=completed) → the service sits at 0/1 indefinitely (or self-heals only after a very long backoff).
  5. A human must run docker service update --force <svc> to mint a fresh task with a new budget. Today that was needed ~7+ times (upload-coordinator, security-scanner, metadata-extractor, docker-monitor, system-remediation, security-monitor, gpu-coordinator).

Data from the live cluster (2026-07-12)

Restart policy distribution (92 services):

68  any        | maxAttempts=0  | NO-WINDOW   (restart always, unlimited — OK)
11  on-failure | maxAttempts=0  | NO-WINDOW   (unlimited retries — OK)
11  on-failure | maxAttempts=3  | NO-WINDOW   <-- the wedge-prone set
 1  any        | maxAttempts=10 | NO-WINDOW
 1  none

No service sets a restart Window. With maxAttempts=3 and no window, a 3-failure burst during a flap permanently exhausts the budget. A window makes maxAttempts count failures within a rolling window, so a transient burst doesn't strand the service.

Single-point-of-failure surface: ~40 services are node.hostname == SERVER, including everything critical:

  • Entire MongoDB — mongo-config-01/02/03, mongo-router, and all 9 shard replicas (mongo-shard0{1,2,3}-{a,b,c}) are ALL pinned to SERVER. The cluster is sharded/replica-setted in name only — a single physical node loss = total DB outage. No HA benefit from the replica sets.
  • minio (object storage), backend (API), angular (frontend), gpu-coordinator + embeddings + the whole book pipeline (upload-coordinator, security-scanner, metadata-extractor, file-movement, quiz-generation), plus mailserver stack, sonarqube, clamav, jaeger, gameserver.

Candidate improvements (cheapest → most structural)

A. Restart-policy hardening — quick win, highest ROI

Give the 11 on-failure/maxAttempts=3 services a restart window (e.g. window: 120s) so a flap-burst can't permanently exhaust the budget, or drop maxAttempts to 0 (unlimited) as 11 of their siblings already do. Directly kills the "stuck 0/1 needs manual force" failure mode. Low risk, compose-file-only change.

B. Auto-reconcile sweep — belt-and-suspenders

A tiny scheduled watcher (there's already docker-monitor / system-remediation in the stack) that detects any service with desired > running and a terminal exhausted-restart task, and runs docker service update --force on it. This is literally the manual step we did ~7× today, automated. Bounds worst-case recovery to the sweep interval instead of "until a human notices."

C. Registry/deploy robustness — seen twice today

Service specs got pinned to image digests not actually pullable → No such image: ...@sha256:... reject loops (quiz-generation ~18:03; image-description phantom-deploy #508). Add a CI gate that verifies the pushed digest is pullable from the registry before updating the service spec, so a half-pushed image can't wedge a rolling update.

D. Degraded-state health checks — see #504

Health checks that report process-liveness but miss degraded data states (minio mc ready local returns healthy at 0 drives). Probe real readiness (e.g. /minio/health/cluster) so swarm restarts a degraded task instead of serving errors indefinitely.

E. De-SPOF the critical stateful tier — biggest, most structural

Relax node.hostname == SERVER for services that don't truly need that exact box, and spread replicas across physical nodes:

  • Mongo: place config-server and per-shard replica members on different physical nodes so the replica sets actually provide HA (survive one-node loss). Today they're all on SERVER.
  • minio: multi-node/distributed erasure set, or at least a warm standby, so object storage survives a SERVER loss.
  • gpu-coordinator: single replica is a lease-broker SPOF; explore an HA pair with leader election, or fast reschedule off SERVER. (Note: GPU lanes are physically tied to specific GPU hosts, so this is constrained — scope carefully.)
  • General: audit which of the ~40 SERVER pins are incidental vs. hardware-required (Fusion-io mount, specific GPU) and unpin the incidental ones.

F. Node-level stability — root cause, see #503

Eliminate the flaps themselves: memtest86+/EDAC to confirm/locate the Bad page state fault; if load-induced, add cgroup/resource protection so heavy load can't take down dockerd. Every item above is mitigation; this is the cure.

G. Overlay-network stability across reforms

Investigate the recurring unknown network jaeger/seq-attachable <id> cleanup errors during swarm reform — whether attachable overlay lifecycle can be made robust so task (re)starts don't fail on stale network refs mid-flap.

Suggested sequencing

A + B first (days, low risk, kills the manual-toil failure mode) → C + D (deploy/health correctness) → F (node root cause) → E (the real HA project, larger effort). Recommend this ticket produce a short design doc + follow-up implementation tickets per lettered item.

Related

  • #503 — SERVER flaps / Bad page state (the trigger; item F).
  • #481 — jaeger_elasticsearch MaxAttempts-exhausted (concrete instance of item A).
  • #504 — minio degraded-state health check gap (item D).
  • Prior manual recoveries this session (items A/B) and the image-description phantom deploy #508 (item C).
## Goal Explore concrete ways to make the swarm survive a **SERVER node flap** with little/no manual intervention. On 2026-07-12 SERVER flapped/crashed several times (see #503). Each event caused the same painful pattern, and recovery repeatedly required a human running `docker service update --force`. This is a spike to scope the fixes, roughly ordered cheapest-first. ## What actually happens during a flap (observed today) 1. SERVER (a manager) briefly drops → swarm control plane momentarily "not active". 2. Every SERVER-pinned task that tries to (re)start in that window fails with transient errors: - `cannot create a swarm scoped network when swarm is not active` - `error while removing network: unknown network jaeger/seq-attachable <id>` 3. Those tasks **burn their restart budget** (`RestartPolicy: on-failure, maxAttempts=3, no window`) on the transient errors. 4. Once the budget is exhausted, swarm **stops reconciling** the slot (UpdateStatus=completed) → the service sits at **0/1 indefinitely** (or self-heals only after a very long backoff). 5. A human must run `docker service update --force <svc>` to mint a fresh task with a new budget. Today that was needed ~7+ times (upload-coordinator, security-scanner, metadata-extractor, docker-monitor, system-remediation, security-monitor, gpu-coordinator). ## Data from the live cluster (2026-07-12) **Restart policy distribution (92 services):** ``` 68 any | maxAttempts=0 | NO-WINDOW (restart always, unlimited — OK) 11 on-failure | maxAttempts=0 | NO-WINDOW (unlimited retries — OK) 11 on-failure | maxAttempts=3 | NO-WINDOW <-- the wedge-prone set 1 any | maxAttempts=10 | NO-WINDOW 1 none ``` **No service sets a restart `Window`.** With `maxAttempts=3` and no window, a 3-failure burst during a flap permanently exhausts the budget. A window makes maxAttempts count failures *within a rolling window*, so a transient burst doesn't strand the service. **Single-point-of-failure surface: ~40 services are `node.hostname == SERVER`,** including everything critical: - **Entire MongoDB** — `mongo-config-01/02/03`, `mongo-router`, and all 9 shard replicas (`mongo-shard0{1,2,3}-{a,b,c}`) are ALL pinned to SERVER. The cluster is sharded/replica-setted **in name only** — a single physical node loss = total DB outage. No HA benefit from the replica sets. - **minio** (object storage), **backend** (API), **angular** (frontend), **gpu-coordinator** + **embeddings** + the whole book pipeline (upload-coordinator, security-scanner, metadata-extractor, file-movement, quiz-generation), plus mailserver stack, sonarqube, clamav, jaeger, gameserver. ## Candidate improvements (cheapest → most structural) ### A. Restart-policy hardening — quick win, highest ROI Give the 11 `on-failure/maxAttempts=3` services a **restart `window`** (e.g. `window: 120s`) so a flap-burst can't permanently exhaust the budget, or drop `maxAttempts` to 0 (unlimited) as 11 of their siblings already do. Directly kills the "stuck 0/1 needs manual force" failure mode. Low risk, compose-file-only change. ### B. Auto-reconcile sweep — belt-and-suspenders A tiny scheduled watcher (there's already `docker-monitor` / `system-remediation` in the stack) that detects any service with `desired > running` **and** a terminal exhausted-restart task, and runs `docker service update --force` on it. This is literally the manual step we did ~7× today, automated. Bounds worst-case recovery to the sweep interval instead of "until a human notices." ### C. Registry/deploy robustness — seen twice today Service specs got pinned to image digests not actually pullable → `No such image: ...@sha256:...` reject loops (quiz-generation ~18:03; image-description phantom-deploy #508). Add a CI gate that **verifies the pushed digest is pullable from the registry before updating the service spec**, so a half-pushed image can't wedge a rolling update. ### D. Degraded-state health checks — see #504 Health checks that report process-liveness but miss degraded data states (minio `mc ready local` returns healthy at 0 drives). Probe real readiness (e.g. `/minio/health/cluster`) so swarm restarts a degraded task instead of serving errors indefinitely. ### E. De-SPOF the critical stateful tier — biggest, most structural Relax `node.hostname == SERVER` for services that don't truly need that exact box, and spread replicas across physical nodes: - **Mongo:** place config-server and per-shard replica members on *different* physical nodes so the replica sets actually provide HA (survive one-node loss). Today they're all on SERVER. - **minio:** multi-node/distributed erasure set, or at least a warm standby, so object storage survives a SERVER loss. - **gpu-coordinator:** single replica is a lease-broker SPOF; explore an HA pair with leader election, or fast reschedule off SERVER. (Note: GPU lanes are physically tied to specific GPU hosts, so this is constrained — scope carefully.) - General: audit which of the ~40 SERVER pins are incidental vs. hardware-required (Fusion-io mount, specific GPU) and unpin the incidental ones. ### F. Node-level stability — root cause, see #503 Eliminate the flaps themselves: memtest86+/EDAC to confirm/locate the `Bad page state` fault; if load-induced, add cgroup/resource protection so heavy load can't take down dockerd. Every item above is mitigation; this is the cure. ### G. Overlay-network stability across reforms Investigate the recurring `unknown network jaeger/seq-attachable <id>` cleanup errors during swarm reform — whether attachable overlay lifecycle can be made robust so task (re)starts don't fail on stale network refs mid-flap. ## Suggested sequencing A + B first (days, low risk, kills the manual-toil failure mode) → C + D (deploy/health correctness) → F (node root cause) → E (the real HA project, larger effort). Recommend this ticket produce a short design doc + follow-up implementation tickets per lettered item. ## Related - #503 — SERVER flaps / Bad page state (the trigger; item F). - #481 — jaeger_elasticsearch MaxAttempts-exhausted (concrete instance of item A). - #504 — minio degraded-state health check gap (item D). - Prior manual recoveries this session (items A/B) and the image-description phantom deploy #508 (item C).
spikerj commented 2026-07-12 18:32:28 +00:00
Author
Owner
Copy Link
Copy Source

Concrete recurring instance of item C filed as 511 — quiz-generation's rolling update races its own image push: spec pinned to @sha256:X → swarm 'No such image' → same digest pullable minutes later → converges, but UpdateStatus flips to paused. Seen across 3 digests (5f25f967/17d60d8b/fc0b0731) in ~10 min today. The item-C guard (verify docker manifest inspect <repo>@<digest> before the service update) would prevent it.

Concrete recurring instance of item C filed as 511 — quiz-generation's rolling update races its own image push: spec pinned to @sha256:X → swarm 'No such image' → same digest pullable minutes later → converges, but UpdateStatus flips to paused. Seen across 3 digests (5f25f967/17d60d8b/fc0b0731) in ~10 min today. The item-C guard (verify `docker manifest inspect <repo>@<digest>` before the service update) would prevent it.
spikerj commented 2026-07-12 19:04:31 +00:00
Author
Owner
Copy Link
Copy Source

Progress: A done (infra PR #45 — restart windows on the 10 wedge-prone services; gameserver-init/mongodb deliberately excluded with rationale), C done fleet-wide (backend #239 — pushed-digest-pullable gate on all 28 image workflows), D done (#504/infra #42 — MinIO drive-aware healthcheck, deployed). Remaining: B (auto-reconcile sweep — natural home is system-remediation, happy to build it as a follow-up), E (de-SPOF design — deserves its own doc; note the Mongo all-on-SERVER finding is the scariest line in this ticket), F (#503 hardware — the actual cure), G (overlay cleanup errors). Recommend spinning B and E into their own tickets.

Progress: **A done** (infra PR #45 — restart windows on the 10 wedge-prone services; gameserver-init/mongodb deliberately excluded with rationale), **C done fleet-wide** (backend #239 — pushed-digest-pullable gate on all 28 image workflows), **D done** (#504/infra #42 — MinIO drive-aware healthcheck, deployed). Remaining: **B** (auto-reconcile sweep — natural home is system-remediation, happy to build it as a follow-up), **E** (de-SPOF design — deserves its own doc; note the Mongo all-on-SERVER finding is the scariest line in this ticket), **F** (#503 hardware — the actual cure), **G** (overlay cleanup errors). Recommend spinning B and E into their own tickets.
spikerj commented 2026-07-12 19:12:23 +00:00
Author
Owner
Copy Link
Copy Source

B done: backend PR #240 — auto-reconcile sweep in docker-monitor (automated force-update with tested safety rails: one-shot-job protection, active-task deference, quiet window, cooldown, kill switch). Scorecard once #45 + #240 merge: A ✅ B ✅ C ✅ D ✅. Remaining are the structural ones: E (de-SPOF — recommend its own design ticket; the all-of-Mongo-on-SERVER finding deserves top billing), F (#503 hardware), G (overlay lifecycle). Suggest keeping this ticket open as the umbrella until E/F/G have owners.

**B done**: backend PR #240 — auto-reconcile sweep in docker-monitor (automated force-update with tested safety rails: one-shot-job protection, active-task deference, quiet window, cooldown, kill switch). Scorecard once #45 + #240 merge: **A ✅ B ✅ C ✅ D ✅**. Remaining are the structural ones: E (de-SPOF — recommend its own design ticket; the all-of-Mongo-on-SERVER finding deserves top billing), F (#503 hardware), G (overlay lifecycle). Suggest keeping this ticket open as the umbrella until E/F/G have owners.
spikerj commented 2026-07-12 21:40:14 +00:00
Author
Owner
Copy Link
Copy Source

Item B (auto-reconcile sweep) has LANDED — commit b96976bd 'feat(docker-monitor): auto-reconcile sweep for stranded services (#510-B)'. Adds ServiceAutoReconciler.cs (167 LOC) wired into docker-monitor's Program.cs + config, with 7 xunit tests (SpikerSoft.EventHandlers.DockerMonitor.Tests/Services/ServiceAutoReconcilerTests.cs) covering the right cases: Stranded_AllTasksFailedAndQuiet_Reconciles (the wedge), RunningTask_NeverReconciles, CompletedOneShotJob_NeverReconciles (won't churn redis-init etc.), RecentFailure_InsideQuietWindow_WaitsForSwarm (quiet window before forcing), FailedOneShot_NonZeroExit_IsReconciled, NeverScheduled_NoTasks_DoesNotReconcile. Deployed in docker-monitor (new digest 8afdb709 @ 21:35). Couldn't run the xunit locally (no .NET SDK on the watch host) but tests are on master. This should auto-recover the restart-budget wedges that needed manual service update --force ~8× today — item B effectively resolved. Also seen: #505 MinIO cluster-health alarm landed in the same service (MinioHealthWatcher + tests) touching item D's territory. Remaining #510 items still open: A (restart windows), C (deploy digest verify — see #511), D (degraded healthchecks), E (de-SPOF), F (#503 node root cause), G (overlay-net).

**Item B (auto-reconcile sweep) has LANDED** — commit b96976bd 'feat(docker-monitor): auto-reconcile sweep for stranded services (#510-B)'. Adds `ServiceAutoReconciler.cs` (167 LOC) wired into docker-monitor's Program.cs + config, with **7 xunit tests** (`SpikerSoft.EventHandlers.DockerMonitor.Tests/Services/ServiceAutoReconcilerTests.cs`) covering the right cases: Stranded_AllTasksFailedAndQuiet_Reconciles (the wedge), RunningTask_NeverReconciles, CompletedOneShotJob_NeverReconciles (won't churn redis-init etc.), RecentFailure_InsideQuietWindow_WaitsForSwarm (quiet window before forcing), FailedOneShot_NonZeroExit_IsReconciled, NeverScheduled_NoTasks_DoesNotReconcile. Deployed in docker-monitor (new digest 8afdb709 @ 21:35). Couldn't run the xunit locally (no .NET SDK on the watch host) but tests are on master. This should auto-recover the restart-budget wedges that needed manual `service update --force` ~8× today — item B effectively resolved. Also seen: #505 MinIO cluster-health alarm landed in the same service (MinioHealthWatcher + tests) touching item D's territory. Remaining #510 items still open: A (restart windows), C (deploy digest verify — see #511), D (degraded healthchecks), E (de-SPOF), F (#503 node root cause), G (overlay-net).
spikerj commented 2026-07-13 04:45:04 +00:00
Author
Owner
Copy Link
Copy Source

New wedge class found (2026-07-13 ~04:45Z), invisible to both item A and item B: after the 04:2x SERVER hard-down recovery, metadata-extractor, security-monitor and system-remediation each started, hit unreachable dependencies mid-recovery, and shut down gracefully — exit 0, task state Complete. Consequences:

  • restart_policy: on-failure never fires on a clean exit → stuck 0/1 (same hole minio hit at 02:31Z, fixed for minio by 1a6eec2's condition: any).
  • The #510-B ServiceAutoReconciler correctly skips them: its CompletedOneShotJob_NeverReconciles rule can't distinguish a gracefully-dying long-running worker from a finished one-shot job. Verified in its logs — sweeping fine, zero reconcile events for these three across 3 cycles.

So the 02:53Z sweep worked because those tasks FAILED; tonight's trio COMPLETED and is invisible. Proposals:

  1. Roll restart_policy.condition: any out to the long-running worker services (they should never legitimately exit 0), like minio and jaeger already got. Cheapest and closes both holes at once.
  2. Optionally teach the reconciler: a replicated service with desired-replicas>0 whose ONLY task history is Complete/Rejected during a flap window is a wedge, not a job — reconcile it. (Service mode is distinguishable: one-shots here are restart-condition: none or labeled init/backup stacks.)

Immediate: the three services need a manual docker service update --force (my session is permission-blocked from doing it).

**New wedge class found (2026-07-13 ~04:45Z), invisible to both item A and item B:** after the 04:2x SERVER hard-down recovery, metadata-extractor, security-monitor and system-remediation each started, hit unreachable dependencies mid-recovery, and shut down **gracefully — exit 0, task state Complete**. Consequences: - `restart_policy: on-failure` never fires on a clean exit → stuck 0/1 (same hole minio hit at 02:31Z, fixed for minio by 1a6eec2's `condition: any`). - The #510-B ServiceAutoReconciler **correctly skips them**: its CompletedOneShotJob_NeverReconciles rule can't distinguish a gracefully-dying long-running worker from a finished one-shot job. Verified in its logs — sweeping fine, zero reconcile events for these three across 3 cycles. So the 02:53Z sweep worked because those tasks FAILED; tonight's trio COMPLETED and is invisible. Proposals: 1. Roll `restart_policy.condition: any` out to the long-running worker services (they should never legitimately exit 0), like minio and jaeger already got. Cheapest and closes both holes at once. 2. Optionally teach the reconciler: a replicated service with desired-replicas>0 whose ONLY task history is Complete/Rejected during a flap window is a wedge, not a job — reconcile it. (Service mode is distinguishable: one-shots here are `restart-condition: none` or labeled init/backup stacks.) Immediate: the three services need a manual `docker service update --force` (my session is permission-blocked from doing it).
spikerj commented 2026-07-13 17:10:58 +00:00
Author
Owner
Copy Link
Copy Source

Exit-0 wedge occurrence #3 (2026-07-13 ~18:2xZ, post-GRUB-reboot recovery): metadata-extractor + security-scanner + upload-coordinator all Complete-wedged again — upload pipeline down until manual force-updates. That's three occurrences in ~14h; the restart_policy.condition: any rollout to the long-running workers (proposal 1 above) has clearly graduated from nice-to-have to needed — it's a small stack-file change per service and eliminates this entire class, same as it already did for minio (3 clean-exit self-recoveries since 1a6eec2).

Exit-0 wedge occurrence #3 (2026-07-13 ~18:2xZ, post-GRUB-reboot recovery): metadata-extractor + security-scanner + upload-coordinator all Complete-wedged again — upload pipeline down until manual force-updates. That's three occurrences in ~14h; the `restart_policy.condition: any` rollout to the long-running workers (proposal 1 above) has clearly graduated from nice-to-have to needed — it's a small stack-file change per service and eliminates this entire class, same as it already did for minio (3 clean-exit self-recoveries since 1a6eec2).
spikerj commented 2026-07-14 03:21:36 +00:00
Author
Owner
Copy Link
Copy Source

Audit delivered — detailed findings relocated (2026-07-14): the full #510 resilience audit (per-service SPOF table, strand-pattern findings, bind/placement gaps, and the credential-variable inventory for #545/#546 targeting) was originally posted here, but this issues repo is public and that level of infrastructure/credential detail does not belong on a public tracker. It has been removed from this comment and preserved for the team; it can be re-homed to the private spikersoft-infrastructure repo (docs/ or the PR #69 thread) — Joey's call.

Non-sensitive headline results:

  • Mechanical fix PR: spikersoft-infrastructure #69 (restart-policy exit-0-wedge fixes on 5 services, the #481/minio pattern; inert until each stack's next redeploy).
  • The devices: blocks this ticket's thread worried about were already swept from master (51d5f8d) — zero remain.
  • Counts only: 45 single-node-pinned single-replica services, 8 restart-strand findings (6 actionable), 5 bind-without-placement gaps, 30 plaintext credential variables across 10 stack files (inventory feeds the OpenBao #543 program).

— macbook-claude-session

**Audit delivered — detailed findings relocated (2026-07-14):** the full #510 resilience audit (per-service SPOF table, strand-pattern findings, bind/placement gaps, and the credential-variable inventory for #545/#546 targeting) was originally posted here, but **this issues repo is public** and that level of infrastructure/credential detail does not belong on a public tracker. It has been removed from this comment and preserved for the team; it can be re-homed to the private spikersoft-infrastructure repo (docs/ or the PR #69 thread) — Joey's call. **Non-sensitive headline results:** - Mechanical fix PR: spikersoft-infrastructure **#69** (restart-policy exit-0-wedge fixes on 5 services, the #481/minio pattern; inert until each stack's next redeploy). - The `devices:` blocks this ticket's thread worried about were already swept from master (51d5f8d) — zero remain. - Counts only: 45 single-node-pinned single-replica services, 8 restart-strand findings (6 actionable), 5 bind-without-placement gaps, 30 plaintext credential variables across 10 stack files (inventory feeds the OpenBao #543 program). — macbook-claude-session
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#510