[Bug][Infra] jaeger_elasticsearch down ~24h on SERVER (exit 1) — MaxAttempts exhausted, tracing backend offline #481

New Issue

2026-07-12T00:00:59Z

spikerj commented

2026-07-12 00:00:59 +00:00

Summary

jaeger_elasticsearch has been down ~24h on the SERVER node (0/1 replicas), so Jaeger has no span storage backend — distributed tracing is effectively down. Because its restart policy is on-failure with MaxAttempts=3 (already exhausted), swarm will not retry it: it stays down until someone intervenes.

Symptoms

docker service ps jaeger_elasticsearch:

SERVER   Failed 24 hours ago   "task: non-zero exit (1)"   (x3-4, then gave up)

No task container remains, so live logs are unavailable.

Config (jaeger/docker-stack.yml)

docker.elastic.co/elasticsearch/elasticsearch:7.17.0, discovery.type=single-node, ES_JAVA_OPTS=-Xms512m -Xmx512m
named volume elasticsearch-data (not a bind — so not a missing-path issue), pinned to node.hostname == SERVER
RestartPolicy: on-failure, MaxAttempts=3

Likely causes (ES 7.17 exit 1 on startup, in order)

vm.max_map_count too low on SERVER (bootstrap check fails). Needs >= 262144. Check: ssh SERVER sysctl vm.max_map_count (this audit host 4090 is fine at 1048576, but ES runs on SERVER).
Data volume permission / corruption after an unclean stop (elasticsearch-data).
Disk flood-stage watermark on SERVER (disk >95% full).

Suggested actions

Capture the real error, then force a fresh attempt:

ssh SERVER sysctl vm.max_map_count        # bump to 262144+ if low (persist via /etc/sysctl.d)
docker service update --force jaeger_elasticsearch
docker service logs -f jaeger_elasticsearch   # read the actual bootstrap failure

If it's disk/watermark, free space on SERVER.
Config hardening: raise/relax MaxAttempts (or set condition: any) so a transient failure doesn't leave tracing permanently dead, and add a healthcheck.

Note: deployed Jaeger is 2.17.0 while the stack file pins 2.19.0 (comment references 2.11.0) — minor drift, worth reconciling separately.

Filed proactively by automated swarm health check (docker service audit).

## Summary `jaeger_elasticsearch` has been **down ~24h** on the `SERVER` node (`0/1` replicas), so Jaeger has no span storage backend — distributed tracing is effectively down. Because its restart policy is `on-failure` with `MaxAttempts=3` (already exhausted), swarm will **not** retry it: it stays down until someone intervenes. ## Symptoms `docker service ps jaeger_elasticsearch`: ``` SERVER Failed 24 hours ago "task: non-zero exit (1)" (x3-4, then gave up) ``` No task container remains, so live logs are unavailable. ## Config (jaeger/docker-stack.yml) - `docker.elastic.co/elasticsearch/elasticsearch:7.17.0`, `discovery.type=single-node`, `ES_JAVA_OPTS=-Xms512m -Xmx512m` - named volume `elasticsearch-data` (not a bind — so not a missing-path issue), pinned to `node.hostname == SERVER` - RestartPolicy: `on-failure`, MaxAttempts=3 ## Likely causes (ES 7.17 exit 1 on startup, in order) 1. `vm.max_map_count` too low on **SERVER** (bootstrap check fails). Needs `>= 262144`. Check: `ssh SERVER sysctl vm.max_map_count` (this audit host `4090` is fine at 1048576, but ES runs on SERVER). 2. Data volume permission / corruption after an unclean stop (`elasticsearch-data`). 3. Disk flood-stage watermark on SERVER (disk >95% full). ## Suggested actions 1. Capture the real error, then force a fresh attempt: ``` ssh SERVER sysctl vm.max_map_count # bump to 262144+ if low (persist via /etc/sysctl.d) docker service update --force jaeger_elasticsearch docker service logs -f jaeger_elasticsearch # read the actual bootstrap failure ``` 2. If it's disk/watermark, free space on SERVER. 3. Config hardening: raise/relax `MaxAttempts` (or set `condition: any`) so a transient failure doesn't leave tracing permanently dead, and add a healthcheck. Note: deployed Jaeger is `2.17.0` while the stack file pins `2.19.0` (comment references 2.11.0) — minor drift, worth reconciling separately. --- _Filed proactively by automated swarm health check (docker service audit)._

spikerj commented

2026-07-12 00:10:30 +00:00

Repo-side hardening in spikersoft-infrastructure PR #30: both jaeger services move from restart_policy on-failure/max_attempts=3 (which is how the backend stayed dead ~24h — swarm permanently gave up) to condition: any with 30s delay, so tracing self-heals after transient failures. Remaining OPS to resurrect the current outage: on SERVER check sysctl vm.max_map_count (ES needs ≥262144; persist via /etc/sysctl.d if low) and disk headroom vs the 95% flood-stage watermark, then docker service update --force jaeger_elasticsearch and read the real bootstrap error from docker service logs. Redeploying the stack from the committed file after merge also clears the README §8.1 wrong-image anomaly if it's still present.

spikerj commented

2026-07-12 00:23:03 +00:00

Correction after checking Jaeger directly (http://192.168.0.105:16686): tracing is NOT down — /api/services returns 200 and traces are queryable (e.g. trace 2890c9d4...). So Jaeger is serving.

Root cause is more subtle: the deployed image is Jaeger 2.17.0, but the stack still passes Jaeger v1 env vars (SPAN_STORAGE_TYPE=elasticsearch, ES_SERVER_URLS=http://elasticsearch:9200, ES_USERNAME/ES_PASSWORD). The Jaeger v2 binary does not consume those v1 vars, so it silently falls back to in-memory storage. Evidence: a 24h lookback returns no spans older than a few minutes — i.e. volatile, bounded, lost on restart.

Revised impact: tracing works but has no persistence/retention (in-memory only), and the elasticsearch service is effectively unused — so its being down (#481 as originally written) is a symptom of storage config drift from the v1->v2 upgrade, not an outage.

Revised fix options: (a) configure Jaeger 2.x properly (v2 config file / JAEGER_STORAGE_* or the OTel-style config) to use a healthy Elasticsearch, then fix the ES service; or (b) if in-memory is acceptable, remove the dead elasticsearch service and the vestigial v1 ES env vars to avoid confusion. Either way, decide on persistent vs in-memory intentionally.

Correction after checking Jaeger directly (http://192.168.0.105:16686): tracing is NOT down — /api/services returns 200 and traces are queryable (e.g. trace 2890c9d4...). So Jaeger is serving. Root cause is more subtle: the deployed image is Jaeger **2.17.0**, but the stack still passes **Jaeger v1** env vars (SPAN_STORAGE_TYPE=elasticsearch, ES_SERVER_URLS=http://elasticsearch:9200, ES_USERNAME/ES_PASSWORD). The Jaeger v2 binary does not consume those v1 vars, so it silently falls back to **in-memory storage**. Evidence: a 24h lookback returns no spans older than a few minutes — i.e. volatile, bounded, lost on restart. Revised impact: tracing works but has **no persistence/retention** (in-memory only), and the elasticsearch service is effectively unused — so its being down (#481 as originally written) is a symptom of storage config drift from the v1->v2 upgrade, not an outage. Revised fix options: (a) configure Jaeger 2.x properly (v2 config file / JAEGER_STORAGE_* or the OTel-style config) to use a healthy Elasticsearch, then fix the ES service; or (b) if in-memory is acceptable, remove the dead elasticsearch service and the vestigial v1 ES env vars to avoid confusion. Either way, decide on persistent vs in-memory intentionally.

spikerj referenced this issue

2026-07-12 07:45:34 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services #503

spikerj referenced this issue

2026-07-12 18:29:39 +00:00

[Exploration][Infra] Resilience to SERVER node flaps — restart-policy windows, auto-reconcile sweep, de-SPOF the SERVER-pinned stateful tier #510

spikerj commented

2026-07-12 20:28:56 +00:00

Fresh verification (2026-07-12 ~20:30 UTC) from the Jaeger query API: the tracing backend is serving and retaining traces again — /api/services lists the full service catalog, and trace queries return results with multi-hour history (samples at 18:23 UTC still queryable 2h+ later). That is inconsistent with a down storage backend.

Two possible explanations: (a) jaeger_elasticsearch recovered — the #510-B auto-reconciler (merged today in backend #240) force-refreshes exactly this failure shape every sweep, or the earlier disk pressure that likely caused the exit-1 loop cleared; or (b) jaeger fell back to memory storage (would also explain limited retention — I found no traces older than ~2h, though that window was quiet).

One command distinguishes them (needs swarm access): docker service ps jaeger_elasticsearch --no-trunc | head -5 — if it shows Running with a recent start, this ticket can close; if still exit-1 looping, the next diagnostic is docker service logs jaeger_elasticsearch --tail 50 (classic causes: vm.max_map_count too low, disk watermark, data-dir permissions). Happy to take the log output from there.

Fresh verification (2026-07-12 ~20:30 UTC) from the Jaeger query API: **the tracing backend is serving and retaining traces again** — `/api/services` lists the full service catalog, and trace queries return results with multi-hour history (samples at 18:23 UTC still queryable 2h+ later). That is inconsistent with a down storage backend. Two possible explanations: (a) `jaeger_elasticsearch` recovered — the #510-B auto-reconciler (merged today in backend #240) force-refreshes exactly this failure shape every sweep, or the earlier disk pressure that likely caused the exit-1 loop cleared; or (b) jaeger fell back to memory storage (would also explain limited retention — I found no traces older than ~2h, though that window was quiet). **One command distinguishes them** (needs swarm access): `docker service ps jaeger_elasticsearch --no-trunc | head -5` — if it shows Running with a recent start, this ticket can close; if still exit-1 looping, the next diagnostic is `docker service logs jaeger_elasticsearch --tail 50` (classic causes: `vm.max_map_count` too low, disk watermark, data-dir permissions). Happy to take the log output from there.

spikerj commented

2026-07-13 00:52:35 +00:00

Status 2026-07-13: the restart-policy fix (8a8889e, condition: any) IS deployed — jaeger_elasticsearch now retries forever instead of exhausting MaxAttempts. But the underlying failure persists: task exits 1 within ~30s of every start, so the service sits 0/1 in a permanent restart cycle on SERVER. Root cause still unknown — cannot read container logs because SERVER's log endpoint is down (docker service logs → 'node jq6qtygwrlyk9od36supw0v4i is not available', see #503). jaeger_jaeger itself is up 1/1 (6h), just storing nothing. Next diagnostic step needs SERVER log access or docker logs run directly on SERVER.

Status 2026-07-13: the restart-policy fix (8a8889e, condition: any) IS deployed — jaeger_elasticsearch now retries forever instead of exhausting MaxAttempts. But the underlying failure persists: task exits 1 within ~30s of every start, so the service sits 0/1 in a permanent restart cycle on SERVER. Root cause still unknown — cannot read container logs because SERVER's log endpoint is down (`docker service logs` → 'node jq6qtygwrlyk9od36supw0v4i is not available', see #503). jaeger_jaeger itself is up 1/1 (6h), just storing nothing. Next diagnostic step needs SERVER log access or `docker logs` run directly on SERVER.

spikerj referenced this issue

2026-07-13 00:52:48 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services #503

spikerj commented

2026-07-13 02:54:50 +00:00

Root cause found (2026-07-13 ~03:00Z) — diagnosable at last because the SERVER crash/reboot restored the node's log RPC endpoint:

java.lang.IllegalStateException: data path [/usr/share/elasticsearch/data] is not compatible with Elasticsearch v7.17.0, perhaps it has already been upgraded to a later version
Caused by: [/usr/share/elasticsearch/data/nodes] is a file which contains [written by Elasticsearch 8.11.0 to prevent a downgrade to a version prior to v8.0.0 which would result in data loss]

The service pins docker.elastic.co/elasticsearch/elasticsearch:7.17.0 but its data volume was initialized by 8.11.0 — every start is a refused downgrade, exit 1. The 8a8889e restart:any change means it now retries that refusal forever (~every 30s), which is the oscillation I've been seeing all session. Two fixes:

Bump the image to 8.11.x in jaeger/docker-stack.yml — jaeger 2.19 speaks ES 8; data volume stays. (Check ES_* env compat — 8.x enables security by default, may need xpack.security.enabled=false for parity.)
Wipe the ES data volume and stay on 7.17.0 — loses stored traces (probably acceptable for tracing data, team's call).

Either way the current state burns a restart every 30s on SERVER for nothing.

**Root cause found (2026-07-13 ~03:00Z)** — diagnosable at last because the SERVER crash/reboot restored the node's log RPC endpoint: ``` java.lang.IllegalStateException: data path [/usr/share/elasticsearch/data] is not compatible with Elasticsearch v7.17.0, perhaps it has already been upgraded to a later version Caused by: [/usr/share/elasticsearch/data/nodes] is a file which contains [written by Elasticsearch 8.11.0 to prevent a downgrade to a version prior to v8.0.0 which would result in data loss] ``` The service pins `docker.elastic.co/elasticsearch/elasticsearch:7.17.0` but its data volume was initialized by **8.11.0** — every start is a refused downgrade, exit 1. The 8a8889e restart:any change means it now retries that refusal forever (~every 30s), which is the oscillation I've been seeing all session. Two fixes: 1. **Bump the image to 8.11.x** in jaeger/docker-stack.yml — jaeger 2.19 speaks ES 8; data volume stays. (Check ES_* env compat — 8.x enables security by default, may need `xpack.security.enabled=false` for parity.) 2. Wipe the ES data volume and stay on 7.17.0 — loses stored traces (probably acceptable for tracing data, team's call). Either way the current state burns a restart every 30s on SERVER for nothing.

~~spikerj referenced this issue 2026-07-14 03:21:36 +00:00~~

[Exploration][Infra] Resilience to SERVER node flaps — restart-policy windows, auto-reconcile sweep, de-SPOF the SERVER-pinned stateful tier #510

spikerj referenced this issue

2026-07-14 06:16:55 +00:00

[Bug][Infra][Jaeger] jaeger_elasticsearch crash-loops (0/1) — ES 7.17.0 image started against a data volume written by ES 8.11.0, restarting every ~30s #551

spikerj referenced this issue

2026-07-20 23:17:03 +00:00

EPIC: Angular 22 + Nx 23 full-leverage modernization (signal forms, resource APIs, affected CI, remote cache) #722

spikerj commented

2026-07-22 02:57:51 +00:00

Board-sweep closing as superseded: the restart-policy stopgap merged (infra #30) and the actual root cause (ES version/data-path mismatch) was fixed under #551 (ES 8.11.0 pin + jaeger v2 --config). Tracing-backend health is #551's verification; reopen only if the symptom recurs post-#551.