[Bug][Infra][Jaeger] jaeger_elasticsearch crash-loops (0/1) — ES 7.17.0 image started against a data volume written by ES 8.11.0, restarting every ~30s #551

New Issue

2026-07-14T02:23:32Z

spikerj commented

2026-07-14 02:23:32 +00:00

QA Team — found during QA log watch 2026-07-14 ~02:20Z.

Symptom: jaeger_elasticsearch (stack jaeger, image docker.elastic.co/elasticsearch/elasticsearch:7.17.0) is stuck 0/1 in a tight crash-loop on SERVER — task: non-zero exit (1) roughly every 30s, indefinitely (4+ failures in the last few minutes of task history alone, and it has been looping for a while).

Root cause (from service logs):

java.lang.IllegalStateException: data path [/usr/share/elasticsearch/data] is not compatible with Elasticsearch v7.17.0, perhaps it has already been upgraded to a later version
Likely root cause: [/usr/share/elasticsearch/data/nodes] is a file which contains [written by Elasticsearch 8.11.0 to prevent a downgrade to a version prior to v8.0.0 which would result in data loss]

The data volume was last written by ES 8.11.0; the stack now pins 7.17.0. ES refuses the downgrade by design. Every restart hits the same wall — it can never self-recover.

Impact: No user-facing trace loss observed — Jaeger itself (jaeger_jaeger, 2.19.0) is healthy: UI answers 200 and /api/services reports all 19 services, so this ES instance is not the active trace store. Cost is an infinite restart loop on SERVER (CPU churn, log spam, permanent 0/1 noise in docker service ls that masks real failures).

Fix options (pick one):

If this ES is vestigial after the Jaeger v2 migration (likely — Jaeger works without it): remove the service from the jaeger stack.
If it's meant to be the trace backend: bump the image back to 8.11.x to match the volume, or point 7.17.0 at a fresh empty volume (accepting loss of the old index data).

Either way, decide deliberately — option 2's fresh-volume path deletes whatever the 8.11.0 data holds.

**QA Team** — found during QA log watch 2026-07-14 ~02:20Z. **Symptom:** `jaeger_elasticsearch` (stack `jaeger`, image `docker.elastic.co/elasticsearch/elasticsearch:7.17.0`) is stuck 0/1 in a tight crash-loop on SERVER — `task: non-zero exit (1)` roughly every 30s, indefinitely (4+ failures in the last few minutes of task history alone, and it has been looping for a while). **Root cause (from service logs):** ``` java.lang.IllegalStateException: data path [/usr/share/elasticsearch/data] is not compatible with Elasticsearch v7.17.0, perhaps it has already been upgraded to a later version Likely root cause: [/usr/share/elasticsearch/data/nodes] is a file which contains [written by Elasticsearch 8.11.0 to prevent a downgrade to a version prior to v8.0.0 which would result in data loss] ``` The data volume was last written by **ES 8.11.0**; the stack now pins **7.17.0**. ES refuses the downgrade by design. Every restart hits the same wall — it can never self-recover. **Impact:** No user-facing trace loss observed — Jaeger itself (`jaeger_jaeger`, 2.19.0) is healthy: UI answers 200 and `/api/services` reports all 19 services, so this ES instance is not the active trace store. Cost is an infinite restart loop on SERVER (CPU churn, log spam, permanent 0/1 noise in `docker service ls` that masks real failures). **Fix options (pick one):** 1. If this ES is vestigial after the Jaeger v2 migration (likely — Jaeger works without it): remove the service from the jaeger stack. 2. If it's meant to be the trace backend: bump the image back to 8.11.x to match the volume, or point 7.17.0 at a fresh empty volume (accepting loss of the old index data). Either way, decide deliberately — option 2's fresh-volume path deletes whatever the 8.11.0 data holds.

spikerj commented

2026-07-14 06:16:55 +00:00

Root cause is deeper than the crash loop, and the crash loop was hiding it.

The image pin is fixed — ES 8.11.0 (matching what wrote the data volume) is now 1/1, cluster green, stable through multiple restart windows. The 7.17.0 downgrade could never have started; ES refuses to open a data path written by a newer version.

But Jaeger was never using Elasticsearch at all. jaegertracing/jaeger:2.19.0 is Jaeger v2, an OpenTelemetry Collector distribution. It does not read the v1 environment variables the stack was setting (SPAN_STORAGE_TYPE, ES_SERVER_URLS, ES_USERNAME/ES_PASSWORD, COLLECTOR_OTLP_*, QUERY_BASE_PATH). Those were inert. With no --config flag the binary logs:

No '--config' flags detected, using default All-in-One configuration with memory storage.

Confirmed in prod: after 13 hours of uptime, ES held zero indices. Every span was in RAM and died with the container — so the ES crash loop in this ticket was, ironically, damaging nothing, and fixing it alone would have changed nothing either.

Fix (infrastructure PR #76, merged): add a Jaeger v2 config.yaml selecting the ES backend, pass --config, ship it as a swarm config object, and delete the dead env block. Derived from upstream's config-elasticsearch.yaml at the v2.19.0 tag, with shards 1 / replicas 0 because ES here is single-node (replicas > 0 would park the cluster at YELLOW forever with unassignable shards).

Schema was validated by running the real 2.19.0 image against the config before touching the live service: all components built and storage initialized; the only error was DNS for the ES hostname, expected with no network attached.

That validation also surfaced a durability point worth keeping: v2 exits non-zero if ES is unreachable at startup. That makes the condition: any restart policy from #481 load-bearing — an on-failure+max_attempts policy would permanently strand tracing the first time ES came up slower than Jaeger.

Staying open until the stack is redeployed and jaeger-main-* indices appear. Acceptance: a trace survives a Jaeger restart.

— macbook-claude-session

Root cause is deeper than the crash loop, and the crash loop was hiding it. **The image pin is fixed** — ES 8.11.0 (matching what wrote the data volume) is now 1/1, cluster green, stable through multiple restart windows. The 7.17.0 downgrade could never have started; ES refuses to open a data path written by a newer version. **But Jaeger was never using Elasticsearch at all.** `jaegertracing/jaeger:2.19.0` is Jaeger **v2**, an OpenTelemetry Collector distribution. It does not read the v1 environment variables the stack was setting (`SPAN_STORAGE_TYPE`, `ES_SERVER_URLS`, `ES_USERNAME`/`ES_PASSWORD`, `COLLECTOR_OTLP_*`, `QUERY_BASE_PATH`). Those were inert. With no `--config` flag the binary logs: ``` No '--config' flags detected, using default All-in-One configuration with memory storage. ``` Confirmed in prod: after 13 hours of uptime, ES held **zero indices**. Every span was in RAM and died with the container — so the ES crash loop in this ticket was, ironically, damaging nothing, and fixing it alone would have changed nothing either. **Fix** (infrastructure PR #76, merged): add a Jaeger v2 `config.yaml` selecting the ES backend, pass `--config`, ship it as a swarm config object, and delete the dead env block. Derived from upstream's `config-elasticsearch.yaml` at the v2.19.0 tag, with shards 1 / replicas 0 because ES here is single-node (replicas > 0 would park the cluster at YELLOW forever with unassignable shards). Schema was validated by running the real 2.19.0 image against the config before touching the live service: all components built and storage initialized; the only error was DNS for the ES hostname, expected with no network attached. That validation also surfaced a durability point worth keeping: **v2 exits non-zero if ES is unreachable at startup**. That makes the `condition: any` restart policy from #481 load-bearing — an `on-failure`+`max_attempts` policy would permanently strand tracing the first time ES came up slower than Jaeger. Staying open until the stack is redeployed and `jaeger-main-*` indices appear. Acceptance: a trace survives a Jaeger restart. — macbook-claude-session

spikerj referenced this issue

2026-07-22 02:57:51 +00:00

[Bug][Infra] jaeger_elasticsearch down ~24h on SERVER (exit 1) — MaxAttempts exhausted, tracing backend offline #481

spikerj commented

2026-07-22 02:57:52 +00:00

Board-sweep closing on merged evidence: ES pinned 8.11.0 to match the data volume (infra #66) AND jaeger v2 given --config so ES storage is actually used instead of silent in-memory (infra #76 — the deeper issue from the last comment). Reopen if spans still vanish across a jaeger restart.

spikerj closed this issue

2026-07-22 02:57:52 +00:00

spikerj referenced this issue

2026-07-23 04:33:41 +00:00

CourseModule constructor never assigns this.number — every module.number is undefined #809

spikerj referenced this issue

2026-07-23 04:59:07 +00:00

Boss.selectBossType() is a broken dead stub — throws on any call (empty bossTypesForFloor) #810

spikerj referenced this issue

2026-07-23 05:47:02 +00:00

wasm-voxel AES is broken — round function missing, keystream is input/IV-independent (not real AES) #811

spikerj referenced this issue