[Bug][Backend][InfluxDashboard] dreamstream7 is invisible to the metrics dashboard — Hostnames config lists DreamStream1-6 while the swarm runs 7 jetson reporters #571

New Issue

2026-07-14T14:30:23Z

spikerj commented

2026-07-14 14:30:23 +00:00

QA Team — found during the README audit's config verification, 2026-07-14.

The bug: SpikerSoft.EventHandlers.InfluxDashboard/appsettings.Production.json configures:

"Hostnames": [ "DreamStream6", "DreamStream5", "DreamStream4", "DreamStream3", "DreamStream2", "DreamStream1" ]

— six nodes. But the swarm runs seven jetson-influx reporters (jetson-influxdb_jetson-influx-1 … -7, each pinned to its own dreamstream host, all 1/1 as of 2026-07-14), and docker node ls confirms dreamstream1–7 exist.

Impact: dreamstream7's CPU/RAM/storage metrics are being written to InfluxDB by its reporter but never queried or displayed — the node is a monitoring blind spot. Notably, dreamstream7 is where keycloak_postgres-keycloak landed after the 2026-07-14 failover (#552), so the node currently hosting the identity database is the one node nobody is watching.

Fix: add "DreamStream7" to the Hostnames array (and check the CpuFields list is right for that board — it lists cpu1..cpu6, which is a per-board core count, not a node count, so it likely needs no change).

Worth a second look: whether this list should be derived from swarm node membership rather than hand-maintained — the same class of drift will recur the next time a node is added.

**QA Team** — found during the README audit's config verification, 2026-07-14. **The bug:** `SpikerSoft.EventHandlers.InfluxDashboard/appsettings.Production.json` configures: ```json "Hostnames": [ "DreamStream6", "DreamStream5", "DreamStream4", "DreamStream3", "DreamStream2", "DreamStream1" ] ``` — six nodes. But the swarm runs **seven** jetson-influx reporters (`jetson-influxdb_jetson-influx-1` … `-7`, each pinned to its own dreamstream host, all 1/1 as of 2026-07-14), and `docker node ls` confirms dreamstream1–7 exist. **Impact:** dreamstream7's CPU/RAM/storage metrics are being written to InfluxDB by its reporter but **never queried or displayed** — the node is a monitoring blind spot. Notably, dreamstream7 is where `keycloak_postgres-keycloak` landed after the 2026-07-14 failover (#552), so the node currently hosting the identity database is the one node nobody is watching. **Fix:** add `"DreamStream7"` to the Hostnames array (and check the `CpuFields` list is right for that board — it lists cpu1..cpu6, which is a per-board core count, not a node count, so it likely needs no change). **Worth a second look:** whether this list should be derived from swarm node membership rather than hand-maintained — the same class of drift will recur the next time a node is added.

spikerj referenced this issue

2026-07-14 17:20:10 +00:00

[Backend][Metrics] SpikerSoft.Business.Metrics is inert — IInfluxMetricsService has no consumer, metric models are duplicated, namespace doesn't match the project, and InfluxDashboard re-implements the aggregation #577

spikerj commented

2026-07-14 21:55:11 +00:00

Fix in spikersoft-backend PR #293 (open, awaiting merge).

Confirmed exactly as filed — I checked the jetson-influxdb stack: seven reporters, jetson-influx-1..7, each pinned to node.hostname == dreamstream1..7. The allowlist listed six. DreamStream7 is now in both Production and Development.

And you were right about CpuFields needing no change: cpu1..cpu6 is a per-board core count, not a node count.

Your "worth a second look" — taken

whether this list should be derived from swarm node membership rather than hand-maintained — the same class of drift will recur the next time a node is added.

It can be derived, and from a better source than the swarm: InfluxDB already knows who is writing to it. Every query in IInfluxMetricsRepository filters to the allowlist, which is precisely why none of them could ever notice a node missing from it. Added DiscoverReportingHostsAsync — schema.tagValues on the host tag — which asks the inverse question. The worker now reconciles at startup and logs a MONITORING BLIND SPOT error naming any host that is reporting but unconfigured (plus a warning for configured hosts that have gone quiet).

So the next node addition announces itself instead of producing another silent blind spot.

What I stopped short of: having the dashboard query auto-discovered hosts rather than the configured list. That changes what a live dashboard renders based on whatever happens to be writing to the bucket, and I didn't want that blast radius on a monitoring service without you deciding it. Detecting the drift loudly gets the safety; retiring the allowlist entirely is a one-liner from here if you want it.

Two decisions that determine whether the alarm is usable

Case-insensitive comparison. The swarm's hostnames are lowercase (dreamstream7 — what docker node ls shows, and what the stack files pin placement to), and that's what lands in InfluxDB's host tag. The config spells them DreamStream7. A case-sensitive check would flag all seven nodes as both unwatched and silent on every cycle — a permanently-screaming alarm, which is the fastest way to train everyone to ignore it, and then the real dreamstream8 hides in the noise.
Empty discovery = "cannot tell", not "nobody is reporting." Otherwise a failed Influx query manufactures a false alarm claiming the whole fleet went dark.

Tests

9 new. DashboardHostnamesConfigTests asserts the actual appsettings files rather than a config object built in the test — the bug was a value in a JSON file, and only reading that file would have caught it. Verified it bites: reverting appsettings.Production.json to its 6-node state fails that test and only that test.

The node hosting the identity database being the one node nobody was watching is a good argument that this class of bug deserves the guard, not just the one-line fix.

Fix in spikersoft-backend PR #293 (open, awaiting merge). Confirmed exactly as filed — I checked the `jetson-influxdb` stack: seven reporters, `jetson-influx-1..7`, each pinned to `node.hostname == dreamstream1..7`. The allowlist listed six. `DreamStream7` is now in both Production and Development. And you were right about `CpuFields` needing no change: `cpu1..cpu6` is a per-board **core** count, not a node count. ## Your "worth a second look" — taken > whether this list should be derived from swarm node membership rather than hand-maintained — the same class of drift will recur the next time a node is added. It can be derived, and from a better source than the swarm: **InfluxDB already knows who is writing to it.** Every query in `IInfluxMetricsRepository` filters *to* the allowlist, which is precisely why none of them could ever notice a node missing from it. Added `DiscoverReportingHostsAsync` — `schema.tagValues` on the `host` tag — which asks the inverse question. The worker now reconciles at startup and logs a **`MONITORING BLIND SPOT` error** naming any host that is reporting but unconfigured (plus a warning for configured hosts that have gone quiet). So the next node addition announces itself instead of producing another silent blind spot. **What I stopped short of:** having the dashboard *query* auto-discovered hosts rather than the configured list. That changes what a live dashboard renders based on whatever happens to be writing to the bucket, and I didn't want that blast radius on a monitoring service without you deciding it. Detecting the drift loudly gets the safety; retiring the allowlist entirely is a one-liner from here if you want it. ## Two decisions that determine whether the alarm is usable - **Case-insensitive comparison.** The swarm's hostnames are lowercase (`dreamstream7` — what `docker node ls` shows, and what the stack files pin placement to), and that's what lands in InfluxDB's `host` tag. The config spells them `DreamStream7`. A case-sensitive check would flag all seven nodes as *both* unwatched *and* silent on every cycle — a permanently-screaming alarm, which is the fastest way to train everyone to ignore it, and then the real dreamstream8 hides in the noise. - **Empty discovery = "cannot tell", not "nobody is reporting."** Otherwise a failed Influx query manufactures a false alarm claiming the whole fleet went dark. ## Tests 9 new. `DashboardHostnamesConfigTests` asserts the **actual appsettings files** rather than a config object built in the test — the bug was a value in a JSON file, and only reading that file would have caught it. Verified it bites: reverting `appsettings.Production.json` to its 6-node state fails that test and only that test. The node hosting the identity database being the one node nobody was watching is a good argument that this class of bug deserves the guard, not just the one-line fix.

spikerj referenced this issue

2026-07-14 22:12:46 +00:00

Retire the three orphaned IInfluxMetricsRepository methods (QueryCpuMetricsAsync, QueryMetricsAsync, StreamMetricsAsync) #601

spikerj commented

2026-07-14 22:12:52 +00:00

Resolved in spikersoft-backend PR #293 (merged to master).

dreamstream7 is now in the InfluxDashboard worker's Hostnames for both Production and Development, so its metrics reach the dashboard.

More importantly, the next time a host drifts out of the config it won't go unnoticed: MetricsHostReconciliation compares the configured hostnames against the hosts actually reporting to InfluxDB (schema.tagValues on host, -1h lookback) and logs a MONITORING BLIND SPOT error naming any host that is reporting but unwatched. Comparison is case-insensitive, and an empty discovery result is treated as "cannot tell" rather than "nothing is reporting" — so a broken query can't masquerade as a clean bill of health.

Closing.

Resolved in spikersoft-backend PR #293 (merged to `master`). `dreamstream7` is now in the InfluxDashboard worker's `Hostnames` for both Production and Development, so its metrics reach the dashboard. More importantly, the next time a host drifts out of the config it won't go unnoticed: `MetricsHostReconciliation` compares the configured hostnames against the hosts actually reporting to InfluxDB (`schema.tagValues` on `host`, `-1h` lookback) and logs a `MONITORING BLIND SPOT` error naming any host that is reporting but unwatched. Comparison is case-insensitive, and an empty discovery result is treated as "cannot tell" rather than "nothing is reporting" — so a broken query can't masquerade as a clean bill of health. Closing.

spikerj closed this issue

2026-07-14 22:12:56 +00:00

spikerj referenced this issue

2026-07-24 20:02:41 +00:00

PhpInterpreterService: capabilities.canRun not reactive to setWorkspace (non-signal hasBlocks) #813

spikerj referenced this issue

2026-07-24 20:02:41 +00:00

QrCodeComponent: WiFi field escaping only escapes the first special char (regex missing /g) #814

spikerj referenced this issue

2026-07-24 20:34:37 +00:00

PhpInterpreterService: capabilities.canRun not reactive to setWorkspace (non-signal hasBlocks) #813

spikerj referenced this issue