spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Infra] InfluxDB rejecting a client every ~10s ("authorization not found") — stale/missing token on a poller #483

New Issue
Open
opened 2026-07-12 00:04:27 +00:00 by spikerj · 8 comments
spikerj commented 2026-07-12 00:04:27 +00:00
Owner
Copy Link
Copy Source

Summary

influxDB_influxdb logs a rejected request every ~10 seconds (90 in a 15-min sample) with Unauthorized / "authorization not found". A steady automated client is polling an authenticated InfluxDB endpoint (query or write) with a missing/invalid token, so whatever data path it drives is silently failing.

Symptoms

lvl=info msg=Unauthorized log_id=... error="authorization not found"
  • Exactly one per ~10s -> an automated scraper/poller, not ad-hoc traffic.
  • All lvl=info (Influx logs auth rejections at info), no error/warn — easy to miss.
  • Cadence rules out an unauthenticated liveness probe (/health//ping don't log this); it's a real query/write being rejected.

Likely sources to check (10s cadence)

  • Grafana InfluxDB datasource with a stale/rotated token (graphana_grafana).
  • spikersoft-influx-dashboard service token/env.
  • A Telegraf/collector or healthcheck configured against /api/v2/* with no token.

Suggested investigation

  1. Correlate by client IP: temporarily raise Influx HTTP request logging, or check the reverse-proxy/traefik access logs for the 10s-cadence caller to influxdb:8086.
  2. Once identified, re-issue/rotate the InfluxDB token and update the client's config (env/secret).
  3. Confirm the 10s Unauthorized stream stops.

Impact

Low urgency but a real broken data path — a dashboard/metrics pipeline is likely receiving no data, and the noise masks genuine auth failures.

Filed proactively by automated swarm health check (docker service log audit).

## Summary `influxDB_influxdb` logs a rejected request **every ~10 seconds** (90 in a 15-min sample) with `Unauthorized / "authorization not found"`. A steady automated client is polling an authenticated InfluxDB endpoint (query or write) with a missing/invalid token, so whatever data path it drives is silently failing. ## Symptoms ``` lvl=info msg=Unauthorized log_id=... error="authorization not found" ``` - Exactly one per ~10s -> an automated scraper/poller, not ad-hoc traffic. - All `lvl=info` (Influx logs auth rejections at info), no error/warn — easy to miss. - Cadence rules out an unauthenticated liveness probe (`/health`/`/ping` don't log this); it's a real query/write being rejected. ## Likely sources to check (10s cadence) - Grafana InfluxDB datasource with a stale/rotated token (`graphana_grafana`). - `spikersoft-influx-dashboard` service token/env. - A Telegraf/collector or healthcheck configured against `/api/v2/*` with no token. ## Suggested investigation 1. Correlate by client IP: temporarily raise Influx HTTP request logging, or check the reverse-proxy/traefik access logs for the 10s-cadence caller to `influxdb:8086`. 2. Once identified, re-issue/rotate the InfluxDB token and update the client's config (env/secret). 3. Confirm the 10s `Unauthorized` stream stops. ## Impact Low urgency but a real broken data path — a dashboard/metrics pipeline is likely receiving no data, and the noise masks genuine auth failures. --- _Filed proactively by automated swarm health check (docker service log audit)._
spikerj commented 2026-07-12 00:12:21 +00:00
Author
Owner
Copy Link
Copy Source

Likely caller identified: spikersoft-backend (SpikerSoft.Api) polls InfluxDB every ~10s. Its InfluxDB health check hits /ping and /ready (no auth -> the health check reports Healthy), but the app also fires POST /api/v2/query and POST /api/v2/write on the same cadence, which DO require a token. The backend's InfluxDB health check was seen flapping Unhealthy->Degraded ('InfluxDB has 1 issues') in the same window. Check the InfluxDB token/org/bucket config the backend uses (SpikerSoft.Api InfluxDB settings) — a stale/rotated token there matches the steady 10s 'authorization not found' stream on the Influx side.

Likely caller identified: `spikersoft-backend` (SpikerSoft.Api) polls InfluxDB every ~10s. Its InfluxDB health check hits `/ping` and `/ready` (no auth -> the health check reports Healthy), but the app also fires `POST /api/v2/query` and `POST /api/v2/write` on the same cadence, which DO require a token. The backend's InfluxDB health check was seen flapping Unhealthy->Degraded ('InfluxDB has 1 issues') in the same window. Check the InfluxDB token/org/bucket config the backend uses (SpikerSoft.Api InfluxDB settings) — a stale/rotated token there matches the steady 10s 'authorization not found' stream on the Influx side.
spikerj commented 2026-07-12 00:12:30 +00:00
Author
Owner
Copy Link
Copy Source

Likely caller: spikersoft-backend (SpikerSoft.Api) polls InfluxDB every ~10s. Its health check hits /ping and /ready (no auth, so the check reports Healthy), but the app also fires POST /api/v2/query and POST /api/v2/write on the same cadence, which require a token — matching the steady 10s 'authorization not found' stream. Its InfluxDB health check was also seen flapping Unhealthy->Degraded. Check the InfluxDB token/org/bucket the backend uses for a stale/rotated token.

Likely caller: spikersoft-backend (SpikerSoft.Api) polls InfluxDB every ~10s. Its health check hits /ping and /ready (no auth, so the check reports Healthy), but the app also fires POST /api/v2/query and POST /api/v2/write on the same cadence, which require a token — matching the steady 10s 'authorization not found' stream. Its InfluxDB health check was also seen flapping Unhealthy->Degraded. Check the InfluxDB token/org/bucket the backend uses for a stale/rotated token.
spikerj commented 2026-07-12 00:17:58 +00:00
Author
Owner
Copy Link
Copy Source

Triage from the repo side (no prod probing from this session): the ~10s cadence exactly matches Traefik's influxdb2 metrics push — traefik/docker-stack.yml pushes to https://influxdb.spikersoft.com every 10s (default pushInterval) with the token committed at line 41, org=spikersoft bucket=traefik. Prime suspect: that token is stale/revoked (it's also the README §9 committed-secret wart, so it may have been rotated in Influx without the stack file following). Quick confirmation on any box: curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Token <token from traefik/docker-stack.yml:41>" 'https://influxdb.spikersoft.com/api/v2/buckets?name=traefik' — 401 confirms Traefik as the caller; then issue a fresh token scoped write-only to the traefik bucket, update the stack file + redeploy traefik, and the Unauthorized stream should stop. If that returns 200, next suspects in order: grafana's Influx datasource and the jetson monitors' INFLUX_TOKEN env.

Triage from the repo side (no prod probing from this session): the ~10s cadence exactly matches Traefik's influxdb2 metrics push — traefik/docker-stack.yml pushes to https://influxdb.spikersoft.com every 10s (default pushInterval) with the token committed at line 41, org=spikersoft bucket=traefik. Prime suspect: that token is stale/revoked (it's also the README §9 committed-secret wart, so it may have been rotated in Influx without the stack file following). Quick confirmation on any box: curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Token <token from traefik/docker-stack.yml:41>" 'https://influxdb.spikersoft.com/api/v2/buckets?name=traefik' — 401 confirms Traefik as the caller; then issue a fresh token scoped write-only to the traefik bucket, update the stack file + redeploy traefik, and the Unauthorized stream should stop. If that returns 200, next suspects in order: grafana's Influx datasource and the jetson monitors' INFLUX_TOKEN env.
spikerj commented 2026-07-12 19:26:29 +00:00
Author
Owner
Copy Link
Copy Source

Fix up: infra PR #46 (fix/483-traefik-influx-token-env). Attribution firmed up without probing the credential: SpikerSoft.Api's InfluxDB health check is currently Healthy in prod (its authenticated query/write path works), which eliminates it as the unauthorized caller — leaving traefik's 10s influxdb2 metrics push, whose token is the one committed in traefik/docker-stack.yml.

The PR parameterizes the token to ${TRAEFIK_INFLUX_TOKEN} (shell-env deploy, README §10 documents the command) and marks the committed token as burned. After merge, operator steps for Joey (traefik has no CI deploy):

  1. Influx UI → Load Data → API Tokens → issue a write-only token scoped to bucket traefik, and revoke the old committed token.
  2. Pull /mnt/infrastructure, then export TRAEFIK_INFLUX_TOKEN=... + sudo -E docker stack deploy --with-registry-auth -c docker-stack.yml traefik.
  3. Verify: the 10s authorization not found stream stops; Grafana's traefik dashboard starts receiving fresh data.

Will close once deployed + the log stream is confirmed quiet.

Fix up: **infra PR #46** (`fix/483-traefik-influx-token-env`). Attribution firmed up without probing the credential: SpikerSoft.Api's InfluxDB health check is currently **Healthy** in prod (its authenticated query/write path works), which eliminates it as the unauthorized caller — leaving traefik's 10s influxdb2 metrics push, whose token is the one committed in `traefik/docker-stack.yml`. The PR parameterizes the token to `${TRAEFIK_INFLUX_TOKEN}` (shell-env deploy, README §10 documents the command) and marks the committed token as burned. **After merge, operator steps for Joey** (traefik has no CI deploy): 1. Influx UI → Load Data → API Tokens → issue a **write-only token scoped to bucket `traefik`**, and **revoke the old committed token**. 2. Pull /mnt/infrastructure, then `export TRAEFIK_INFLUX_TOKEN=...` + `sudo -E docker stack deploy --with-registry-auth -c docker-stack.yml traefik`. 3. Verify: the 10s `authorization not found` stream stops; Grafana's traefik dashboard starts receiving fresh data. Will close once deployed + the log stream is confirmed quiet.
spikerj commented 2026-07-13 19:10:16 +00:00
Author
Owner
Copy Link
Copy Source

QA watch 2026-07-13 ~19:10Z — still reproducing, exactly 6/min: 60 'authorization not found' in the 10 min to 19:07Z on influxDB_influxdb (dreamstream4), steady 10s cadence. The server log line carries NO client identity (msg=Unauthorized, no addr/path — request logging off), and none of the obvious .NET pollers (influx-dashboard, node-agent, docker-monitor) log a matching 401/unauthorized client-side, so the offender is likely a telegraf/jetson collector or an external scraper with a stale token. Cheapest next diagnostic: either enable Influx http request logging briefly, or tcpdump port 8086 on dreamstream4 for ~30s and read the source IP — the 10s period makes it trivial to catch.

QA watch 2026-07-13 ~19:10Z — still reproducing, exactly 6/min: 60 'authorization not found' in the 10 min to 19:07Z on influxDB_influxdb (dreamstream4), steady 10s cadence. The server log line carries NO client identity (msg=Unauthorized, no addr/path — request logging off), and none of the obvious .NET pollers (influx-dashboard, node-agent, docker-monitor) log a matching 401/unauthorized client-side, so the offender is likely a telegraf/jetson collector or an external scraper with a stale token. Cheapest next diagnostic: either enable Influx http request logging briefly, or tcpdump port 8086 on dreamstream4 for ~30s and read the source IP — the 10s period makes it trivial to catch.
spikerj commented 2026-07-14 06:30:36 +00:00
Author
Owner
Copy Link
Copy Source

Found it: it's Traefik's InfluxDB metrics push

Traefik's metrics.influxdb2 exporter pushes on a 10s interval by default — an exact match for the reported cadence (90 rejections per 15-min sample = 1 per 10s). It was never Grafana, the dashboard service, or a healthcheck.

This ticket was "fixed" once and is still broken

The original cause was a burned token committed to git. The earlier fix moved it out of git and into a value supplied by the deploying shell — but via a bare interpolation, which has two nasty properties:

  1. If the deployer forgets to export it, it silently interpolates to an empty string. No warning, anywhere.
  2. Traefik does not disable influxdb2 metrics on an empty token. It keeps pushing every 10s, and Influx keeps rejecting.

That is exactly what the 2026-07-13 17:01Z redeploy did. I confirmed the live service is running right now with a zero-length token argument.

So #483 never actually stopped — it only changed flavour, from burned token to empty token. Same 401 every 10s, same permanently broken metrics path. (The server-side log wording differs between the two: a presented-but-deleted token logs authorization not found, which is what was originally sampled.)

Fix

infrastructure PR #79: switch to a required-variable interpolation so docker stack deploy hard-fails (rc=1) when the value is unset or empty. Traefik can then never again be deployed with an empty metrics token. Same guard the object-store stack already uses for its credentials.

Verified against the real engine with docker stack config:

case result
unset rc=1, hard failure with a clear message
set but empty rc=1 — this is the 17:01Z bug, now impossible
set to a value rc=0, interpolates correctly

It also corrects a stale comment claiming stack deploy mishandles required-variable syntax. It doesn't — it only corrupts the interpolation when the message contains parentheses or #. Noted inline so the next person doesn't re-learn it.

Still needed after merge

The old token is burned, so merging only makes the failure loud. The metrics path starts working again once a fresh write-scoped token for the metrics bucket is issued and exported at deploy time (documented in the infra README). Until then the deploy will correctly refuse to run rather than silently ship a broken push.

Relationship to #554

Adjacent, not the same root cause — they share only the 17:01Z redeploy wave and a coincidental ~10s cadence. #554 is a host-side jtop socket failure and has already self-recovered; commented there.

Leaving open for merge + verify.

— macbook-claude-session

## Found it: it's Traefik's InfluxDB metrics push Traefik's `metrics.influxdb2` exporter pushes on a **10s interval by default** — an exact match for the reported cadence (90 rejections per 15-min sample = 1 per 10s). It was never Grafana, the dashboard service, or a healthcheck. ## This ticket was "fixed" once and is still broken The original cause was a **burned token committed to git**. The earlier fix moved it out of git and into a value supplied by the deploying shell — but via a **bare** interpolation, which has two nasty properties: 1. If the deployer forgets to export it, it silently interpolates to an **empty string**. No warning, anywhere. 2. Traefik does **not** disable influxdb2 metrics on an empty token. It keeps pushing every 10s, and Influx keeps rejecting. That is exactly what the **2026-07-13 17:01Z** redeploy did. I confirmed the live service is running **right now** with a **zero-length** token argument. So #483 never actually stopped — it only changed flavour, from *burned token* to *empty token*. Same 401 every 10s, same permanently broken metrics path. (The server-side log wording differs between the two: a presented-but-deleted token logs `authorization not found`, which is what was originally sampled.) ## Fix infrastructure PR **#79**: switch to a **required-variable** interpolation so `docker stack deploy` **hard-fails (`rc=1`)** when the value is unset *or empty*. Traefik can then never again be deployed with an empty metrics token. Same guard the object-store stack already uses for its credentials. Verified against the real engine with `docker stack config`: | case | result | |---|---| | unset | `rc=1`, hard failure with a clear message | | set but **empty** | `rc=1` — this is the 17:01Z bug, now impossible | | set to a value | `rc=0`, interpolates correctly | It also corrects a stale comment claiming `stack deploy` mishandles required-variable syntax. It doesn't — it only corrupts the interpolation when the *message* contains parentheses or `#`. Noted inline so the next person doesn't re-learn it. ## Still needed after merge The old token is **burned**, so merging only makes the failure *loud*. The metrics path starts working again once a **fresh write-scoped token for the metrics bucket** is issued and exported at deploy time (documented in the infra README). Until then the deploy will correctly **refuse to run** rather than silently ship a broken push. ## Relationship to #554 Adjacent, **not** the same root cause — they share only the 17:01Z redeploy wave and a coincidental ~10s cadence. #554 is a host-side jtop socket failure and has already self-recovered; commented there. Leaving open for merge + verify. — macbook-claude-session
spikerj commented 2026-07-22 02:58:16 +00:00
Author
Owner
Copy Link
Copy Source

Board-sweep status (2026-07-22): token out of git (infra #46) + deploy fails-closed on missing token (#79). REMAINING: provision a VALID influxdb2 token and verify the Traefik metrics-push rejections stop ('fixed once and still broken' per the last comment).

Board-sweep status (2026-07-22): token out of git (infra #46) + deploy fails-closed on missing token (#79). REMAINING: provision a VALID influxdb2 token and verify the Traefik metrics-push rejections stop ('fixed once and still broken' per the last comment).
spikerj commented 2026-07-29 15:49:19 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — the deploy-time guard shipped; the underlying rejection stream is not fixed. Staying open. Also flagging a secrets-hygiene item the audit turned up.

Landed (commits e2e3ebd / PR #46 and 086441d / PR #79): traefik/docker-stack.yml:64 now reads

--metrics.influxdb2.token=${TRAEFIK_INFLUX_TOKEN:?export TRAEFIK_INFLUX_TOKEN before deploy, see README section 10}

The :? required-variable form means a deploy fails closed rather than silently shipping an empty token, and the comment block at :44-59 documents the original trap. That's the right shape.

Not fixed: the ticket's actual symptom. A valid write-scoped token for the traefik bucket still has to be issued in InfluxDB and exported at deploy, and then the ~10s rejection stream confirmed stopped. No record of that operator step exists in any repo. Settling evidence would be: Influx log quiet for 15 minutes, plus the Grafana Traefik dashboard actually receiving data.

Secrets-hygiene item found while checking this. The old token is still committed in the tree at traefik/docker-stack-backup-10-3-2025.yml:21 — an un-scrubbed backup file. Removing the live reference from the active stack didn't remove the value from the repo, so whatever token that is remains readable to anyone with repo access and needs rotating, not just deleting (same reasoning as #600). Worth handling in the same pass as #633, which is tracking four other committed credentials.

Suggested order: rotate → issue the new write-scoped token into OpenBao → export at deploy → verify the stream stops → delete or scrub the backup file.

Audited against `origin/master` — **the deploy-time guard shipped; the underlying rejection stream is not fixed.** Staying open. Also flagging a secrets-hygiene item the audit turned up. **Landed** (commits `e2e3ebd` / PR #46 and `086441d` / PR #79): `traefik/docker-stack.yml:64` now reads ``` --metrics.influxdb2.token=${TRAEFIK_INFLUX_TOKEN:?export TRAEFIK_INFLUX_TOKEN before deploy, see README section 10} ``` The `:?` required-variable form means a deploy **fails closed** rather than silently shipping an empty token, and the comment block at `:44-59` documents the original trap. That's the right shape. **Not fixed:** the ticket's actual symptom. A valid write-scoped token for the `traefik` bucket still has to be issued in InfluxDB and exported at deploy, and then the ~10s rejection stream confirmed stopped. No record of that operator step exists in any repo. Settling evidence would be: Influx log quiet for 15 minutes, plus the Grafana Traefik dashboard actually receiving data. **Secrets-hygiene item found while checking this.** The old token is still committed in the tree at `traefik/docker-stack-backup-10-3-2025.yml:21` — an un-scrubbed backup file. Removing the live reference from the active stack didn't remove the value from the repo, so whatever token that is remains readable to anyone with repo access and needs rotating, not just deleting (same reasoning as #600). Worth handling in the same pass as #633, which is tracking four other committed credentials. Suggested order: rotate → issue the new write-scoped token into OpenBao → export at deploy → verify the stream stops → delete or scrub the backup file.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#483