[Bug][Backend][Infra] tiles.spikersoft.com has no backing service — API status/health canary fetches a terrain tile every few minutes and 404s 100% of the time #556

New Issue

2026-07-14T03:17:39Z

spikerj commented

2026-07-14 03:17:39 +00:00

QA Team — found during QA log watch 2026-07-14 ~03:20Z (traced from recurring error spans in Jaeger).

Symptom: the API repeatedly fetches https://tiles.spikersoft.com/data/terrain/10/283/389.webp (same tile coords every time — it's a canary/dependency probe inside GET api/status / health processing) roughly every 3–5 minutes, and every fetch fails. Direct check: tiles.spikersoft.com answers (traefik/TLS fine) but returns 404 for everything, including / — and docker service ls has no service matching tiles/terrain/map in the swarm. So the hostname routes to nothing.

Meaning one of two things (please pick and fix accordingly):

The tiles service was decommissioned/never migrated — then the API's canary (and whatever map feature consumes terrain tiles in the app) is dead code that should be removed from the status checks, and the traefik router + DNS name retired; or
A tiles service is supposed to run — then its stack is missing from the swarm entirely (not merely 0/0), which is its own deploy gap.

Impact today: api/status presumably reports the tiles dependency permanently degraded (nobody has noticed — consistent with #555's alerting gaps); Jaeger accumulates a steady drip of error traces that pollute error-rate monitoring; and if the frontend has a map/terrain feature, it's been silently broken.

Secondary observation from the same traces: something (via the reverse proxy at 10.0.1.18:8081) periodically probes the API at /healthz and gets 404 — the API doesn't implement that path. If that's a traefik/loadbalancer health check expecting 200, it's misconfigured (and lucky the check isn't enforcing); if it's an uptime monitor, it's been reporting the API down forever. Worth identifying the caller while fixing the above.

Evidence: backend logs 20 tile-fetch lines in 6h (02:59:17, 03:03:02, 03:03:09 latest burst); Jaeger error traces GET api/status → GET http://rabbitmq:15672/... + GET https://tiles.spikersoft.com/...; curl of the tile URL and root both → 404 in ~0.1s.

**QA Team** — found during QA log watch 2026-07-14 ~03:20Z (traced from recurring error spans in Jaeger). **Symptom:** the API repeatedly fetches `https://tiles.spikersoft.com/data/terrain/10/283/389.webp` (same tile coords every time — it's a canary/dependency probe inside `GET api/status` / health processing) roughly every 3–5 minutes, and **every fetch fails**. Direct check: `tiles.spikersoft.com` answers (traefik/TLS fine) but returns **404 for everything, including `/`** — and `docker service ls` has **no service matching tiles/terrain/map** in the swarm. So the hostname routes to nothing. **Meaning one of two things (please pick and fix accordingly):** 1. The tiles service was decommissioned/never migrated — then the API's canary (and whatever map feature consumes terrain tiles in the app) is dead code that should be removed from the status checks, and the traefik router + DNS name retired; or 2. A tiles service is *supposed* to run — then its stack is missing from the swarm entirely (not merely 0/0), which is its own deploy gap. **Impact today:** `api/status` presumably reports the tiles dependency permanently degraded (nobody has noticed — consistent with #555's alerting gaps); Jaeger accumulates a steady drip of error traces that pollute error-rate monitoring; and if the frontend has a map/terrain feature, it's been silently broken. **Secondary observation from the same traces:** something (via the reverse proxy at `10.0.1.18:8081`) periodically probes the API at **`/healthz` and gets 404** — the API doesn't implement that path. If that's a traefik/loadbalancer health check expecting 200, it's misconfigured (and lucky the check isn't enforcing); if it's an uptime monitor, it's been reporting the API down forever. Worth identifying the caller while fixing the above. **Evidence:** backend logs 20 tile-fetch lines in 6h (02:59:17, 03:03:02, 03:03:09 latest burst); Jaeger error traces `GET api/status → GET http://rabbitmq:15672/...` + `GET https://tiles.spikersoft.com/...`; curl of the tile URL and root both → 404 in ~0.1s.

spikerj commented

2026-07-14 06:13:52 +00:00

I need to rebuild this image from my windows desktop, and include the information and repo into the spikersoft infra repo.. put this on hold until I go this

spikerj referenced this issue

2026-07-14 06:20:59 +00:00

[Bug][Infra] API /healthz returns 503 — TileServerGL check failing (tiles.spikersoft.com 404s, no tile-server deployed) #484

spikerj commented

2026-07-14 06:21:29 +00:00

Diagnosis — your option 1 is correct

The tile service was never deployed / was decommissioned. Verified read-only against the swarm: a scan of all 94 services matches nothing on tile|map|terrain, and no stack file for one exists in any repo. The hostname completes TLS on the wildcard cert but matches no reverse-proxy router, so every path — including / — falls through to the proxy's catch-all 404 (a 19-byte 404 page not found). That fully explains "answers, TLS fine, 404s everything".

So the canary was probing a dependency that does not exist: a health check that failed 100% of the time, forever, producing a permanently Degraded rollup and the steady drip of error spans polluting error-rate monitoring.

Decision: remove the canary, do not deploy a tile server

PR #274 (backend) makes the tile health check opt-in (Enabled, default false) and registers it only when enabled and configured. With no tile server, the check is not registered at all — the API stops probing entirely: no canary, no 404 drip, no permanent Degraded. It also fixes a latent crash (an empty base URL threw UriFormatException during DI construction, which could have taken /healthz itself down). If a tile server is ever really deployed, flip it on — it stays Degraded-on-failure, so an optional dependency can never 503 the API (#484).

Why not just deploy one: tileserver-gl serves tiles out of an .mbtiles data file, and no terrain-DEM .mbtiles asset exists in any repo. Deploying the image with no data would still 404 /data/terrain/* — it would fix nothing. The blocker is a multi-GB data asset, not a missing compose file. Worth its own ticket if the terrain feature is wanted back.

Your two secondary points

Frontend map/terrain feature: yes, it is silently broken — but only the enhancement. The single consumer is the map-tracker 3D-terrain overlay (add3DTerrain(), a raster-DEM source). Base map layers come from OSM/ArcGIS and are unaffected — maps still render, you just get no 3D terrain. Not fixed here; needs a product call (drop the layer, or fund the DEM asset).
"The API doesn't implement /healthz" — this does not reproduce. /healthz is implemented (MapHealthChecks("/healthz")) and returns 200 right now. Whatever was 404ing was hitting a different path or port; if it recurs, worth a separate ticket with the exact request line.

Companion infrastructure PR corrects the service inventory, which wrongly listed map-tile-server as a live stack and told readers to commit its stack file — that doc is what made this look like a real dependency in the first place.

Leaving open for you to close after merge + verify.

— macbook-claude-session

## Diagnosis — your option 1 is correct **The tile service was never deployed / was decommissioned.** Verified read-only against the swarm: a scan of all **94 services matches nothing on `tile|map|terrain`**, and **no stack file for one exists in any repo**. The hostname completes TLS on the wildcard cert but **matches no reverse-proxy router**, so every path — including `/` — falls through to the proxy's catch-all 404 (a 19-byte `404 page not found`). That fully explains "answers, TLS fine, 404s everything". So the canary was probing a dependency that does not exist: **a health check that failed 100% of the time, forever**, producing a permanently Degraded rollup and the steady drip of error spans polluting error-rate monitoring. ## Decision: remove the canary, do not deploy a tile server **PR #274 (backend)** makes the tile health check **opt-in** (`Enabled`, default **false**) and registers it **only** when enabled *and* configured. With no tile server, the check is not registered at all — **the API stops probing entirely**: no canary, no 404 drip, no permanent Degraded. It also fixes a latent crash (an empty base URL threw `UriFormatException` during DI construction, which could have taken `/healthz` itself down). If a tile server is ever really deployed, flip it on — it stays Degraded-on-failure, so an optional dependency can never 503 the API (#484). **Why not just deploy one:** `tileserver-gl` serves tiles out of an `.mbtiles` data file, and **no terrain-DEM `.mbtiles` asset exists in any repo**. Deploying the image with no data would still 404 `/data/terrain/*` — it would fix nothing. **The blocker is a multi-GB data asset, not a missing compose file.** Worth its own ticket if the terrain feature is wanted back. ## Your two secondary points - **Frontend map/terrain feature: yes, it is silently broken — but only the enhancement.** The single consumer is the `map-tracker` **3D-terrain overlay** (`add3DTerrain()`, a raster-DEM source). **Base map layers come from OSM/ArcGIS and are unaffected** — maps still render, you just get no 3D terrain. Not fixed here; needs a product call (drop the layer, or fund the DEM asset). - **"The API doesn't implement /healthz" — this does not reproduce.** `/healthz` **is** implemented (`MapHealthChecks("/healthz")`) and returns **200** right now. Whatever was 404ing was hitting a different path or port; if it recurs, worth a separate ticket with the exact request line. Companion infrastructure PR corrects the service inventory, which wrongly listed `map-tile-server` as a live stack and told readers to commit its stack file — that doc is what made this look like a real dependency in the first place. Leaving open for you to close after merge + verify. — macbook-claude-session

spikerj referenced this issue

2026-07-14 06:23:15 +00:00

[Bug][Infra] API /healthz returns 503 — TileServerGL check failing (tiles.spikersoft.com 404s, no tile-server deployed) #484

spikerj commented

2026-07-22 02:57:52 +00:00

Board-sweep verified complete: TileServerGL healthcheck no longer probes when unconfigured and honors Degraded without 503ing (backend #274, tree-verified; docs retired in infra #77). Note: appsettings still defaults BaseUrl=tiles.spikersoft.com with no Production override — residual Degraded NOISE belongs to #484, not this ticket. Closing.

spikerj closed this issue

2026-07-22 02:57:52 +00:00

spikerj referenced this issue

2026-07-23 19:40:27 +00:00

e2e route-coverage red on master: location-fund routes unclassified + ratchet dip (regression from #552) #819

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#556