spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Backend] Missing reference data on prod — GeoLite2-City.mmdb + postal-code dataset absent, features degraded #489

New Issue
Open
opened 2026-07-12 00:32:20 +00:00 by spikerj · 5 comments
spikerj commented 2026-07-12 00:32:20 +00:00
Owner
Copy Link
Copy Source

Summary

Two bundled reference-data files are missing on the running backend, each degrading a feature and showing up as Degraded/Warning health signals in Seq:

  1. Postal code data missing — DefaultHealthCheckService: "Postal code data is missing — state/province dropdowns will be ..." (Postal Code Data health check = Degraded).
  2. GeoIP database missing — GeoIpService: "GeoIP database not found at /app/data/GeoLite2-City.mmdb. Geolocation lookups will be unavailable."

Both are recurring (postal ~6, geoip ~4 in the recent window) — i.e. persistent provisioning gaps, not one-offs.

Impact

  • Postal: state/province dropdowns unavailable / degraded in address forms.
  • GeoIP: geolocation lookups silently unavailable (any geo-based feature/analytics returns nothing).
    Neither is an outage, but both silently degrade product features and keep /healthz in a Degraded sub-state.

Proposed fix

  • GeoIP: provision GeoLite2-City.mmdb at /app/data/ — bake into the ArtPipe/backend image or mount a volume, and (recommended) add a MaxMind license-key + geoipupdate sidecar/cron so it stays current. (MaxMind requires a free license key to download GeoLite2.)
  • Postal code data: provision the postal/state-province dataset the Postal Code Data health check looks for (confirm the expected path/format in the check) and bundle it the same way (image layer or mounted volume).
  • Consider promoting these from Degraded to a startup log if they're optional, or documenting them as required provisioning so fresh deploys don't silently lose the features.

Verify

curl -s https://api.spikersoft.com/healthz | jq '.checks[] | select(.name|test("Postal|Geo"))'
# both -> Healthy; Seq no longer shows the "missing" warnings

Priority: low-medium (degraded features, no outage).

Filed proactively by automated health check (Seq Warning audit).

## Summary Two bundled reference-data files are missing on the running backend, each degrading a feature and showing up as `Degraded`/`Warning` health signals in Seq: 1. **Postal code data missing** — `DefaultHealthCheckService`: *"Postal code data is missing — state/province dropdowns will be ..."* (Postal Code Data health check = Degraded). 2. **GeoIP database missing** — `GeoIpService`: *"GeoIP database not found at `/app/data/GeoLite2-City.mmdb`. Geolocation lookups will be unavailable."* Both are recurring (postal ~6, geoip ~4 in the recent window) — i.e. persistent provisioning gaps, not one-offs. ## Impact - Postal: state/province dropdowns unavailable / degraded in address forms. - GeoIP: geolocation lookups silently unavailable (any geo-based feature/analytics returns nothing). Neither is an outage, but both silently degrade product features and keep `/healthz` in a `Degraded` sub-state. ## Proposed fix - **GeoIP:** provision `GeoLite2-City.mmdb` at `/app/data/` — bake into the ArtPipe/backend image or mount a volume, and (recommended) add a MaxMind license-key + `geoipupdate` sidecar/cron so it stays current. (MaxMind requires a free license key to download GeoLite2.) - **Postal code data:** provision the postal/state-province dataset the `Postal Code Data` health check looks for (confirm the expected path/format in the check) and bundle it the same way (image layer or mounted volume). - Consider promoting these from `Degraded` to a startup log if they're optional, or documenting them as required provisioning so fresh deploys don't silently lose the features. ## Verify ``` curl -s https://api.spikersoft.com/healthz | jq '.checks[] | select(.name|test("Postal|Geo"))' # both -> Healthy; Seq no longer shows the "missing" warnings ``` Priority: low-medium (degraded features, no outage). --- _Filed proactively by automated health check (Seq Warning audit)._
spikerj commented 2026-07-12 00:44:54 +00:00
Author
Owner
Copy Link
Copy Source

Triage — mostly provisioning, and the GeoIP half already has machinery: SpikerSoft.EventHandlers.Scheduler/TaskRunners/GeoIpUpdateTaskRunner.cs downloads GeoLite2 from MaxMind given a LicenseKey payload — so the missing piece is (1) a free MaxMind license key (account-holder action) and (2) scheduling that task with the key + ensuring /app/data persists (it's container-local today, so the download evaporates on redeploy — needs a volume or the download must run on startup). Postal data: need to confirm the expected path/format from PostalData_HealthCheck and bundle it as an image layer (it's static reference data — baking beats mounting). Blocked on the MaxMind key; the rest is PR-able once that exists.

Triage — mostly provisioning, and the GeoIP half already has machinery: SpikerSoft.EventHandlers.Scheduler/TaskRunners/GeoIpUpdateTaskRunner.cs downloads GeoLite2 from MaxMind given a LicenseKey payload — so the missing piece is (1) a free MaxMind license key (account-holder action) and (2) scheduling that task with the key + ensuring /app/data persists (it's container-local today, so the download evaporates on redeploy — needs a volume or the download must run on startup). Postal data: need to confirm the expected path/format from PostalData_HealthCheck and bundle it as an image layer (it's static reference data — baking beats mounting). Blocked on the MaxMind key; the rest is PR-able once that exists.
spikerj commented 2026-07-12 20:56:10 +00:00
Author
Owner
Copy Link
Copy Source

Status split (2026-07-12, verified against prod /healthz):

  • Postal code data: RESOLVED — the health check now reports Healthy — Postal code data available: 317,784 records. Half of this ticket is done.
  • GeoLite2-City.mmdb: still missing. This one is blocked on a credential/licensing decision, not code: MaxMind requires a (free) account + license key to download GeoLite2, and automated refresh needs that key as a deploy secret. Options, in order of preference:
    1. MaxMind license key as a Gitea Actions secret (MAXMIND_LICENSE_KEY) + a download step in the API image build (or a volume-provisioning job) — canonical data, auto-refreshable. Needs you to create the account/key.
    2. db-ip.com "IP to City Lite" — no account needed (CC-BY 4.0, attribution required), same .mmdb format, slightly lower accuracy. I can wire this end-to-end without any credential.
    3. Drop the feature — remove the GeoIpService check if geolocation isn't product-relevant.

GeoIpService already degrades gracefully (warns, returns nothing), so there's no urgency beyond the standing Degraded signal. Name a lane and I'll build it.

Status split (2026-07-12, verified against prod `/healthz`): - **Postal code data: RESOLVED** — the health check now reports `Healthy — Postal code data available: 317,784 records`. Half of this ticket is done. - **GeoLite2-City.mmdb: still missing.** This one is blocked on a credential/licensing decision, not code: MaxMind requires a (free) account + license key to download GeoLite2, and automated refresh needs that key as a deploy secret. Options, in order of preference: 1. **MaxMind license key** as a Gitea Actions secret (`MAXMIND_LICENSE_KEY`) + a download step in the API image build (or a volume-provisioning job) — canonical data, auto-refreshable. Needs you to create the account/key. 2. **db-ip.com "IP to City Lite"** — no account needed (CC-BY 4.0, attribution required), same .mmdb format, slightly lower accuracy. I can wire this end-to-end without any credential. 3. **Drop the feature** — remove the GeoIpService check if geolocation isn't product-relevant. `GeoIpService` already degrades gracefully (warns, returns nothing), so there's no urgency beyond the standing Degraded signal. Name a lane and I'll build it.
spikerj commented 2026-07-13 19:00:48 +00:00
Author
Owner
Copy Link
Copy Source

GeoIP half: backend PR #253 (under #530) makes GeoIpService self-provision GeoLite2 from the MinIO reference-data bucket — fresh deploys stop needing /app/data. Postal half remains: it's a Mongo seeding gap, not storage (see #530 notes).

GeoIP half: backend PR #253 (under #530) makes GeoIpService self-provision GeoLite2 from the MinIO reference-data bucket — fresh deploys stop needing /app/data. Postal half remains: it's a Mongo seeding gap, not storage (see #530 notes).
spikerj commented 2026-07-13 19:26:22 +00:00
Author
Owner
Copy Link
Copy Source

Postal half appears RESOLVED in prod: /healthz (checked 2026-07-13 ~15:35Z) shows 'Postal Code Data' Healthy — only TileServerGL (#484) is non-Healthy. GeoIP half: fix chain is in review (backend #253 self-provisioning from MinIO reference-data + #254/infra #57 standing up the Scheduler that actually runs GeoIpUpdate — see #530/#540). Once those merge and the bucket is seeded, this ticket should be closeable after the /healthz + Seq verification in the description.

Postal half appears RESOLVED in prod: /healthz (checked 2026-07-13 ~15:35Z) shows 'Postal Code Data' Healthy — only TileServerGL (#484) is non-Healthy. GeoIP half: fix chain is in review (backend #253 self-provisioning from MinIO reference-data + #254/infra #57 standing up the Scheduler that actually runs GeoIpUpdate — see #530/#540). Once those merge and the bucket is seeded, this ticket should be closeable after the /healthz + Seq verification in the description.
spikerj commented 2026-07-22 02:58:16 +00:00
Author
Owner
Copy Link
Copy Source

Board-sweep status (2026-07-22): Postal half Healthy in prod (07-13); GeoIP half merged (self-provision #253 + scheduler stack). REMAINING: gated on #540's scheduler actually deploying + the reference-data bucket seed.

Board-sweep status (2026-07-22): Postal half Healthy in prod (07-13); GeoIP half merged (self-provision #253 + scheduler stack). REMAINING: gated on #540's scheduler actually deploying + the reference-data bucket seed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#489