spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Backend][MinIO] GeoLite2 → reference-data bucket; self-provisioning GeoIP (epic #413, resolves GeoIP half of #489) #530

New Issue
Open
opened 2026-07-13 00:40:42 +00:00 by spikerj · 5 comments
spikerj commented 2026-07-13 00:40:42 +00:00
Owner
Copy Link
Copy Source

Parent epic: #413. Also resolves the GeoIP half of #489.

Scope: /app/data/GeoLite2-City.mmdb. Reader: SpikerSoft.Business/Services/GeoIpService.cs:12. Writer: SpikerSoft.EventHandlers.Scheduler/TaskRunners/GeoIpUpdateTaskRunner.cs:41-58 (downloads from MaxMind). Postal-code data is not filesystem (seeded into Mongo) — the postal half of #489 is a seeding gap, not storage.

Plan: new reference-data bucket; GeoIpUpdateTaskRunner uploads the fresh mmdb to S3 after download; GeoIpService (and any service needing geo lookups) pulls from S3 into a local temp on startup / on a change poll (MaxMind reader needs a local file — small, so copy-on-start is fine). Kills the /app/data provisioning gap class entirely: fresh deploys self-provision from the bucket.

Requires a MaxMind license key as a secret (per #489).

Parent epic: #413. Also resolves the GeoIP half of #489. **Scope:** `/app/data/GeoLite2-City.mmdb`. Reader: `SpikerSoft.Business/Services/GeoIpService.cs:12`. Writer: `SpikerSoft.EventHandlers.Scheduler/TaskRunners/GeoIpUpdateTaskRunner.cs:41-58` (downloads from MaxMind). Postal-code data is **not** filesystem (seeded into Mongo) — the postal half of #489 is a seeding gap, not storage. **Plan:** new `reference-data` bucket; `GeoIpUpdateTaskRunner` uploads the fresh mmdb to S3 after download; `GeoIpService` (and any service needing geo lookups) pulls from S3 into a local temp on startup / on a change poll (MaxMind reader needs a local file — small, so copy-on-start is fine). Kills the `/app/data` provisioning gap class entirely: fresh deploys self-provision from the bucket. Requires a MaxMind license key as a secret (per #489).
spikerj commented 2026-07-13 19:00:47 +00:00
Author
Owner
Copy Link
Copy Source

PR spikersoft-backend#253 (feat/530-geolite2-reference-data, stacked on #252/#533): GeoIpUpdateTaskRunner uploads each fresh mmdb to the reference-data bucket; GeoIpService lazily self-provisions from the bucket into a local tmp cache (atomic rename, length-checked reuse) with legacy /app/data fallback. Resolves the GeoIP half of #489 once deployed. Infra follow-up needed: reference-data bucket + scoped keys (scheduler=write, api=read), Storage__UseS3 + creds on the api/scheduler stacks.

PR spikersoft-backend#253 (feat/530-geolite2-reference-data, stacked on #252/#533): GeoIpUpdateTaskRunner uploads each fresh mmdb to the reference-data bucket; GeoIpService lazily self-provisions from the bucket into a local tmp cache (atomic rename, length-checked reuse) with legacy /app/data fallback. Resolves the GeoIP half of #489 once deployed. Infra follow-up needed: reference-data bucket + scoped keys (scheduler=write, api=read), Storage__UseS3 + creds on the api/scheduler stacks.
spikerj commented 2026-07-13 19:08:50 +00:00
Author
Owner
Copy Link
Copy Source

Infra half: spikersoft-infrastructure PR #56 (GATED) — API stack Storage__ env + reference-data provisioning recipe in the migration doc. Backend PR #253 also gained the API_S3_SECRET_KEY deploy guard (scanner pattern). NOTE: the Scheduler service that hosts the bucket's writer is built/deployed NOWHERE (filed as its own ticket) — seed the bucket manually until it exists.

Infra half: spikersoft-infrastructure PR #56 (GATED) — API stack Storage__ env + reference-data provisioning recipe in the migration doc. Backend PR #253 also gained the API_S3_SECRET_KEY deploy guard (scanner pattern). NOTE: the Scheduler service that hosts the bucket's writer is built/deployed NOWHERE (filed as its own ticket) — seed the bucket manually until it exists.
spikerj commented 2026-07-14 22:49:15 +00:00
Author
Owner
Copy Link
Copy Source

QA verification — code complete, leaving OPEN pending runtime verification.

The S3 self-provisioning code is fully implemented on current master (HEAD bc55a9ff):

  • Reader: GeoIpService.ResolveDatabaseFile (GeoIpService.cs:155-189) pulls geoip/GeoLite2-City.mmdb from the reference-data bucket into a temp cache with atomic rename and length-match reuse; disk /app/data is only the fallback. Registered when Storage:UseS3 (ServiceCollectionExtensions.cs:915-917, 947).
  • Writer: GeoIpUpdateTaskRunner (TaskRunners/GeoIpUpdateTaskRunner.cs:144-160) uploads the fresh mmdb to reference-data after the MaxMind download; upload failure is a hard Fail (not best-effort). Registered in SpikerSoft.EventHandlers.Scheduler/Program.cs:39-42.
  • Landed in PR #253 (commit ab76037f, ancestor of master). Test-covered by GeoIpServiceS3ProvisioningTests (download-into-cache, cache reuse, object-absent fallback, bucket-failure non-throwing fallback).

NOT closing, deliberately: this depends on #540's Scheduler actually running with a valid MaxMind license key to populate the bucket — a runtime step not verifiable from a static checkout, and gated on the same open #540. Recommend closing once #540 is confirmed deployed and reference-data/geoip/GeoLite2-City.mmdb exists in MinIO.

**QA verification — code complete, leaving OPEN pending runtime verification.** The S3 self-provisioning code is fully implemented on current master (HEAD bc55a9ff): - Reader: `GeoIpService.ResolveDatabaseFile` (GeoIpService.cs:155-189) pulls `geoip/GeoLite2-City.mmdb` from the `reference-data` bucket into a temp cache with atomic rename and length-match reuse; disk `/app/data` is only the fallback. Registered when `Storage:UseS3` (ServiceCollectionExtensions.cs:915-917, 947). - Writer: `GeoIpUpdateTaskRunner` (TaskRunners/GeoIpUpdateTaskRunner.cs:144-160) uploads the fresh mmdb to `reference-data` after the MaxMind download; upload failure is a hard Fail (not best-effort). Registered in `SpikerSoft.EventHandlers.Scheduler/Program.cs:39-42`. - Landed in PR #253 (commit ab76037f, ancestor of master). Test-covered by `GeoIpServiceS3ProvisioningTests` (download-into-cache, cache reuse, object-absent fallback, bucket-failure non-throwing fallback). NOT closing, deliberately: this depends on #540's Scheduler actually running with a valid MaxMind license key to populate the bucket — a runtime step not verifiable from a static checkout, and gated on the same open #540. Recommend closing once #540 is confirmed deployed and `reference-data/geoip/GeoLite2-City.mmdb` exists in MinIO.
spikerj commented 2026-07-18 18:00:59 +00:00
Author
Owner
Copy Link
Copy Source

Epic #413 accuracy pass (2026-07-18) — status re-confirmed: still correctly OPEN, gated on #540.

Re-verified on current backend master: the QA assessment above still holds exactly — GeoIpService self-provisions from reference-data (with /app/data fallback) and GeoIpUpdateTaskRunner uploads post-download with hard-fail semantics; full test coverage present. Nothing further to do in code.

The gate is unchanged: #540 is still open — SpikerSoft.EventHandlers.Scheduler is built/deployed nowhere, so nothing can populate reference-data/geoip/GeoLite2-City.mmdb (bucket exists since the 2026-07-14 phase 0, empty). Close this when #540 lands and the object appears (or after a manual seed if that's preferred sooner). This is the only #413 child besides the cutover tickets that stays open, and its blocker is outside the epic.

**Epic #413 accuracy pass (2026-07-18) — status re-confirmed: still correctly OPEN, gated on #540.** Re-verified on current backend master: the QA assessment above still holds exactly — `GeoIpService` self-provisions from `reference-data` (with `/app/data` fallback) and `GeoIpUpdateTaskRunner` uploads post-download with hard-fail semantics; full test coverage present. Nothing further to do in code. The gate is unchanged: **#540 is still open** — `SpikerSoft.EventHandlers.Scheduler` is built/deployed nowhere, so nothing can populate `reference-data/geoip/GeoLite2-City.mmdb` (bucket exists since the 2026-07-14 phase 0, empty). Close this when #540 lands and the object appears (or after a manual seed if that's preferred sooner). This is the only #413 child besides the cutover tickets that stays open, and its blocker is outside the epic.
spikerj commented 2026-07-29 15:51:46 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — the code is complete and the #540 blocker is genuinely gone, but nothing shows the job has ever run. Staying open on that last step.

Code, confirmed:

  • SpikerSoft.Business/Services/GeoIpService.cs — S3 self-provisioning with an /app/data fallback (log at :179).
  • SpikerSoft.EventHandlers.Scheduler/TaskRunners/GeoIpUpdateTaskRunner.cs — [SchedulableTask("GeoIpUpdate"…)], downloading then uploading to the bucket; DI-registered at Program.cs:74.

The blocker is resolved. #540 (Scheduler built and deployed nowhere) is now closed — verified: .gitea/workflows/spikersoft-scheduler.yml builds image spikersoft-scheduler with deploy_secret_bao_path: secret/ci/backend/minio/scheduler, and spikersoft-scheduler/docker-stack.yml exists (:41 image, :47 Storage__UseS3=true, :52 internal minio URL, :53-54 scheduler-svc creds). The MinIO policy is scripted too: openbao/provision-minio-svc-users.sh:19 grants scheduler-svc rw on reference-data. So the whole path is now wired end to end.

What's still unproven, and why git can't settle it: the MaxMind license key is a per-task payload field (GeoIpUpdateTaskRunner.cs:43, payload schema at :13), not repo configuration. And there is no seed or registration of a GeoIpUpdate scheduled task anywhere on master — git grep GeoIpUpdate outside the runner returns only README, tests and DI wiring. So nothing establishes that the job has ever been created with a valid key, nor that reference-data/geoip/GeoLite2-City.mmdb exists in the bucket.

Remaining: create the GeoIpUpdate scheduled task with a valid MaxMind license key, run it once, and confirm the object lands. Settling evidence: mc stat spiker/reference-data/geoip/GeoLite2-City.mmdb plus a successful task-run record. Worth doing now that the blocker is cleared — this was the GeoIP half of #489.

Audited against `origin/master` — **the code is complete and the #540 blocker is genuinely gone, but nothing shows the job has ever run.** Staying open on that last step. **Code, confirmed:** - `SpikerSoft.Business/Services/GeoIpService.cs` — S3 self-provisioning with an `/app/data` fallback (log at `:179`). - `SpikerSoft.EventHandlers.Scheduler/TaskRunners/GeoIpUpdateTaskRunner.cs` — `[SchedulableTask("GeoIpUpdate"…)]`, downloading then uploading to the bucket; DI-registered at `Program.cs:74`. **The blocker is resolved.** #540 (Scheduler built and deployed nowhere) is now closed — verified: `.gitea/workflows/spikersoft-scheduler.yml` builds image `spikersoft-scheduler` with `deploy_secret_bao_path: secret/ci/backend/minio/scheduler`, and `spikersoft-scheduler/docker-stack.yml` exists (`:41` image, `:47` `Storage__UseS3=true`, `:52` internal minio URL, `:53-54` `scheduler-svc` creds). The MinIO policy is scripted too: `openbao/provision-minio-svc-users.sh:19` grants `scheduler-svc` rw on `reference-data`. So the whole path is now wired end to end. **What's still unproven, and why git can't settle it:** the MaxMind license key is a **per-task payload field** (`GeoIpUpdateTaskRunner.cs:43`, payload schema at `:13`), not repo configuration. And there is **no seed or registration of a GeoIpUpdate scheduled task anywhere on master** — `git grep GeoIpUpdate` outside the runner returns only README, tests and DI wiring. So nothing establishes that the job has ever been created with a valid key, nor that `reference-data/geoip/GeoLite2-City.mmdb` exists in the bucket. **Remaining:** create the GeoIpUpdate scheduled task with a valid MaxMind license key, run it once, and confirm the object lands. Settling evidence: `mc stat spiker/reference-data/geoip/GeoLite2-City.mmdb` plus a successful task-run record. Worth doing now that the blocker is cleared — this was the GeoIP half of #489.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#530