spikerj/spikersoft-issues
1
0 Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][Network] LAN traffic to *.spikersoft.com NAT-hairpins through the UDM — local DNS records + direct Gitea→MinIO path #538

New Issue
Closed
opened 2026-07-13 05:27:37 +00:00 by spikerj · 6 comments
spikerj commented 2026-07-13 05:27:37 +00:00
Owner
Copy link
Copy source

Problem (observed 2026-07-13 during the first big #516 image pushes)

All LAN hosts resolve *.spikersoft.com via public DNS (GoDaddy) to the WAN IP 204.197.150.99 — SERVER is even pointed at ISP resolvers directly. Every internal service call (registry pushes/pulls, S3, API, Keycloak) therefore NAT-hairpins through the UniFi Dream Machine, and the ingress architecture amplifies it: one ~20 GB image push = 4090 → Traefik (hairpin 1) → Gitea → MinIO via minio.spikersoft.com:443 through Traefik again (hairpin 2) ≈ ~80 GB of gateway traffic for 20 GB of payload, all counted as internet up/down on the UniFi dashboard. Verified: dig git.spikersoft.com / getent hosts minio.spikersoft.com return the WAN IP on Mac and SERVER; UniFi Traffic Flows for the 4090 show the top destination as our own WAN IP.

This taxes every future push (#515 phases, #521 mirror syncs) and every S3-migrated service (#413).

Fix — three tiers

1. Quick win: UDM local DNS records (no new infra).
In UniFi Network → Settings → Routing → DNS (Local DNS records), point the hot names at a stable, always-on swarm node's LAN IP — the swarm routing mesh answers :443 on every node, so any node works; pick one that is NOT SERVER (crashy, #503) and NOT the 4090 (part-time), e.g. dreamstream1 192.168.0.105:

  • git.spikersoft.com → 192.168.0.105
  • minio.spikersoft.com → 192.168.0.105
    (+ ids.spikersoft.com, api.spikersoft.com etc. as desired; add a wildcard *.spikersoft.com record if the Network app version accepts one). TLS keeps working — same hostname, same Let's Encrypt cert. Requires LAN clients to use the UDM as resolver (DHCP default) — note SERVER currently has ISP resolvers hardcoded (204.197.159.2/144.2, netplan/systemd-resolved) and needs repointing at the UDM.

2. Kill the double-traversal: Gitea → MinIO direct.
Gitea's package storage endpoint is minio.spikersoft.com:443 (through Traefik). Change to a direct path so blobs stop transiting the ingress twice: either join the gitea service to an overlay shared with minio and use the service name, or publish minio :9000 (host mode on SERVER) and point Gitea at the LAN IP. This helps even after DNS is fixed — it removes a full LAN round-trip per blob. (Predicted in docs/minio-storage-migration.md: "a direct overlay hop may beat hairpinning S3 traffic through the ingress.")

3. Longer term (optional): promote the Technitium experiment.
The dns-server stack (Technitium on laptop-server) already exists but was never adopted. Making it the LAN resolver (UDM DHCP hands it out) gives proper split-horizon with a real wildcard zone for spikersoft.com + local zones for the fleet — cleaner than maintaining per-name records on the UDM. Decide after tier 1 proves the win.

Verify

  • dig git.spikersoft.com from a LAN client returns the LAN IP.
  • Repeat a big image push; UniFi gateway internet counters stay flat aside from genuine upstream downloads (PyPI/HF); push wall-clock should also improve (switch-speed instead of gateway NAT path).

Related: #515/#516 (push volumes), #521 (mirror syncs), #413 (S3 traffic growth), #503 (why not to pin DNS at SERVER).

## Problem (observed 2026-07-13 during the first big #516 image pushes) All LAN hosts resolve `*.spikersoft.com` via public DNS (GoDaddy) to the WAN IP `204.197.150.99` — SERVER is even pointed at ISP resolvers directly. Every internal service call (registry pushes/pulls, S3, API, Keycloak) therefore **NAT-hairpins through the UniFi Dream Machine**, and the ingress architecture amplifies it: one ~20 GB image push = 4090 → Traefik (hairpin 1) → Gitea → MinIO via `minio.spikersoft.com:443` through Traefik again (hairpin 2) ≈ **~80 GB of gateway traffic for 20 GB of payload**, all counted as internet up/down on the UniFi dashboard. Verified: `dig git.spikersoft.com` / `getent hosts minio.spikersoft.com` return the WAN IP on Mac and SERVER; UniFi Traffic Flows for the 4090 show the top destination as our own WAN IP. This taxes every future push (#515 phases, #521 mirror syncs) and every S3-migrated service (#413). ## Fix — three tiers **1. Quick win: UDM local DNS records (no new infra).** In UniFi Network → Settings → Routing → DNS (Local DNS records), point the hot names at a stable, always-on swarm node's LAN IP — the swarm routing mesh answers :443 on every node, so any node works; pick one that is NOT SERVER (crashy, #503) and NOT the 4090 (part-time), e.g. dreamstream1 `192.168.0.105`: - `git.spikersoft.com` → 192.168.0.105 - `minio.spikersoft.com` → 192.168.0.105 (+ `ids.spikersoft.com`, `api.spikersoft.com` etc. as desired; add a wildcard `*.spikersoft.com` record if the Network app version accepts one). TLS keeps working — same hostname, same Let's Encrypt cert. Requires LAN clients to use the UDM as resolver (DHCP default) — note SERVER currently has ISP resolvers hardcoded (204.197.159.2/144.2, netplan/systemd-resolved) and needs repointing at the UDM. **2. Kill the double-traversal: Gitea → MinIO direct.** Gitea's package storage endpoint is `minio.spikersoft.com:443` (through Traefik). Change to a direct path so blobs stop transiting the ingress twice: either join the gitea service to an overlay shared with minio and use the service name, or publish minio :9000 (host mode on SERVER) and point Gitea at the LAN IP. This helps even after DNS is fixed — it removes a full LAN round-trip per blob. (Predicted in docs/minio-storage-migration.md: "a direct overlay hop may beat hairpinning S3 traffic through the ingress.") **3. Longer term (optional): promote the Technitium experiment.** The dns-server stack (Technitium on laptop-server) already exists but was never adopted. Making it the LAN resolver (UDM DHCP hands it out) gives proper split-horizon with a real wildcard zone for spikersoft.com + local zones for the fleet — cleaner than maintaining per-name records on the UDM. Decide after tier 1 proves the win. ## Verify - `dig git.spikersoft.com` from a LAN client returns the LAN IP. - Repeat a big image push; UniFi gateway internet counters stay flat aside from genuine upstream downloads (PyPI/HF); push wall-clock should also improve (switch-speed instead of gateway NAT path). Related: #515/#516 (push volumes), #521 (mirror syncs), #413 (S3 traffic growth), #503 (why not to pin DNS at SERVER).
spikerj commented 2026-07-14 02:31:59 +00:00
Author
Owner
Copy link
Copy source

QA Team — supporting datapoint 2026-07-14 ~02:32Z: from the swarm manager (LAN), https://git.spikersoft.com hard-refused connections (dial tcp 204.197.150.99:443: connect: connection refused) for roughly a minute, then recovered to 200 — while both gitea_gitea (up 22h) and traefik_traefik (up 43h on dreamstream3) never restarted. So the outage window existed only on the hairpin path through the UDM, exactly the fragility this ticket describes. (Coincidentally a 3-day-old zombie traefik task on dreamstream1 — rejected back then with No such image: traefik:v3.7.0 — was reaped at the same minute; possibly related if the UDM forwards to a node whose ingress hiccupped during reconciliation.) Local DNS records pointing *.spikersoft.com at the swarm directly would have made this blip invisible to LAN clients — CI/tea calls failed during the window.

**QA Team** — supporting datapoint 2026-07-14 ~02:32Z: from the swarm manager (LAN), `https://git.spikersoft.com` hard-refused connections (`dial tcp 204.197.150.99:443: connect: connection refused`) for roughly a minute, then recovered to 200 — while both `gitea_gitea` (up 22h) and `traefik_traefik` (up 43h on dreamstream3) never restarted. So the outage window existed only on the hairpin path through the UDM, exactly the fragility this ticket describes. (Coincidentally a 3-day-old zombie traefik task on dreamstream1 — rejected back then with `No such image: traefik:v3.7.0` — was reaped at the same minute; possibly related if the UDM forwards to a node whose ingress hiccupped during reconciliation.) Local DNS records pointing *.spikersoft.com at the swarm directly would have made this blip invisible to LAN clients — CI/tea calls failed during the window.
spikerj commented 2026-07-14 03:22:05 +00:00
Author
Owner
Copy link
Copy source

QA Team — stronger evidence 2026-07-14 ~03:30Z that this is more than LAN-hairpin fragility: during dreamstream1's crash/reboot window tonight (down ~02:27:50Z, boot 02:28:06Z — see #552), all four healthy CI runners logged failed to fetch task ... dial tcp 204.197.150.99:443: connect: connection refused from ~02:29:29Z until ~02:32Z (≈20 retries each), exactly matching the blip QA hit directly at 02:32Z.

connection refused (not timeout) means something at that IP answered with the port closed — consistent with the UDM's 443 forward pointing at dreamstream1's swarm ingress, which during early boot was up but not yet listening. Traefik itself never moved (running 43h+ on dreamstream3 throughout). If the forward target is really ds1, then every ds1 hiccup takes down the entire edge — public/WAN included, not just LAN — and ds1 is now a triple SPOF (edge ingress target + suspected SMB server for keycloak/Seq data + a node that already crashed once tonight).

Asks: confirm the UDM 443/80 port-forward target; if it's a single node IP, either point it at a healthier/stabler node, add the planned local-DNS split so LAN traffic skips the UDM entirely, or front with a VIP/keepalived across 2+ ingress nodes.

**QA Team** — stronger evidence 2026-07-14 ~03:30Z that this is more than LAN-hairpin fragility: during dreamstream1's crash/reboot window tonight (down ~02:27:50Z, boot 02:28:06Z — see #552), **all four healthy CI runners logged `failed to fetch task ... dial tcp 204.197.150.99:443: connect: connection refused`** from ~02:29:29Z until ~02:32Z (≈20 retries each), exactly matching the blip QA hit directly at 02:32Z. `connection refused` (not timeout) means something at that IP answered with the port closed — consistent with the **UDM's 443 forward pointing at dreamstream1's swarm ingress**, which during early boot was up but not yet listening. Traefik itself never moved (running 43h+ on dreamstream3 throughout). If the forward target is really ds1, then every ds1 hiccup takes down the entire edge — public/WAN included, not just LAN — and ds1 is now a triple SPOF (edge ingress target + suspected SMB server for keycloak/Seq data + a node that already crashed once tonight). **Asks:** confirm the UDM 443/80 port-forward target; if it's a single node IP, either point it at a healthier/stabler node, add the planned local-DNS split so LAN traffic skips the UDM entirely, or front with a VIP/keepalived across 2+ ingress nodes.
spikerj commented 2026-07-14 04:02:45 +00:00
Author
Owner
Copy link
Copy source

Confirmed live tonight (2026-07-14) during a large artpipe env-image push: the registry write path pays this tax twice, because since #498 Gitea does not store package blobs — it relays them to MinIO, so each push is a second hop that currently goes out through the gateway and back via the public name and ingress. A single multi-GB layer took over 40 minutes to upload; the gateway (not the LAN or the disks) is the throughput ceiling.

Fix for the hottest path is up as a PR in the private infrastructure repo (details there, not here): the registry-to-object-storage hop moves onto the shared overlay so it stays node-to-node on the LAN. Route changes, data does not.

The broader half of this ticket still stands and is the general fix: the swarm hosts do not use the gateway's local DNS records (they resolve our own names to the public edge), so EVERY service's object-storage traffic — not just registry pushes — takes the long way around. Fixing the hosts' resolvers fixes all of #413's traffic at once, and should happen before the next bulk operation (the image-mirror sync in #521 would be the worst offender — I am holding that dispatch until this is decided).

— macbook-claude-session

Confirmed live tonight (2026-07-14) during a large artpipe env-image push: the registry write path pays this tax twice, because since #498 Gitea does not store package blobs — it relays them to MinIO, so each push is a second hop that currently goes out through the gateway and back via the public name and ingress. A single multi-GB layer took over 40 minutes to upload; the gateway (not the LAN or the disks) is the throughput ceiling. Fix for the hottest path is up as a PR in the private infrastructure repo (details there, not here): the registry-to-object-storage hop moves onto the shared overlay so it stays node-to-node on the LAN. Route changes, data does not. The broader half of this ticket still stands and is the general fix: the swarm hosts do not use the gateway's local DNS records (they resolve our own names to the public edge), so EVERY service's object-storage traffic — not just registry pushes — takes the long way around. Fixing the hosts' resolvers fixes all of #413's traffic at once, and should happen before the next bulk operation (the image-mirror sync in #521 would be the worst offender — I am holding that dispatch until this is decided). — macbook-claude-session
spikerj commented 2026-07-22 02:58:16 +00:00
Author
Owner
Copy link
Copy source

Board-sweep status (2026-07-22): partial mitigations merged (Gitea→MinIO direct overlay path infra #70; CI pull resilience #276). REMAINING: the headline — LAN-local DNS for *.spikersoft.com on the UDM (ops); the hairpin tax is still live.

Board-sweep status (2026-07-22): partial mitigations merged (Gitea→MinIO direct overlay path infra #70; CI pull resilience #276). REMAINING: the headline — LAN-local DNS for *.spikersoft.com on the UDM (ops); the hairpin tax is still live.
spikerj commented 2026-07-29 16:01:02 +00:00
Author
Owner
Copy link
Copy source

Audited against origin/master — tier 2 is done and annotated; tier 1 (the headline fix) has no evidence of any kind. Staying open.

Tier 2 — Gitea→MinIO direct path: DONE, and pleasingly self-documenting:

  • gitea/docker-stack.yml:64 carries the explicit marker — # DIRECT OVERLAY PATH (#538): minio:9000, NOT minio.spikersoft.com:443.
  • :76 GITEA__storage_0X2E_packages__MINIO_ENDPOINT=minio:9000, :89 MINIO_USE_SSL=false; LFS storage likewise at :103.

Related and worth noting as reinforcement: ~20 service stacks now also set Storage__ServiceUrl=http://minio:9000 (see #696), so the hairpin is off the table for those paths too. The direct-overlay pattern has spread further than this ticket's own scope.

Tier 1 — UDM local DNS records for *.spikersoft.com, plus repointing SERVER's hardcoded ISP resolvers: NOT DONE / unrecorded. git grep -rln "hairpin\|Local DNS\|local-dns\|split-horizon" origin/master returns only stack files that reference the problem (gitea, minio, and 9 service stacks) — no runbook, no doc, no DNS config. Tier 3 (promoting the Technitium dns-server stack to LAN resolver) is likewise unrecorded.

Being straight about the evidence: this is a change on the UniFi controller, so git cannot prove it either way — but there's also no committed artifact recording it as done, and that's the gap worth closing regardless. Right now nothing in the repo would tell a future reader whether LAN DNS is handled.

What would settle it: dig git.spikersoft.com from a LAN host returning a LAN address (e.g. 192.168.0.105) rather than 204.197.150.99 — or, better, a committed runbook entry documenting the UDM records so the next person doesn't have to re-derive this.

Tier 2 delivering most of the practical benefit is probably why tier 1 stalled; if that's a deliberate call, worth saying so on the ticket and narrowing the scope rather than leaving it looking untouched.

Audited against `origin/master` — **tier 2 is done and annotated; tier 1 (the headline fix) has no evidence of any kind.** Staying open. **Tier 2 — Gitea→MinIO direct path: DONE**, and pleasingly self-documenting: - `gitea/docker-stack.yml:64` carries the explicit marker — `# DIRECT OVERLAY PATH (#538): minio:9000, NOT minio.spikersoft.com:443.` - `:76` `GITEA__storage_0X2E_packages__MINIO_ENDPOINT=minio:9000`, `:89` `MINIO_USE_SSL=false`; LFS storage likewise at `:103`. Related and worth noting as reinforcement: ~20 service stacks now also set `Storage__ServiceUrl=http://minio:9000` (see #696), so the hairpin is off the table for those paths too. The direct-overlay pattern has spread further than this ticket's own scope. **Tier 1 — UDM local DNS records for `*.spikersoft.com`, plus repointing SERVER's hardcoded ISP resolvers: NOT DONE / unrecorded.** `git grep -rln "hairpin\|Local DNS\|local-dns\|split-horizon" origin/master` returns only stack files that *reference* the problem (gitea, minio, and 9 service stacks) — no runbook, no doc, no DNS config. Tier 3 (promoting the Technitium `dns-server` stack to LAN resolver) is likewise unrecorded. Being straight about the evidence: this is a change on the UniFi controller, so **git cannot prove it either way** — but there's also no committed artifact recording it as done, and that's the gap worth closing regardless. Right now nothing in the repo would tell a future reader whether LAN DNS is handled. **What would settle it:** `dig git.spikersoft.com` from a LAN host returning a LAN address (e.g. 192.168.0.105) rather than 204.197.150.99 — or, better, a committed runbook entry documenting the UDM records so the next person doesn't have to re-derive this. Tier 2 delivering most of the practical benefit is probably why tier 1 stalled; if that's a deliberate call, worth saying so on the ticket and narrowing the scope rather than leaving it looking untouched.
spikerj commented 2026-08-07 13:48:27 +00:00
Author
Owner
Copy link
Copy source

Migrated to spikerj/spikersoft-infrastructure#175 as part of the umbrella-tracker breakup.

Verified 2026-08-07 — Code: spikersoft-infrastructure@86d03ff — tier 2 is done and self-documenting: gitea/docker-stack.yml:64 carries # DIRECT OVERLAY PATH (#538): minio:9000, NOT minio.spikersoft.com:443, with :76 MINIO_ENDPOINT=minio:9000, :89 MINIO_USE_SSL=false, and LFS likewise at :103. ~20 service stacks also set Storage__ServiceUrl=http://minio:9000. Tier 1 has no committed artifact — no runbook, no DNS config, nothing recording the UDM records. Live: Tier 1 is in fact largely DONE on the UDM, which git could not show. dig +short git.spikersoft.com @192.168.0.1 → traefik.spikersoft.com. 192.168.0.105; same for minio, api, learn, ids, seq — all LAN. But the hosts do not use it: resolvectl status on the 4090 reports Current DNS Server: 204.197.144.2, DNS Servers: 204.197.159.2 204.197.144.2 — the hardcoded ISP resolvers this ticket called out. Consequently getent hosts minio.spikersoft.com still returns the WAN IP 204.197.150.99, so LAN traffic still hairpins. (git.spikersoft.com only resolves LAN-side here because of a local /etc/hosts line.) bao.spikersoft.com also still has no UDM record → 204.197.150.99.
Status: partially done — tier 2 (Gitea→MinIO direct overlay) shipped; the UDM local-DNS records exist but the hosts still resolve via ISP nameservers, so the hairpin is still live

Closing here. Work now lives in the repo that holds the fix, so fixes #<N> in a PR will
auto-close it on merge. The umbrella tracker keeps cross-repo epics only.

— Opus 5 Agent

Migrated to **spikerj/spikersoft-infrastructure#175** as part of the umbrella-tracker breakup. Verified 2026-08-07 — **Code:** `spikersoft-infrastructure@86d03ff` — **tier 2 is done and self-documenting**: `gitea/docker-stack.yml:64` carries `# DIRECT OVERLAY PATH (#538): minio:9000, NOT minio.spikersoft.com:443`, with `:76` `MINIO_ENDPOINT=minio:9000`, `:89` `MINIO_USE_SSL=false`, and LFS likewise at `:103`. ~20 service stacks also set `Storage__ServiceUrl=http://minio:9000`. Tier 1 has **no committed artifact** — no runbook, no DNS config, nothing recording the UDM records. **Live:** **Tier 1 is in fact largely DONE on the UDM, which git could not show.** `dig +short git.spikersoft.com @192.168.0.1` → `traefik.spikersoft.com. 192.168.0.105`; same for `minio`, `api`, `learn`, `ids`, `seq` — all LAN. **But the hosts do not use it:** `resolvectl status` on the 4090 reports `Current DNS Server: 204.197.144.2`, `DNS Servers: 204.197.159.2 204.197.144.2` — the hardcoded ISP resolvers this ticket called out. Consequently `getent hosts minio.spikersoft.com` still returns the WAN IP `204.197.150.99`, so LAN traffic still hairpins. (`git.spikersoft.com` only resolves LAN-side here because of a local `/etc/hosts` line.) `bao.spikersoft.com` also still has no UDM record → `204.197.150.99`. Status: partially done — tier 2 (Gitea→MinIO direct overlay) shipped; the UDM local-DNS records exist but the hosts still resolve via ISP nameservers, so the hairpin is still live Closing here. Work now lives in the repo that holds the fix, so `fixes #<N>` in a PR will auto-close it on merge. The umbrella tracker keeps cross-repo epics only. — Opus 5 Agent
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 epic
Cross-repo epic — children live in per-repo trackers
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#538