spikerj/spikersoft-issues
1
0 Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][Registry] Gitea upstream-image mirror (skopeo → mirror/* org) + incremental stack repointing #521

New Issue
Closed
opened 2026-07-13 00:40:20 +00:00 by spikerj · 6 comments
spikerj commented 2026-07-13 00:40:20 +00:00
Owner
Copy link
Copy source

Goal

Every upstream image the fleet pulls (Docker Hub, quay, docker.elastic.co, lscr.io) gets a cached copy in the Gitea registry, so if an upstream tag vanishes or rate-limits we still have it. Then repoint stacks to the mirror incrementally so the cache is actually exercised.

Design

Gitea's container registry has no pull-through-proxy mode, so this is a scheduled push-mirror:

  • spikersoft-infrastructure/docker/upstream-images.txt — one source-ref mirror-name:tag per line, generated from the stack-file image inventory (~30 refs: mongo:8.2.6, redis:8@sha256:... (digest-pinned), postgres:14/15/16, traefik:v3.7.7, quay.io/keycloak/keycloak:26.7, minio/minio:RELEASE.2025-09-07...-cpuv1, gitea/act_runner:nightly, gitea/gitea:1.26, jaegertracing/jaeger:2.19.0, docker.elastic.co elasticsearch:7.17.0, datalust/seq, influxdb:2.7.12-alpine, redis:6/6.0-alpine, mariadb:10.5, nginx variants, atmoz/sftp, technitium, osixia/, lscr.io/linuxserver/calibre-web, clamav/clamav, prom/prometheus, grafana-enterprise, redislabs/redisinsight, heidiks/rabbitmq-delayed-message-exchange, mailserver2/, opny/peer-server, ...).
  • .gitea/workflows/mirror-upstream-images.yml — weekly schedule + workflow_dispatch; loops skopeo copy --all docker://<src> docker://git.spikersoft.com/mirror/<name>:<tag>.
  • skopeo, NOT docker pull/tag/push — docker flattens multi-arch manifest lists to the runner's arch, which would break the arm64 Jetson stacks; skopeo copy --all preserves manifest lists byte-for-byte (digest-stable, which matters for the digest-pinned redis:8 ref).
  • One-time: create Gitea org mirror (namespace isolation + its own cleanup policy).
  • Staleness: weekly refresh only affects :latest/nightly refs; everything else is tag/digest-pinned. Budget ~15-25 GB additional in the gitea-packages MinIO bucket.

Repointing (phase 2 of this ticket, incremental)

Critical tier first — mongo, redis, traefik, keycloak, postgres stacks flip image: to git.spikersoft.com/mirror/...; remaining stacks as they're next touched. Verify per repoint: skopeo inspect mirror digest == upstream digest; redeployed stack pulls on all arches (Jetson gets arm64).

Related: #413 (gitea-packages on MinIO, #498), #514 (runner disk — skopeo copies stream, no local docker storage needed).

## Goal Every upstream image the fleet pulls (Docker Hub, quay, docker.elastic.co, lscr.io) gets a cached copy in the Gitea registry, so if an upstream tag vanishes or rate-limits we still have it. Then repoint stacks to the mirror **incrementally** so the cache is actually exercised. ## Design Gitea's container registry has **no pull-through-proxy mode**, so this is a scheduled push-mirror: - `spikersoft-infrastructure/docker/upstream-images.txt` — one `source-ref mirror-name:tag` per line, generated from the stack-file image inventory (~30 refs: mongo:8.2.6, redis:8@sha256:... (digest-pinned), postgres:14/15/16, traefik:v3.7.7, quay.io/keycloak/keycloak:26.7, minio/minio:RELEASE.2025-09-07...-cpuv1, gitea/act_runner:nightly, gitea/gitea:1.26, jaegertracing/jaeger:2.19.0, docker.elastic.co elasticsearch:7.17.0, datalust/seq, influxdb:2.7.12-alpine, redis:6/6.0-alpine, mariadb:10.5, nginx variants, atmoz/sftp, technitium, osixia/*, lscr.io/linuxserver/calibre-web, clamav/clamav, prom/prometheus, grafana-enterprise, redislabs/redisinsight, heidiks/rabbitmq-delayed-message-exchange, mailserver2/*, opny/peer-server, ...). - `.gitea/workflows/mirror-upstream-images.yml` — weekly `schedule` + `workflow_dispatch`; loops `skopeo copy --all docker://<src> docker://git.spikersoft.com/mirror/<name>:<tag>`. - **skopeo, NOT docker pull/tag/push** — docker flattens multi-arch manifest lists to the runner's arch, which would break the arm64 Jetson stacks; `skopeo copy --all` preserves manifest lists byte-for-byte (digest-stable, which matters for the digest-pinned redis:8 ref). - One-time: create Gitea org `mirror` (namespace isolation + its own cleanup policy). - Staleness: weekly refresh only affects `:latest`/`nightly` refs; everything else is tag/digest-pinned. Budget ~15-25 GB additional in the `gitea-packages` MinIO bucket. ## Repointing (phase 2 of this ticket, incremental) Critical tier first — mongo, redis, traefik, keycloak, postgres stacks flip `image:` to `git.spikersoft.com/mirror/...`; remaining stacks as they're next touched. Verify per repoint: `skopeo inspect` mirror digest == upstream digest; redeployed stack pulls on all arches (Jetson gets arm64). Related: #413 (gitea-packages on MinIO, #498), #514 (runner disk — skopeo copies stream, no local docker storage needed).
spikerj commented 2026-07-13 19:18:01 +00:00
Author
Owner
Copy link
Copy source

QA watch 2026-07-13 ~19:18Z — the mirror is EMPTY and all three runs so far failed as outage collateral. MirrorUpstreamImages runs 10776/10779/10780 (02:10, 03:07, 04:00Z — manual dispatches; the weekly cron is Mondays 04:00 and hasn't fired yet) all failed squarely inside the MinIO/registry outage window (minio clean-exited 02:31Z and the Gitea registry is S3-backed by it → every push would have died; laptop runner disk trouble overlapped too). Nobody has re-dispatched since recovery: the mirror org currently holds 0 of the 67 refs in docker/upstream-images.txt, so the fleet still has zero protection from an upstream tag vanishing/rate-limiting.

Recommend: re-dispatch MirrorUpstreamImages now that minio + registry + runners are healthy (it's mostly network-bound; ~67 multi-arch skopeo copies will take a while on the laptop runner but shouldn't strain disk since skopeo streams). If it fails again on healthy infra, THAT's a real workflow bug worth debugging — today's failures aren't. First cron fire is tomorrow (Mon) 04:00Z; would be good to have one green manual run before trusting it.

QA watch 2026-07-13 ~19:18Z — the mirror is EMPTY and all three runs so far failed as outage collateral. MirrorUpstreamImages runs 10776/10779/10780 (02:10, 03:07, 04:00Z — manual dispatches; the weekly cron is Mondays 04:00 and hasn't fired yet) all failed squarely inside the MinIO/registry outage window (minio clean-exited 02:31Z and the Gitea registry is S3-backed by it → every push would have died; laptop runner disk trouble overlapped too). Nobody has re-dispatched since recovery: the mirror org currently holds 0 of the 67 refs in docker/upstream-images.txt, so the fleet still has zero protection from an upstream tag vanishing/rate-limiting. Recommend: re-dispatch MirrorUpstreamImages now that minio + registry + runners are healthy (it's mostly network-bound; ~67 multi-arch skopeo copies will take a while on the laptop runner but shouldn't strain disk since skopeo streams). If it fails again on healthy infra, THAT's a real workflow bug worth debugging — today's failures aren't. First cron fire is tomorrow (Mon) 04:00Z; would be good to have one green manual run before trusting it.
spikerj commented 2026-07-13 19:33:02 +00:00
Author
Owner
Copy link
Copy source

Pre-flight hardening PR: spikersoft-infrastructure#59 — --preserve-digests on every copy + optional authenticated Hub pulls (32/37 refs are Hub; an anonymous multi-arch sweep is ~100+ manifest requests, right at the anonymous 100/6h/IP ceiling — a real healthy-infra failure mode for the next dispatch). Ref list validated clean (37 active refs — the earlier '67' counted comment lines; no dupes/CRLF). Recommend: merge #59, optionally set DOCKERHUB_USER/TOKEN secrets, then re-dispatch AFTER the in-flight artpipe pushes finish (sweep pushes ~15-25GB into the same MinIO-backed registry), before Monday's 04:00Z cron.

Pre-flight hardening PR: spikersoft-infrastructure#59 — --preserve-digests on every copy + optional authenticated Hub pulls (32/37 refs are Hub; an anonymous multi-arch sweep is ~100+ manifest requests, right at the anonymous 100/6h/IP ceiling — a real healthy-infra failure mode for the next dispatch). Ref list validated clean (37 active refs — the earlier '67' counted comment lines; no dupes/CRLF). Recommend: merge #59, optionally set DOCKERHUB_USER/TOKEN secrets, then re-dispatch AFTER the in-flight artpipe pushes finish (sweep pushes ~15-25GB into the same MinIO-backed registry), before Monday's 04:00Z cron.
spikerj commented 2026-07-13 20:33:05 +00:00
Author
Owner
Copy link
Copy source

Hardening MERGED (infra #59 — --preserve-digests + optional authed Hub pulls). Mirror is still EMPTY: re-dispatch MirrorUpstreamImages once the artpipe pushes finish (optionally set DOCKERHUB_USER/TOKEN secrets first), ideally before the Monday 04:00Z cron's first blind fire.

Hardening MERGED (infra #59 — --preserve-digests + optional authed Hub pulls). Mirror is still EMPTY: re-dispatch MirrorUpstreamImages once the artpipe pushes finish (optionally set DOCKERHUB_USER/TOKEN secrets first), ideally before the Monday 04:00Z cron's first blind fire.
spikerj commented 2026-07-22 02:58:16 +00:00
Author
Owner
Copy link
Copy source

Board-sweep status (2026-07-22): mirror workflow + hardening merged (infra #51/#59) but the mirror is still EMPTY (needs re-dispatch), and the incremental stack-repointing half has zero commits.

Board-sweep status (2026-07-22): mirror workflow + hardening merged (infra #51/#59) but the mirror is still EMPTY (needs re-dispatch), and the incremental stack-repointing half has zero commits.
spikerj commented 2026-07-29 15:57:45 +00:00
Author
Owner
Copy link
Copy source

Audited against origin/master — the workflow shipped, but the mirror is empty and zero stacks were repointed. Staying open. Concrete numbers so the remaining work is measurable:

Built: .gitea/workflows/mirror-upstream-images.yml plus the ref list at docker/upstream-images.txt — 37 active refs (non-comment, non-blank).

Mirrored: 0 of 37. A package listing on owner mirror returns [] — the org is completely empty, unchanged since the 2026-07-22 comment. The workflow has never completed successfully into the registry.

Repointed: 0 of 37. git grep -n "git.spikersoft.com/mirror/" origin/master -- '*.yml' across all 81 infra stack files returns exactly one hit, and it's the SRC=/DEST= example line inside the mirror workflow itself (.gitea/workflows/mirror-upstream-images.yml:93) — not a stack. No docker-stack*.yml uses a mirror image anywhere. The critical tier this ticket exists to protect — mongo, redis, traefik, keycloak, postgres — is entirely untouched.

So this ticket is at 0% of its actual goal despite the tooling being merged, and "incremental by design" currently means zero increments rather than partial progress.

Remaining:

  1. Re-dispatch MirrorUpstreamImages until it lands green (optionally setting DOCKERHUB_USER / DOCKERHUB_TOKEN first, if rate limiting is what's failing it).
  2. All 37 repoints, starting with the critical tier.

Worth weighing against the current risk picture: #775 (registry blob corruption) and #700 (large registry push restarts single-node MinIO → Gitea crash-loop) both suggest the registry itself is not yet a reliable place to depend on for critical-tier images. Mirroring mongo/postgres/keycloak into a registry that has been observed truncating blobs could convert an upstream-availability risk into a local-corruption risk. Probably worth sequencing #775 ahead of the critical-tier repoints, even though the mirror job itself can proceed now.

Audited against `origin/master` — **the workflow shipped, but the mirror is empty and zero stacks were repointed.** Staying open. Concrete numbers so the remaining work is measurable: **Built:** `.gitea/workflows/mirror-upstream-images.yml` plus the ref list at `docker/upstream-images.txt` — **37 active refs** (non-comment, non-blank). **Mirrored: 0 of 37.** A package listing on owner `mirror` returns `[]` — the org is completely empty, unchanged since the 2026-07-22 comment. The workflow has never completed successfully into the registry. **Repointed: 0 of 37.** `git grep -n "git.spikersoft.com/mirror/" origin/master -- '*.yml'` across all 81 infra stack files returns exactly **one** hit, and it's the `SRC=`/`DEST=` example line inside the mirror workflow itself (`.gitea/workflows/mirror-upstream-images.yml:93`) — not a stack. No `docker-stack*.yml` uses a mirror image anywhere. The critical tier this ticket exists to protect — mongo, redis, traefik, keycloak, postgres — is entirely untouched. So this ticket is at 0% of its actual goal despite the tooling being merged, and "incremental by design" currently means zero increments rather than partial progress. **Remaining:** 1. Re-dispatch MirrorUpstreamImages until it lands green (optionally setting `DOCKERHUB_USER` / `DOCKERHUB_TOKEN` first, if rate limiting is what's failing it). 2. All 37 repoints, starting with the critical tier. Worth weighing against the current risk picture: **#775** (registry blob corruption) and **#700** (large registry push restarts single-node MinIO → Gitea crash-loop) both suggest the registry itself is not yet a reliable place to depend on for critical-tier images. Mirroring mongo/postgres/keycloak into a registry that has been observed truncating blobs could convert an upstream-availability risk into a local-corruption risk. Probably worth sequencing #775 ahead of the critical-tier repoints, even though the mirror job itself can proceed now.
spikerj commented 2026-08-07 13:51:20 +00:00
Author
Owner
Copy link
Copy source

Migrated to spikerj/spikersoft-infrastructure#183 as part of the umbrella-tracker breakup.

Verified 2026-08-07 — Code: spikersoft-infrastructure@86d03ff — phase 1 is merged: docker/upstream-images.txt and .gitea/workflows/mirror-upstream-images.yml both exist (the workflow builds skopeo copy invocations, e.g. :93 SRC="mongo:8.2.6"; DEST="git.spikersoft.com/mirror/mongo:8.2.6"). Phase 2 has not started: git grep "git.spikersoft.com/mirror" --include=*.yml matches only the mirror workflow itself — not one stack file has been repointed. Live: The mirror has never successfully synced. The Gitea org exists (GET /api/v1/orgs/mirror → id 5, visibility limited) but GET /api/v1/packages/mirror returns 0 packages. And docker service ls confirms every upstream image is still pulled from its origin: mongo:8.2.6, redis:8, postgres:15/16, traefik:v3.7.8, quay.io/keycloak/keycloak:26.7, minio/minio:RELEASE.2025-09-07…, gitea/gitea:1.27, gitea/runner:3.0.2, nginx:1.27-alpine, datalust/seq:latest, influxdb:2.7.12-alpine, mariadb:10.5, docker.elastic.co/…/elasticsearch:8.11.0.
Status: partially done — the image list, workflow and mirror org all exist; the sync has never populated a single package and no stack is repointed

Closing here. Work now lives in the repo that holds the fix, so fixes #<N> in a PR will
auto-close it on merge. The umbrella tracker keeps cross-repo epics only.

— Opus 5 Agent

Migrated to **spikerj/spikersoft-infrastructure#183** as part of the umbrella-tracker breakup. Verified 2026-08-07 — **Code:** `spikersoft-infrastructure@86d03ff` — **phase 1 is merged**: `docker/upstream-images.txt` and `.gitea/workflows/mirror-upstream-images.yml` both exist (the workflow builds `skopeo copy` invocations, e.g. `:93` `SRC="mongo:8.2.6"; DEST="git.spikersoft.com/mirror/mongo:8.2.6"`). **Phase 2 has not started**: `git grep "git.spikersoft.com/mirror" --include=*.yml` matches only the mirror workflow itself — not one stack file has been repointed. **Live:** **The mirror has never successfully synced.** The Gitea org exists (`GET /api/v1/orgs/mirror` → `id 5`, visibility `limited`) but `GET /api/v1/packages/mirror` returns **0 packages**. And `docker service ls` confirms every upstream image is still pulled from its origin: `mongo:8.2.6`, `redis:8`, `postgres:15/16`, `traefik:v3.7.8`, `quay.io/keycloak/keycloak:26.7`, `minio/minio:RELEASE.2025-09-07…`, `gitea/gitea:1.27`, `gitea/runner:3.0.2`, `nginx:1.27-alpine`, `datalust/seq:latest`, `influxdb:2.7.12-alpine`, `mariadb:10.5`, `docker.elastic.co/…/elasticsearch:8.11.0`. Status: partially done — the image list, workflow and `mirror` org all exist; the sync has never populated a single package and no stack is repointed Closing here. Work now lives in the repo that holds the fix, so `fixes #<N>` in a PR will auto-close it on merge. The umbrella tracker keeps cross-repo epics only. — Opus 5 Agent
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 epic
Cross-repo epic — children live in per-repo trackers
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#521