spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][Registry] Gitea upstream-image mirror (skopeo → mirror/* org) + incremental stack repointing #521

New Issue
Open
opened 2026-07-13 00:40:20 +00:00 by spikerj · 5 comments
spikerj commented 2026-07-13 00:40:20 +00:00
Owner
Copy Link
Copy Source

Goal

Every upstream image the fleet pulls (Docker Hub, quay, docker.elastic.co, lscr.io) gets a cached copy in the Gitea registry, so if an upstream tag vanishes or rate-limits we still have it. Then repoint stacks to the mirror incrementally so the cache is actually exercised.

Design

Gitea's container registry has no pull-through-proxy mode, so this is a scheduled push-mirror:

  • spikersoft-infrastructure/docker/upstream-images.txt — one source-ref mirror-name:tag per line, generated from the stack-file image inventory (~30 refs: mongo:8.2.6, redis:8@sha256:... (digest-pinned), postgres:14/15/16, traefik:v3.7.7, quay.io/keycloak/keycloak:26.7, minio/minio:RELEASE.2025-09-07...-cpuv1, gitea/act_runner:nightly, gitea/gitea:1.26, jaegertracing/jaeger:2.19.0, docker.elastic.co elasticsearch:7.17.0, datalust/seq, influxdb:2.7.12-alpine, redis:6/6.0-alpine, mariadb:10.5, nginx variants, atmoz/sftp, technitium, osixia/, lscr.io/linuxserver/calibre-web, clamav/clamav, prom/prometheus, grafana-enterprise, redislabs/redisinsight, heidiks/rabbitmq-delayed-message-exchange, mailserver2/, opny/peer-server, ...).
  • .gitea/workflows/mirror-upstream-images.yml — weekly schedule + workflow_dispatch; loops skopeo copy --all docker://<src> docker://git.spikersoft.com/mirror/<name>:<tag>.
  • skopeo, NOT docker pull/tag/push — docker flattens multi-arch manifest lists to the runner's arch, which would break the arm64 Jetson stacks; skopeo copy --all preserves manifest lists byte-for-byte (digest-stable, which matters for the digest-pinned redis:8 ref).
  • One-time: create Gitea org mirror (namespace isolation + its own cleanup policy).
  • Staleness: weekly refresh only affects :latest/nightly refs; everything else is tag/digest-pinned. Budget ~15-25 GB additional in the gitea-packages MinIO bucket.

Repointing (phase 2 of this ticket, incremental)

Critical tier first — mongo, redis, traefik, keycloak, postgres stacks flip image: to git.spikersoft.com/mirror/...; remaining stacks as they're next touched. Verify per repoint: skopeo inspect mirror digest == upstream digest; redeployed stack pulls on all arches (Jetson gets arm64).

Related: #413 (gitea-packages on MinIO, #498), #514 (runner disk — skopeo copies stream, no local docker storage needed).

## Goal Every upstream image the fleet pulls (Docker Hub, quay, docker.elastic.co, lscr.io) gets a cached copy in the Gitea registry, so if an upstream tag vanishes or rate-limits we still have it. Then repoint stacks to the mirror **incrementally** so the cache is actually exercised. ## Design Gitea's container registry has **no pull-through-proxy mode**, so this is a scheduled push-mirror: - `spikersoft-infrastructure/docker/upstream-images.txt` — one `source-ref mirror-name:tag` per line, generated from the stack-file image inventory (~30 refs: mongo:8.2.6, redis:8@sha256:... (digest-pinned), postgres:14/15/16, traefik:v3.7.7, quay.io/keycloak/keycloak:26.7, minio/minio:RELEASE.2025-09-07...-cpuv1, gitea/act_runner:nightly, gitea/gitea:1.26, jaegertracing/jaeger:2.19.0, docker.elastic.co elasticsearch:7.17.0, datalust/seq, influxdb:2.7.12-alpine, redis:6/6.0-alpine, mariadb:10.5, nginx variants, atmoz/sftp, technitium, osixia/*, lscr.io/linuxserver/calibre-web, clamav/clamav, prom/prometheus, grafana-enterprise, redislabs/redisinsight, heidiks/rabbitmq-delayed-message-exchange, mailserver2/*, opny/peer-server, ...). - `.gitea/workflows/mirror-upstream-images.yml` — weekly `schedule` + `workflow_dispatch`; loops `skopeo copy --all docker://<src> docker://git.spikersoft.com/mirror/<name>:<tag>`. - **skopeo, NOT docker pull/tag/push** — docker flattens multi-arch manifest lists to the runner's arch, which would break the arm64 Jetson stacks; `skopeo copy --all` preserves manifest lists byte-for-byte (digest-stable, which matters for the digest-pinned redis:8 ref). - One-time: create Gitea org `mirror` (namespace isolation + its own cleanup policy). - Staleness: weekly refresh only affects `:latest`/`nightly` refs; everything else is tag/digest-pinned. Budget ~15-25 GB additional in the `gitea-packages` MinIO bucket. ## Repointing (phase 2 of this ticket, incremental) Critical tier first — mongo, redis, traefik, keycloak, postgres stacks flip `image:` to `git.spikersoft.com/mirror/...`; remaining stacks as they're next touched. Verify per repoint: `skopeo inspect` mirror digest == upstream digest; redeployed stack pulls on all arches (Jetson gets arm64). Related: #413 (gitea-packages on MinIO, #498), #514 (runner disk — skopeo copies stream, no local docker storage needed).
spikerj commented 2026-07-13 19:18:01 +00:00
Author
Owner
Copy Link
Copy Source

QA watch 2026-07-13 ~19:18Z — the mirror is EMPTY and all three runs so far failed as outage collateral. MirrorUpstreamImages runs 10776/10779/10780 (02:10, 03:07, 04:00Z — manual dispatches; the weekly cron is Mondays 04:00 and hasn't fired yet) all failed squarely inside the MinIO/registry outage window (minio clean-exited 02:31Z and the Gitea registry is S3-backed by it → every push would have died; laptop runner disk trouble overlapped too). Nobody has re-dispatched since recovery: the mirror org currently holds 0 of the 67 refs in docker/upstream-images.txt, so the fleet still has zero protection from an upstream tag vanishing/rate-limiting.

Recommend: re-dispatch MirrorUpstreamImages now that minio + registry + runners are healthy (it's mostly network-bound; ~67 multi-arch skopeo copies will take a while on the laptop runner but shouldn't strain disk since skopeo streams). If it fails again on healthy infra, THAT's a real workflow bug worth debugging — today's failures aren't. First cron fire is tomorrow (Mon) 04:00Z; would be good to have one green manual run before trusting it.

QA watch 2026-07-13 ~19:18Z — the mirror is EMPTY and all three runs so far failed as outage collateral. MirrorUpstreamImages runs 10776/10779/10780 (02:10, 03:07, 04:00Z — manual dispatches; the weekly cron is Mondays 04:00 and hasn't fired yet) all failed squarely inside the MinIO/registry outage window (minio clean-exited 02:31Z and the Gitea registry is S3-backed by it → every push would have died; laptop runner disk trouble overlapped too). Nobody has re-dispatched since recovery: the mirror org currently holds 0 of the 67 refs in docker/upstream-images.txt, so the fleet still has zero protection from an upstream tag vanishing/rate-limiting. Recommend: re-dispatch MirrorUpstreamImages now that minio + registry + runners are healthy (it's mostly network-bound; ~67 multi-arch skopeo copies will take a while on the laptop runner but shouldn't strain disk since skopeo streams). If it fails again on healthy infra, THAT's a real workflow bug worth debugging — today's failures aren't. First cron fire is tomorrow (Mon) 04:00Z; would be good to have one green manual run before trusting it.
spikerj commented 2026-07-13 19:33:02 +00:00
Author
Owner
Copy Link
Copy Source

Pre-flight hardening PR: spikersoft-infrastructure#59 — --preserve-digests on every copy + optional authenticated Hub pulls (32/37 refs are Hub; an anonymous multi-arch sweep is ~100+ manifest requests, right at the anonymous 100/6h/IP ceiling — a real healthy-infra failure mode for the next dispatch). Ref list validated clean (37 active refs — the earlier '67' counted comment lines; no dupes/CRLF). Recommend: merge #59, optionally set DOCKERHUB_USER/TOKEN secrets, then re-dispatch AFTER the in-flight artpipe pushes finish (sweep pushes ~15-25GB into the same MinIO-backed registry), before Monday's 04:00Z cron.

Pre-flight hardening PR: spikersoft-infrastructure#59 — --preserve-digests on every copy + optional authenticated Hub pulls (32/37 refs are Hub; an anonymous multi-arch sweep is ~100+ manifest requests, right at the anonymous 100/6h/IP ceiling — a real healthy-infra failure mode for the next dispatch). Ref list validated clean (37 active refs — the earlier '67' counted comment lines; no dupes/CRLF). Recommend: merge #59, optionally set DOCKERHUB_USER/TOKEN secrets, then re-dispatch AFTER the in-flight artpipe pushes finish (sweep pushes ~15-25GB into the same MinIO-backed registry), before Monday's 04:00Z cron.
spikerj commented 2026-07-13 20:33:05 +00:00
Author
Owner
Copy Link
Copy Source

Hardening MERGED (infra #59 — --preserve-digests + optional authed Hub pulls). Mirror is still EMPTY: re-dispatch MirrorUpstreamImages once the artpipe pushes finish (optionally set DOCKERHUB_USER/TOKEN secrets first), ideally before the Monday 04:00Z cron's first blind fire.

Hardening MERGED (infra #59 — --preserve-digests + optional authed Hub pulls). Mirror is still EMPTY: re-dispatch MirrorUpstreamImages once the artpipe pushes finish (optionally set DOCKERHUB_USER/TOKEN secrets first), ideally before the Monday 04:00Z cron's first blind fire.
spikerj commented 2026-07-22 02:58:16 +00:00
Author
Owner
Copy Link
Copy Source

Board-sweep status (2026-07-22): mirror workflow + hardening merged (infra #51/#59) but the mirror is still EMPTY (needs re-dispatch), and the incremental stack-repointing half has zero commits.

Board-sweep status (2026-07-22): mirror workflow + hardening merged (infra #51/#59) but the mirror is still EMPTY (needs re-dispatch), and the incremental stack-repointing half has zero commits.
spikerj commented 2026-07-29 15:57:45 +00:00
Author
Owner
Copy Link
Copy Source

Audited against origin/master — the workflow shipped, but the mirror is empty and zero stacks were repointed. Staying open. Concrete numbers so the remaining work is measurable:

Built: .gitea/workflows/mirror-upstream-images.yml plus the ref list at docker/upstream-images.txt — 37 active refs (non-comment, non-blank).

Mirrored: 0 of 37. A package listing on owner mirror returns [] — the org is completely empty, unchanged since the 2026-07-22 comment. The workflow has never completed successfully into the registry.

Repointed: 0 of 37. git grep -n "git.spikersoft.com/mirror/" origin/master -- '*.yml' across all 81 infra stack files returns exactly one hit, and it's the SRC=/DEST= example line inside the mirror workflow itself (.gitea/workflows/mirror-upstream-images.yml:93) — not a stack. No docker-stack*.yml uses a mirror image anywhere. The critical tier this ticket exists to protect — mongo, redis, traefik, keycloak, postgres — is entirely untouched.

So this ticket is at 0% of its actual goal despite the tooling being merged, and "incremental by design" currently means zero increments rather than partial progress.

Remaining:

  1. Re-dispatch MirrorUpstreamImages until it lands green (optionally setting DOCKERHUB_USER / DOCKERHUB_TOKEN first, if rate limiting is what's failing it).
  2. All 37 repoints, starting with the critical tier.

Worth weighing against the current risk picture: #775 (registry blob corruption) and #700 (large registry push restarts single-node MinIO → Gitea crash-loop) both suggest the registry itself is not yet a reliable place to depend on for critical-tier images. Mirroring mongo/postgres/keycloak into a registry that has been observed truncating blobs could convert an upstream-availability risk into a local-corruption risk. Probably worth sequencing #775 ahead of the critical-tier repoints, even though the mirror job itself can proceed now.

Audited against `origin/master` — **the workflow shipped, but the mirror is empty and zero stacks were repointed.** Staying open. Concrete numbers so the remaining work is measurable: **Built:** `.gitea/workflows/mirror-upstream-images.yml` plus the ref list at `docker/upstream-images.txt` — **37 active refs** (non-comment, non-blank). **Mirrored: 0 of 37.** A package listing on owner `mirror` returns `[]` — the org is completely empty, unchanged since the 2026-07-22 comment. The workflow has never completed successfully into the registry. **Repointed: 0 of 37.** `git grep -n "git.spikersoft.com/mirror/" origin/master -- '*.yml'` across all 81 infra stack files returns exactly **one** hit, and it's the `SRC=`/`DEST=` example line inside the mirror workflow itself (`.gitea/workflows/mirror-upstream-images.yml:93`) — not a stack. No `docker-stack*.yml` uses a mirror image anywhere. The critical tier this ticket exists to protect — mongo, redis, traefik, keycloak, postgres — is entirely untouched. So this ticket is at 0% of its actual goal despite the tooling being merged, and "incremental by design" currently means zero increments rather than partial progress. **Remaining:** 1. Re-dispatch MirrorUpstreamImages until it lands green (optionally setting `DOCKERHUB_USER` / `DOCKERHUB_TOKEN` first, if rate limiting is what's failing it). 2. All 37 repoints, starting with the critical tier. Worth weighing against the current risk picture: **#775** (registry blob corruption) and **#700** (large registry push restarts single-node MinIO → Gitea crash-loop) both suggest the registry itself is not yet a reliable place to depend on for critical-tier images. Mirroring mongo/postgres/keycloak into a registry that has been observed truncating blobs could convert an upstream-availability risk into a local-corruption risk. Probably worth sequencing #775 ahead of the critical-tier repoints, even though the mirror job itself can proceed now.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#521