CI: prune artpipe container pin tags — keep :latest + 3, they accumulate forever #850

New Issue

2026-07-25T22:18:44Z

spikerj commented

2026-07-25 22:18:44 +00:00

Problem

Both artpipe image workflows push a unique, immutable pin tag on every run and nothing ever removes them, so the Gitea package list grows without bound:

spikersoft-backend/.gitea/workflows/spikersoft-artpipe-processor.yml — tier 3, pushes :latest + :<backendSHA7>-<artpipeSHA7>
spikersoft-artpipe/.gitea/workflows/model-env-images.yml — tier 2, pushes :latest + :<artpipeSHA7>

By 2026-07-25 artpipe-model-safetycheck alone had 32 pin versions; prodstages 22, photostack 21. Instance-wide there were 334 container versions, making the package list unusable.

Nothing consumes a pin tag declaratively — every spikersoft-infrastructure/spikersoft-artpipe-*/docker-stack*.yml pins :latest. Pins exist only as a manual docker service update --image ...:<pin> rollback handle.

Done manually

Swept 99 versions on 2026-07-25 (334 → 235 instance-wide). Every affected package left at :latest + 3 newest pins; all :latest manifests verified still resolving (HTTP 200). This reopens immediately — two new pins landed within hours of the sweep.

Fix

Add a non-fatal prune step to the end of each build job, scoped to only the images that job just built:

keep :latest + the 3 newest pin tags per package
delete by explicit pin pattern (^[0-9a-f]{7}(-[0-9a-f]{7})?$), never by "not in keep list" — an unrecognized tag shape must be kept
never fail the build: pruning is housekeeping and must not waste a multi-GB image build

Consumes the existing ACCESS_TOKEN_GITEA (secret/ci/shared/gitea), which openbao/provision-batch4-secrets.sh already mints with write:package.

Explicitly out of scope

spikersoft-angular (76 versions, the larger offender) is not safe to sweep the same way. Its multi-arch pushes create untagged sha256:... version rows that are the manifest-list children holding the layers; the tagged row is only the index. Deleting a child leaves the tag resolving to nothing and breaks docker pull at manifest resolution. The package API returns no child digests, so that needs the registry v2 API to resolve reachability first. Separate ticket.

## Problem Both artpipe image workflows push a unique, immutable pin tag on every run and nothing ever removes them, so the Gitea package list grows without bound: - `spikersoft-backend/.gitea/workflows/spikersoft-artpipe-processor.yml` — tier 3, pushes `:latest` + `:<backendSHA7>-<artpipeSHA7>` - `spikersoft-artpipe/.gitea/workflows/model-env-images.yml` — tier 2, pushes `:latest` + `:<artpipeSHA7>` By 2026-07-25 `artpipe-model-safetycheck` alone had **32** pin versions; `prodstages` 22, `photostack` 21. Instance-wide there were 334 container versions, making the package list unusable. Nothing consumes a pin tag declaratively — every `spikersoft-infrastructure/spikersoft-artpipe-*/docker-stack*.yml` pins `:latest`. Pins exist only as a manual `docker service update --image ...:<pin>` rollback handle. ## Done manually Swept 99 versions on 2026-07-25 (334 → 235 instance-wide). Every affected package left at `:latest` + 3 newest pins; all `:latest` manifests verified still resolving (HTTP 200). This reopens immediately — two new pins landed within hours of the sweep. ## Fix Add a non-fatal prune step to the end of each build job, scoped to only the images that job just built: - keep `:latest` + the **3** newest pin tags per package - delete by explicit pin pattern (`^[0-9a-f]{7}(-[0-9a-f]{7})?$`), never by "not in keep list" — an unrecognized tag shape must be **kept** - never fail the build: pruning is housekeeping and must not waste a multi-GB image build Consumes the existing `ACCESS_TOKEN_GITEA` (`secret/ci/shared/gitea`), which `openbao/provision-batch4-secrets.sh` already mints with `write:package`. ## Explicitly out of scope `spikersoft-angular` (76 versions, the larger offender) is **not** safe to sweep the same way. Its multi-arch pushes create untagged `sha256:...` version rows that are the manifest-list children holding the layers; the tagged row is only the index. Deleting a child leaves the tag resolving to nothing and breaks `docker pull` at manifest resolution. The package API returns no child digests, so that needs the registry v2 API to resolve reachability first. Separate ticket.

spikerj commented

2026-07-25 22:25:14 +00:00

PRs open (not merged — Joey merges manually):

tier 3 — spikersoft-backend PR #481 ci/prune-artpipe-pin-tags-850
tier 2 — spikersoft-artpipe PR #40 ci/prune-env-pin-tags-850
doc fix — spikersoft-infrastructure PR #151 ci/gitea-token-scope-note-850

The infra one is a genuine pre-existing inaccuracy found along the way: the rotation prompt says secret/ci/shared/gitea needs only write:package, but artpipe's cascade dispatch has been using the same token and needs write:repository. Rotating strictly by those instructions would silently break the cascade.

Both prune steps were verified by extracting the script out of the YAML and running it against the live registry with DELETE intercepted — correct selections, and no-scope / empty-token / already-at-3 / hostile-tag-shape paths all behave (warn + exit 0, never fatal).

Not verified on a real runner: jq presence on ubuntu-amd64-4090. If it's missing the step warns and skips rather than failing, so the first real run's log is the confirmation.

PRs open (not merged — Joey merges manually): - **tier 3** — spikersoft-backend PR #481 `ci/prune-artpipe-pin-tags-850` - **tier 2** — spikersoft-artpipe PR #40 `ci/prune-env-pin-tags-850` - **doc fix** — spikersoft-infrastructure PR #151 `ci/gitea-token-scope-note-850` The infra one is a genuine pre-existing inaccuracy found along the way: the rotation prompt says `secret/ci/shared/gitea` needs only `write:package`, but artpipe's cascade dispatch has been using the same token and needs `write:repository`. Rotating strictly by those instructions would silently break the cascade. Both prune steps were verified by extracting the script out of the YAML and running it against the live registry with `DELETE` intercepted — correct selections, and no-scope / empty-token / already-at-3 / hostile-tag-shape paths all behave (warn + `exit 0`, never fatal). Not verified on a real runner: `jq` presence on `ubuntu-amd64-4090`. If it's missing the step warns and skips rather than failing, so the first real run's log is the confirmation.

spikerj commented

2026-07-25 22:30:21 +00:00

Status: merged, but a follow-up is needed before this is really done.

Merged: backend #481, artpipe #40, infrastructure #151.

All three merged the pre-fixup commits — a review fixup landed on the branches seconds after the merges, so it did not make it in. Two things are live in CI right now that shouldn't be:

!cancelled() in the prune step's if:. always() / failure() / continue-on-error are proven green on this instance, but cancelled() appears in no workflow in any SpikerSoft repo. act silently ignores expressions it can't evaluate, so the step may simply never run — and the log would be indistinguishable from "nothing to prune". Same silent-no-op class as the concurrency caveat already documented in model-env-images.yml.
404 misdiagnosed as a scope failure. Any new model's first build logs token needs write:package when the package just doesn't exist yet.

Follow-up PRs open: backend #482, artpipe #41. Keeping this ticket open until both merge.

If the next ArtPipeProcessor run logs no prune lines at all, cause (1) is the first thing to check — that's exactly the ambiguity #482 removes.

**Status: merged, but a follow-up is needed before this is really done.** Merged: backend #481, artpipe #40, infrastructure #151. All three merged the **pre-fixup** commits — a review fixup landed on the branches seconds after the merges, so it did not make it in. Two things are live in CI right now that shouldn't be: 1. **`!cancelled()` in the prune step's `if:`.** `always()` / `failure()` / `continue-on-error` are proven green on this instance, but `cancelled()` appears in **no** workflow in any SpikerSoft repo. act silently ignores expressions it can't evaluate, so the step may simply never run — and the log would be indistinguishable from "nothing to prune". Same silent-no-op class as the `concurrency` caveat already documented in `model-env-images.yml`. 2. **404 misdiagnosed as a scope failure.** Any new model's first build logs `token needs write:package` when the package just doesn't exist yet. Follow-up PRs open: backend **#482**, artpipe **#41**. Keeping this ticket open until both merge. If the next ArtPipeProcessor run logs no prune lines at all, cause (1) is the first thing to check — that's exactly the ambiguity #482 removes.

spikerj commented

2026-07-29 15:30:09 +00:00

Resolved — and note this supersedes the earlier "manual sweep only" understanding: CI now enforces the policy in both tiers. Merged via spikersoft-artpipe PRs #40/#41, spikersoft-backend PRs #481/#482, spikersoft-infrastructure PRs #151/#152. Verified against the remote default branches:

Tier 2 — spikersoft-artpipe/.gitea/workflows/model-env-images.yml:442, step "Prune old env pin tags (keep :latest + newest 3)": if: env.ENV_BUILT_IMAGES != '', continue-on-error: true, set -uo pipefail (deliberately not -e), scoped to only the images that job actually built. Selector test("^[0-9a-f]{7}$") at :437 matches tier-2's bare <artpipeSHA7> pin shape; :latest, artpipe-base's file-<hash> tags, hand tags and untagged sha256: rows all fail the match and are kept — opt-in by pattern, exactly as the ticket required. Empty-token and missing-jq paths warn and exit 0; paging up to 20 pages; a non-array 200 is treated as failure.

Tier 3 — spikersoft-backend/.gitea/workflows/spikersoft-artpipe-processor.yml:421, identical shape with test("^[0-9a-f]{7}-[0-9a-f]{7}$") for the <backendSHA7>-<artpipeSHA7> pin.

Both follow-ups this ticket was held open for are merged:

!cancelled() is gone from both if: expressions, replaced by the proven env.*_BUILT_IMAGES != '' form with an explanatory comment (artpipe :436-440 / backend :414-420).
404 is now diagnosed as "not in the registry yet — nothing to prune", separately from 401/403 scope failures.
Infra doc fix landed too: openbao/provision-batch4-secrets.sh:102-110 now states the token needs write:package and write:repository for the cascade dispatch.

One item genuinely unverified, per the ticket's own caveat: whether jq is present on ubuntu-amd64-4090 (the step warns and skips if not, so worst case is a silent no-op rather than a failure).

Separately — this pattern is not safe to copy to spikersoft-angular: its sha256 manifest children mean a tag-shaped sweep there would delete referenced layers. Keeping that noted here so nobody generalises the fix.

Closing.

Resolved — and note this supersedes the earlier "manual sweep only" understanding: **CI now enforces the policy in both tiers.** Merged via spikersoft-artpipe PRs #40/#41, spikersoft-backend PRs #481/#482, spikersoft-infrastructure PRs #151/#152. Verified against the remote default branches: **Tier 2** — `spikersoft-artpipe/.gitea/workflows/model-env-images.yml:442`, step "Prune old env pin tags (keep :latest + newest 3)": `if: env.ENV_BUILT_IMAGES != ''`, `continue-on-error: true`, `set -uo pipefail` (deliberately not `-e`), scoped to only the images that job actually built. Selector `test("^[0-9a-f]{7}$")` at `:437` matches tier-2's bare `<artpipeSHA7>` pin shape; `:latest`, `artpipe-base`'s `file-<hash>` tags, hand tags and untagged `sha256:` rows all fail the match and are kept — opt-in by pattern, exactly as the ticket required. Empty-token and missing-`jq` paths warn and `exit 0`; paging up to 20 pages; a non-array 200 is treated as failure. **Tier 3** — `spikersoft-backend/.gitea/workflows/spikersoft-artpipe-processor.yml:421`, identical shape with `test("^[0-9a-f]{7}-[0-9a-f]{7}$")` for the `<backendSHA7>-<artpipeSHA7>` pin. **Both follow-ups this ticket was held open for are merged:** - `!cancelled()` is gone from both `if:` expressions, replaced by the proven `env.*_BUILT_IMAGES != ''` form with an explanatory comment (artpipe `:436-440` / backend `:414-420`). - 404 is now diagnosed as "not in the registry yet — nothing to prune", separately from 401/403 scope failures. - Infra doc fix landed too: `openbao/provision-batch4-secrets.sh:102-110` now states the token needs `write:package` **and** `write:repository` for the cascade dispatch. One item genuinely unverified, per the ticket's own caveat: whether `jq` is present on `ubuntu-amd64-4090` (the step warns and skips if not, so worst case is a silent no-op rather than a failure). Separately — this pattern is **not** safe to copy to spikersoft-angular: its sha256 manifest children mean a tag-shaped sweep there would delete referenced layers. Keeping that noted here so nobody generalises the fix. Closing.

spikerj closed this issue

2026-07-29 15:30:14 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#850