Backend CI: create_manifest breaks on manifest-list arch tags (M5 containerd store) + dead v${VERSION} manifest code #917

New Issue

2026-08-05T15:08:01Z

spikerj commented

2026-08-05 15:08:01 +00:00

Two related defects in the multi-arch manifest stitching, surfaced by the new M5 arm runners (2026-08-05):

docker manifest create --amend rejects manifest-list sources. The M5 runners build with Docker Desktop's containerd image store, which pushes the :arm64v8 tag as an OCI manifest list (image + attestations) instead of a single-platform manifest. create_manifest then fails with ...:arm64v8 is a manifest list (seen on spikersoft-backend run 19420 after the arm job went green on m5-runner-3). Jetson-built tags are classic single manifests, which is why this never fired before. Fix: stitch with docker buildx imagetools create, which accepts both shapes — makes the pipeline agnostic to which runner/store built each arch.
spikersoft-api.yml never pushes versioned tags. The docker manifest create ...:v${VERSION} block sits INSIDE the "Verify image is pullable" step after its exit 0/exit 1 paths — unreachable on every outcome (and VERSION is not defined in that step). The :v<version> tags have silently never existed.

Affected: .gitea/workflows/_build-deploy-service.yml (all services) and .gitea/workflows/spikersoft-api.yml.

Two related defects in the multi-arch manifest stitching, surfaced by the new M5 arm runners (2026-08-05): 1. **`docker manifest create --amend` rejects manifest-list sources.** The M5 runners build with Docker Desktop's containerd image store, which pushes the `:arm64v8` tag as an OCI **manifest list** (image + attestations) instead of a single-platform manifest. `create_manifest` then fails with `...:arm64v8 is a manifest list` (seen on spikersoft-backend run 19420 after the arm job went green on m5-runner-3). Jetson-built tags are classic single manifests, which is why this never fired before. Fix: stitch with `docker buildx imagetools create`, which accepts both shapes — makes the pipeline agnostic to which runner/store built each arch. 2. **`spikersoft-api.yml` never pushes versioned tags.** The `docker manifest create ...:v${VERSION}` block sits INSIDE the "Verify image is pullable" step **after** its `exit 0`/`exit 1` paths — unreachable on every outcome (and `VERSION` is not defined in that step). The `:v<version>` tags have silently never existed. Affected: `.gitea/workflows/_build-deploy-service.yml` (all services) and `.gitea/workflows/spikersoft-api.yml`.

spikerj referenced this issue

2026-08-05 15:55:17 +00:00

file-movement CI: secret/ci/backend/minio/movement never provisioned — pipeline can't deploy #918

spikerj commented

2026-08-05 15:55:17 +00:00

Resolved in spikersoft-backend PR #518 (merged). Verified at scale: the merge fired all service pipelines — 24/25 fully green, including 21 create_manifest successes stitching m5-built manifest-list arm tags with buildx imagetools, plus Jetson-built classic manifests in the same fleet. The single red run (file-movement, 19459) is an unrelated pre-existing Bao provisioning gap, tracked separately. The dead v${VERSION} stitch in spikersoft-api.yml is also live now. Closing.

Resolved in spikersoft-backend PR #518 (merged). Verified at scale: the merge fired all service pipelines — 24/25 fully green, including 21 `create_manifest` successes stitching m5-built manifest-list arm tags with `buildx imagetools`, plus Jetson-built classic manifests in the same fleet. The single red run (file-movement, 19459) is an unrelated pre-existing Bao provisioning gap, tracked separately. The dead `v${VERSION}` stitch in spikersoft-api.yml is also live now. Closing.

spikerj closed this issue

2026-08-05 15:55:17 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#917