[Bug][CI][Registry] #548 was closed but its fix NEVER LANDED — all 23 of 23 multi-arch workflows still --amend mutable :amd64/:arm64v8 tags, and the cancelled-run window from #590 makes mixed-commit :latest reachable today #596

New Issue

2026-07-14T21:27:28Z

spikerj commented

2026-07-14 21:27:28 +00:00

Filed by: QA Team — referencing #548, which is CLOSED but whose fix never landed in a single workflow.

Summary

#548 ("Multi-arch create_manifest amends MUTABLE :amd64/:arm64v8 tags — out-of-order arch pushes can stitch a mixed-commit :latest") was closed on 2026-07-14. The described fix is not present in the repository. Every multi-arch workflow still uses the exact pattern the ticket flagged.

Evidence

Audited every workflow in .gitea/workflows/ that calls docker manifest create:

23 of 23 multi-arch workflows still --amend mutable arch tags

spikersoft-api.yml                    spikersoft-influx-dashboard.yml
spikersoft-artstudio-metrics.yml      spikersoft-keycloak-events.yml
spikersoft-blog-media-processor.yml   spikersoft-lesson-video-processor.yml
spikersoft-book-management.yml        spikersoft-metadata-extractor.yml
spikersoft-calendar-reminders.yml     spikersoft-node-agent.yml
spikersoft-coderunner.yml             spikersoft-notifications.yml
spikersoft-docker-monitor.yml         spikersoft-scheduler.yml
spikersoft-file-movement.yml          spikersoft-security-monitor.yml
spikersoft-game-events.yml            spikersoft-security-scanner.yml
spikersoft-gameserver-init.yml        spikersoft-system-remediation.yml
spikersoft-gameserver.yml             spikersoft-upload-coordinator.yml
spikersoft-gpu-coordinator.yml

The pattern, verbatim from spikersoft-coderunner.yml:75:

docker manifest create git.spikersoft.com/spikerj/spikersoft-coderunner:latest \
  --amend git.spikersoft.com/spikerj/spikersoft-coderunner:amd64 \
  --amend git.spikersoft.com/spikerj/spikersoft-coderunner:arm64v8

Zero workflows were converted to digest-pinned stitching. The ticket appears to have been closed without the change being made — or the change was reverted and nobody noticed, because nothing verifies it.

Why this is worse now than when #548 was written

It compounds #587/#590. The confirmed root cause of #587 is that cancel-in-progress: true can cancel a run after the arch tags are pushed but before create_manifest stitches :latest. Because the arch tags are mutable, this leaves the registry in a state where:

:amd64 may be from commit A (pushed, then the run was cancelled), and
:arm64v8 may be from commit B (a later run that completed),

and the next successful create_manifest will happily --amend those two into a :latest that is a mixed-commit chimera — an amd64 image from one commit and an arm64 image from another, under one tag, with no error and no warning.

That is exactly the failure #548 predicted. The conditions to trigger it (cancelled runs mid-push) are demonstrably occurring right now — see #587, where 10 images are sitting with healthy arch tags and no :latest.

Fix

Stitch from immutable digests, not mutable tags. Capture each arch push's digest as a job output and use docker buildx imagetools create against the digests:

# in build_and_publish, per arch:
- id: push
  run: |
    digest=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE:$ARCH")
    echo "digest=$digest" >> "$GITHUB_OUTPUT"

# in create_manifest:
- run: |
    docker buildx imagetools create -t "$IMAGE:latest" \
      "${{ needs.build_amd64.outputs.digest }}" \
      "${{ needs.build_arm64.outputs.digest }}"

Best combined with the #590 fix — make the arch push and the manifest stitch atomic (single job, or excluded from cancellation), so the tags and the index either both land or neither does. That closes the mixed-commit window and the missing-:latest window in one change.

Suggested process note

#548 was closed with no verification that the change existed. A one-line CI guard would prevent a recurrence:

! grep -rq -- '--amend.*:\(amd64\|arm64v8\)' .gitea/workflows/

**Filed by: QA Team** — referencing **#548**, which is **CLOSED but whose fix never landed in a single workflow.** ## Summary #548 ("Multi-arch `create_manifest` amends MUTABLE `:amd64`/`:arm64v8` tags — out-of-order arch pushes can stitch a mixed-commit `:latest`") was closed on 2026-07-14. **The described fix is not present in the repository.** Every multi-arch workflow still uses the exact pattern the ticket flagged. ## Evidence Audited every workflow in `.gitea/workflows/` that calls `docker manifest create`: ``` 23 of 23 multi-arch workflows still --amend mutable arch tags ``` ``` spikersoft-api.yml spikersoft-influx-dashboard.yml spikersoft-artstudio-metrics.yml spikersoft-keycloak-events.yml spikersoft-blog-media-processor.yml spikersoft-lesson-video-processor.yml spikersoft-book-management.yml spikersoft-metadata-extractor.yml spikersoft-calendar-reminders.yml spikersoft-node-agent.yml spikersoft-coderunner.yml spikersoft-notifications.yml spikersoft-docker-monitor.yml spikersoft-scheduler.yml spikersoft-file-movement.yml spikersoft-security-monitor.yml spikersoft-game-events.yml spikersoft-security-scanner.yml spikersoft-gameserver-init.yml spikersoft-system-remediation.yml spikersoft-gameserver.yml spikersoft-upload-coordinator.yml spikersoft-gpu-coordinator.yml ``` The pattern, verbatim from `spikersoft-coderunner.yml:75`: ```yaml docker manifest create git.spikersoft.com/spikerj/spikersoft-coderunner:latest \ --amend git.spikersoft.com/spikerj/spikersoft-coderunner:amd64 \ --amend git.spikersoft.com/spikerj/spikersoft-coderunner:arm64v8 ``` Zero workflows were converted to digest-pinned stitching. The ticket appears to have been closed without the change being made — or the change was reverted and nobody noticed, because nothing verifies it. ## Why this is worse now than when #548 was written It **compounds #587/#590**. The confirmed root cause of #587 is that `cancel-in-progress: true` can cancel a run *after* the arch tags are pushed but *before* `create_manifest` stitches `:latest`. Because the arch tags are **mutable**, this leaves the registry in a state where: - `:amd64` may be from commit **A** (pushed, then the run was cancelled), and - `:arm64v8` may be from commit **B** (a later run that completed), and the next successful `create_manifest` will happily `--amend` those two into a **`:latest` that is a mixed-commit chimera** — an amd64 image from one commit and an arm64 image from another, under one tag, with no error and no warning. That is exactly the failure #548 predicted. The conditions to trigger it (cancelled runs mid-push) are **demonstrably occurring right now** — see #587, where 10 images are sitting with healthy arch tags and no `:latest`. ## Fix Stitch from **immutable digests**, not mutable tags. Capture each arch push's digest as a job output and use `docker buildx imagetools create` against the digests: ```yaml # in build_and_publish, per arch: - id: push run: | digest=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE:$ARCH") echo "digest=$digest" >> "$GITHUB_OUTPUT" # in create_manifest: - run: | docker buildx imagetools create -t "$IMAGE:latest" \ "${{ needs.build_amd64.outputs.digest }}" \ "${{ needs.build_arm64.outputs.digest }}" ``` Best combined with the **#590** fix — make the arch push and the manifest stitch **atomic** (single job, or excluded from cancellation), so the tags and the index either both land or neither does. That closes the mixed-commit window and the missing-`:latest` window in one change. ## Suggested process note #548 was closed with no verification that the change existed. A one-line CI guard would prevent a recurrence: ```bash ! grep -rq -- '--amend.*:$amd64\|arm64v8$' .gitea/workflows/ ``` ## Related - **#548** (closed) — the original ticket; fix never landed. - **#587** (open) — 10 images with healthy arch tags and no `:latest`; the cancelled-run window that makes this exploitable. - **#590** (open) — `cancel-in-progress` is what creates that window.

spikerj referenced this issue

2026-07-14 21:31:22 +00:00

[Bug][Backend][Observability] #575 was fixed in ONE publisher — BookController (5 publish sites) and 3 others still ship RabbitMQ messages with no traceparent, so user-initiated flows appear in Jaeger as disconnected orphan traces #597

spikerj referenced this issue

2026-07-14 21:35:20 +00:00

[Bug][CI][Deploy] cancel-in-progress on master kills DEPLOY jobs — back-to-back merges cancel each other and fixes silently never ship #590

spikerj referenced this issue

2026-07-14 22:36:55 +00:00

[Bug][Prod][Registry] 14 images have no :latest manifest (manifest unknown / 404) — any reschedule onto a fresh node will fail to start #587

spikerj commented

2026-07-16 05:17:57 +00:00

Partial hardening landed: spikersoft-backend PR #317 (merged) gives the coderunner workflow the concurrency guard the other multi-arch workflows already carry (PR runs cancel, master runs queue so a started master build always finishes its manifest) and gates docker push off PR events. This narrows the interleaving window for that one workflow but is not the fix this ticket asks for — digest-pinned manifest stitching across all 23 workflows remains to be done, so leaving this open.

Partial hardening landed: spikersoft-backend PR #317 (merged) gives the coderunner workflow the `concurrency` guard the other multi-arch workflows already carry (PR runs cancel, master runs queue so a started master build always finishes its manifest) and gates `docker push` off PR events. This narrows the interleaving window for that one workflow but is **not** the fix this ticket asks for — digest-pinned manifest stitching across all 23 workflows remains to be done, so leaving this open.

spikerj closed this issue

2026-07-17 03:55:16 +00:00

spikerj referenced this issue

2026-07-28 03:20:11 +00:00

[Enhancement][ArtPipe][Angular] AI QR code generation — SDXL qrcode_monster ControlNet per-model lane + authenticated AI button in the QR tool #535

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#596