spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Prod][Registry] 14 images have no :latest manifest (manifest unknown / 404) — any reschedule onto a fresh node will fail to start #587

New Issue
Closed
opened 2026-07-14 20:24:06 +00:00 by spikerj · 4 comments
spikerj commented 2026-07-14 20:24:06 +00:00
Owner
Copy Link
Copy Source

QA Team — sweep 2026-07-14 ~21:15Z. Filed against closed #548 (Multi-arch create_manifest amends MUTABLE :amd64/:arm64v8 tags). #548 named the mechanism. This is the damage it left behind, and it is still live.

The bug

14 deployed services have a :latest manifest list whose child manifests do not exist in the registry. Pulling any of them fails. They keep running only because the image is cached on the node they happen to be sitting on.

Worked example — spikersoft-coderunner:latest:

$ GET /v2/spikerj/spikersoft-coderunner/manifests/latest
  INDEX (manifest.list.v2)
    linux/amd64  sha256:8b2dad884ba68...
    linux/arm64  sha256:e91db1f3be049...

$ GET /v2/.../manifests/sha256:8b2dad884ba68...   -> HTTP 404
$ GET /v2/.../manifests/sha256:e91db1f3be049...   -> HTTP 404
$ GET /v2/.../blobs/sha256:e91db1f3be049...       -> HTTP 404

Both children 404. Meanwhile the standalone per-arch tag is healthy and points somewhere else entirely:

$ GET /v2/.../manifests/arm64v8  -> HTTP 200, digest sha256:6ce3457e4bd0a...
   index says arm64 =              sha256:e91db1f3be049...     <-- different, and gone

Not a media-type artifact: the 404 reproduces with Accept: */*, and the :arm64v8 control returns 200 with identical headers.

Mechanism (this is #548, playing out)

  1. create_manifest stitches :latest from the mutable :amd64 / :arm64v8 tags, capturing their digests at that moment.
  2. A later build overwrites :amd64 / :arm64v8 with new digests.
  3. The old manifests are now untagged, and the registry garbage-collects them.
  4. :latest still references the deleted digests → dangling index.

#548 identified the mutable-tag race. What nobody wrote down is the consequence: when the per-arch tag moves, the old digest is GC'd and :latest rots. Closing #548 did not repair the already-rotted indexes.

This is the root cause of #562

#562 ("dreamstream nodes reject fresh tasks with No such image") is not a UDM hairpin, not a slow pull, not registry auth. Verified:

  • coderunner passes --with-registry-auth ✅
  • jetson disk usage is 16–36% across all 7 nodes (via InfluxDB jetsons bucket) — not a disk problem ✅
  • ds1/ds4/ds5 are the same architecture ✅

The asymmetry is entirely explained: ds5 has the image cached (its task has been Running 6 h). ds1 and ds4 must actually pull :latest, they follow the index to a 404, and Docker reports the only thing it can — No such image. They will reject forever; there is nothing to converge to. That is why coderunner has sat at 3/4 all day rather than self-healing in "1-3 minutes" as #562 assumed.

Blast radius — this is a latent cluster-wide outage

Every one of these is unpullable right now. They survive only on cached images:

service replicas
spikersoft-coderunner 3/4 — already failing
spikersoft-backend (the API) 1/1
spikersoft-notifications 1/1
spikersoft-gpu-coordinator 1/1
spikersoft-file-movement 1/1
spikersoft-book-management 1/1
spikersoft-blog-media-processor 1/1
spikersoft-game-events 1/1
spikersoft-keycloak-events 1/1
spikersoft-influx-dashboard 1/1
spikersoft-calendar-reminders 1/1
spikersoft-artstudio-metrics 1/1
spikersoft-docker-monitor 1/1
spikersoft-gameserver-init 0/1

Any event that forces a fresh pull — a node reboot, a failover, a rescheduling, a scale-out, docker system prune on a node — takes that service to zero and it cannot come back. The API is on this list.

This also retro-explains the "No such image" reject-loops in #511 and the ones seen during the ds4/ds6 outages (#552): those nodes were being asked to pull images whose :latest was already dangling.

Immediate remediation

Re-stitch :latest for all 14 from their current, healthy per-arch tags — i.e. re-run create_manifest (or docker buildx imagetools create -t <img>:latest <img>:amd64 <img>:arm64v8). That is a registry-side fix and needs no code change. Do coderunner first — it is the one already degraded.

Real fix

create_manifest must not stitch from mutable tags whose digests can be GC'd out from under it. Options:

  • Tag per-arch images immutably (:amd64-<commit-sha>) and stitch :latest from those, so the referenced digests are never orphaned; or
  • Push the arch manifests by digest and build the index in the same job that pushed them (buildx --push with a single multi-platform build does this atomically — one build, one index, no window); or
  • At minimum, add a post-push verification step: after create_manifest, resolve every child of :latest and fail the job if any 404s. That turns this from a silent time-bomb into a red build.

The verification step is worth adding regardless — it is ~5 lines and it would have caught all 14 of these at the moment they broke, instead of months later via a stuck replica.

Related: closed #548 (the mechanism), #562 (root-caused by this — should be closed in favour of this ticket or linked), #511, #552, #584.

**QA Team** — sweep 2026-07-14 ~21:15Z. Filed against **closed #548** (*Multi-arch create_manifest amends MUTABLE :amd64/:arm64v8 tags*). #548 named the mechanism. This is the damage it left behind, and it is still live. ## The bug **14 deployed services have a `:latest` manifest list whose child manifests do not exist in the registry.** Pulling any of them fails. They keep running *only* because the image is cached on the node they happen to be sitting on. Worked example — `spikersoft-coderunner:latest`: ``` $ GET /v2/spikerj/spikersoft-coderunner/manifests/latest INDEX (manifest.list.v2) linux/amd64 sha256:8b2dad884ba68... linux/arm64 sha256:e91db1f3be049... $ GET /v2/.../manifests/sha256:8b2dad884ba68... -> HTTP 404 $ GET /v2/.../manifests/sha256:e91db1f3be049... -> HTTP 404 $ GET /v2/.../blobs/sha256:e91db1f3be049... -> HTTP 404 ``` **Both children 404.** Meanwhile the standalone per-arch tag is healthy and points somewhere else entirely: ``` $ GET /v2/.../manifests/arm64v8 -> HTTP 200, digest sha256:6ce3457e4bd0a... index says arm64 = sha256:e91db1f3be049... <-- different, and gone ``` Not a media-type artifact: the 404 reproduces with `Accept: */*`, and the `:arm64v8` control returns **200** with identical headers. ## Mechanism (this is #548, playing out) 1. `create_manifest` stitches `:latest` from the **mutable** `:amd64` / `:arm64v8` tags, capturing their digests *at that moment*. 2. A later build **overwrites** `:amd64` / `:arm64v8` with new digests. 3. The old manifests are now untagged, and the registry garbage-collects them. 4. `:latest` still references the deleted digests → **dangling index**. #548 identified the mutable-tag race. What nobody wrote down is the consequence: **when the per-arch tag moves, the old digest is GC'd and `:latest` rots.** Closing #548 did not repair the already-rotted indexes. ## This is the root cause of #562 #562 ("dreamstream nodes reject fresh tasks with `No such image`") is not a UDM hairpin, not a slow pull, not registry auth. Verified: - coderunner passes `--with-registry-auth` ✅ - jetson disk usage is 16–36% across all 7 nodes (via InfluxDB `jetsons` bucket) — **not** a disk problem ✅ - ds1/ds4/ds5 are the **same architecture** ✅ The asymmetry is entirely explained: **ds5 has the image cached** (its task has been Running 6 h). ds1 and ds4 must actually *pull* `:latest`, they follow the index to a 404, and Docker reports the only thing it can — `No such image`. They will reject forever; there is nothing to converge to. That is why coderunner has sat at 3/4 all day rather than self-healing in "1-3 minutes" as #562 assumed. ## Blast radius — this is a latent cluster-wide outage Every one of these is **unpullable right now**. They survive only on cached images: | service | replicas | |---|---| | spikersoft-coderunner | **3/4 — already failing** | | spikersoft-backend (the API) | 1/1 | | spikersoft-notifications | 1/1 | | spikersoft-gpu-coordinator | 1/1 | | spikersoft-file-movement | 1/1 | | spikersoft-book-management | 1/1 | | spikersoft-blog-media-processor | 1/1 | | spikersoft-game-events | 1/1 | | spikersoft-keycloak-events | 1/1 | | spikersoft-influx-dashboard | 1/1 | | spikersoft-calendar-reminders | 1/1 | | spikersoft-artstudio-metrics | 1/1 | | spikersoft-docker-monitor | 1/1 | | spikersoft-gameserver-init | 0/1 | **Any** event that forces a fresh pull — a node reboot, a failover, a rescheduling, a scale-out, `docker system prune` on a node — takes that service to **zero and it cannot come back**. The API is on this list. This also retro-explains the "No such image" reject-loops in **#511** and the ones seen during the ds4/ds6 outages (**#552**): those nodes were being asked to pull images whose `:latest` was already dangling. ## Immediate remediation Re-stitch `:latest` for all 14 from their current, healthy per-arch tags — i.e. re-run `create_manifest` (or `docker buildx imagetools create -t <img>:latest <img>:amd64 <img>:arm64v8`). That is a registry-side fix and needs no code change. **Do coderunner first** — it is the one already degraded. ## Real fix `create_manifest` must not stitch from mutable tags whose digests can be GC'd out from under it. Options: - Tag per-arch images **immutably** (`:amd64-<commit-sha>`) and stitch `:latest` from those, so the referenced digests are never orphaned; or - Push the arch manifests **by digest** and build the index in the same job that pushed them (buildx `--push` with a single multi-platform build does this atomically — one build, one index, no window); or - At minimum, add a **post-push verification** step: after `create_manifest`, resolve every child of `:latest` and fail the job if any 404s. That turns this from a silent time-bomb into a red build. The verification step is worth adding regardless — it is ~5 lines and it would have caught all 14 of these at the moment they broke, instead of months later via a stuck replica. Related: closed **#548** (the mechanism), **#562** (root-caused by this — should be closed in favour of this ticket or linked), **#511**, **#552**, **#584**.
spikerj commented 2026-07-14 20:49:35 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — still live, still causing #562. Re-verified 2026-07-14.

The dangling manifests have not self-healed. Spot-check from a swarm node:

docker manifest inspect git.spikersoft.com/spikerj/spikersoft-quiz-generation:latest   -> OK   (re-pushed by a recent build)
docker manifest inspect git.spikersoft.com/spikerj/spikersoft-coderunner:latest        -> FAIL (still dangling)
docker manifest inspect git.spikersoft.com/spikerj/spikersoft-security-monitor:latest  -> FAIL (still dangling)

Note the pattern: quiz-generation recovered only because it happened to get rebuilt and re-pushed. Nothing repaired it deliberately. Every image that has not been rebuilt since the corruption is still broken — which means this is silently waiting to bite any service that next needs a fresh pull.

#562 is not a separate incident — it is this ticket, happening. spikersoft-coderunner is stuck at 3/4 right now, in an active reject loop on dreamstream1:

$ docker service ps spikersoft-coderunner_spikersoft-coderunner
  dreamstream1  Preparing 28 seconds ago
  dreamstream1  Rejected  31 seconds ago      "No such image: git.spikersoft…"
  dreamstream1  Rejected  about a minute ago  "No such image: git.spikersoft…"
  dreamstream1  Rejected  2 minutes ago       "No such image: git.spikersoft…"

Swarm has been retrying that 4th replica continuously. It cannot ever succeed, because the manifest it is trying to pull 404s.

Fix options, in order of preference:

  1. Re-run each affected service's CI workflow. Rebuilds and re-pushes a correct manifest index. Cleanest — no direct registry surgery, and it is provably what fixed quiz-generation. Downside: 14 workflow runs.
  2. Re-stitch the index in place with docker buildx imagetools create -t <img>:latest <arch-digest> …. Fast, but writes directly to the production registry.

Either way, this needs doing deliberately — it is not going to fix itself, and the 3/4 reject loop is burning scheduler cycles on dreamstream1 continuously.

**QA Team — still live, still causing #562. Re-verified 2026-07-14.** The dangling manifests have **not** self-healed. Spot-check from a swarm node: ``` docker manifest inspect git.spikersoft.com/spikerj/spikersoft-quiz-generation:latest -> OK (re-pushed by a recent build) docker manifest inspect git.spikersoft.com/spikerj/spikersoft-coderunner:latest -> FAIL (still dangling) docker manifest inspect git.spikersoft.com/spikerj/spikersoft-security-monitor:latest -> FAIL (still dangling) ``` Note the pattern: **quiz-generation recovered only because it happened to get rebuilt and re-pushed.** Nothing repaired it deliberately. Every image that has *not* been rebuilt since the corruption is still broken — which means this is silently waiting to bite any service that next needs a fresh pull. **#562 is not a separate incident — it is this ticket, happening.** `spikersoft-coderunner` is stuck at **3/4** right now, in an active reject loop on dreamstream1: ``` $ docker service ps spikersoft-coderunner_spikersoft-coderunner dreamstream1 Preparing 28 seconds ago dreamstream1 Rejected 31 seconds ago "No such image: git.spikersoft…" dreamstream1 Rejected about a minute ago "No such image: git.spikersoft…" dreamstream1 Rejected 2 minutes ago "No such image: git.spikersoft…" ``` Swarm has been retrying that 4th replica continuously. It cannot ever succeed, because the manifest it is trying to pull 404s. **Fix options, in order of preference:** 1. **Re-run each affected service's CI workflow.** Rebuilds and re-pushes a correct manifest index. Cleanest — no direct registry surgery, and it is provably what fixed quiz-generation. Downside: 14 workflow runs. 2. **Re-stitch the index in place** with `docker buildx imagetools create -t <img>:latest <arch-digest> …`. Fast, but writes directly to the production registry. Either way, this needs doing deliberately — it is not going to fix itself, and the 3/4 reject loop is burning scheduler cycles on dreamstream1 continuously.
spikerj commented 2026-07-14 21:07:07 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — RETRACTION. This ticket's claim that it is "the root cause of #562" is WRONG. I filed that; I'm correcting it.

#562 is a pull-path problem, not a manifest problem. git.spikersoft.com resolves to a public IP (204.197.150.99) from inside the cluster, so every node pull hairpins out to the WAN and back. That — not manifest corruption — is what makes fresh pulls fail on the Jetsons. Full evidence and the split-horizon DNS fix are now on #562. The reporter's original hairpin theory was correct.

Specifically, spikersoft-coderunner (the service I cited as proof) is not dangling at all:

  • its :latest index resolves, and both amd64 and arm64 children resolve;
  • the registry serves it anonymously, HTTP 200;
  • three aarch64 Jetsons are running that exact image right now.

Docker surfaces a failed pull as No such image, which is what led me to conflate the two. My mistake.

I've retitled this ticket accordingly. What remains below is still real and still worth fixing.

What IS real: 14 images have no :latest manifest

Deterministic — I probed each 5×, and every result was stable. The failure is a clean manifest unknown (a true 404), not a flake and not an auth error (19 other images resolve fine over the same anonymous path):

jetson-tx2-influxdb-grafana      spikersoft-metadata-extractor
sonar-scanner-datacenter         spikersoft-node-agent
spikersoft-backend               spikersoft-notifications
spikersoft-docker-monitor        spikersoft-security-monitor
spikersoft-gameserver            spikersoft-security-scanner
spikersoft-influx-dashboard      spikersoft-system-remediation
spikersoft-keycloak-events       spikersoft-upload-coordinator

These are currently harmless only because the running tasks were pulled long ago. Any of these services that gets rescheduled, scaled, or drained onto a new node will fail to start, and there is no fallback. It is a landmine, not an outage — but it is 14 landmines.

Note spikersoft-quiz-generation was on this list earlier today and is now OK — not because anything repaired it, but because it happened to get rebuilt and re-pushed by infra #81's deploy. That is the fix, and it confirms the remedy below works.

Fix

Re-run each affected service's CI workflow to rebuild and re-push a correct manifest index. No registry surgery required.

⚠️ Sequencing: the #562 DNS fix should land first. Several of these are arm64 builds, and the ARM runner on dreamstream6 is still crash-looping (#579), so rebuilds may be slow or need retries until both of those are resolved.

**QA Team — RETRACTION. This ticket's claim that it is "the root cause of #562" is WRONG. I filed that; I'm correcting it.** #562 is a **pull-path** problem, not a manifest problem. `git.spikersoft.com` resolves to a **public IP (204.197.150.99) from inside the cluster**, so every node pull hairpins out to the WAN and back. That — not manifest corruption — is what makes fresh pulls fail on the Jetsons. Full evidence and the split-horizon DNS fix are now on **#562**. The reporter's original hairpin theory was correct. Specifically, `spikersoft-coderunner` (the service I cited as proof) is **not** dangling at all: - its `:latest` index resolves, and **both** amd64 and arm64 children resolve; - the registry serves it **anonymously**, HTTP 200; - **three aarch64 Jetsons are running that exact image right now**. Docker surfaces a failed *pull* as `No such image`, which is what led me to conflate the two. My mistake. **I've retitled this ticket accordingly.** What remains below is still real and still worth fixing. --- ## What IS real: 14 images have no `:latest` manifest Deterministic — I probed each 5×, and every result was stable. The failure is a clean `manifest unknown` (a true 404), not a flake and not an auth error (19 other images resolve fine over the same anonymous path): ``` jetson-tx2-influxdb-grafana spikersoft-metadata-extractor sonar-scanner-datacenter spikersoft-node-agent spikersoft-backend spikersoft-notifications spikersoft-docker-monitor spikersoft-security-monitor spikersoft-gameserver spikersoft-security-scanner spikersoft-influx-dashboard spikersoft-system-remediation spikersoft-keycloak-events spikersoft-upload-coordinator ``` These are currently harmless **only** because the running tasks were pulled long ago. Any of these services that gets rescheduled, scaled, or drained onto a new node will fail to start, and there is no fallback. It is a landmine, not an outage — but it is 14 landmines. Note `spikersoft-quiz-generation` was on this list earlier today and is now OK — **not** because anything repaired it, but because it happened to get rebuilt and re-pushed by infra #81's deploy. That is the fix, and it confirms the remedy below works. ## Fix Re-run each affected service's CI workflow to rebuild and re-push a correct manifest index. No registry surgery required. ⚠️ Sequencing: the **#562 DNS fix should land first**. Several of these are arm64 builds, and the ARM runner on dreamstream6 is still crash-looping (**#579**), so rebuilds may be slow or need retries until both of those are resolved.
spikerj changed title from [Bug][Prod][Registry] 14 services have DANGLING :latest manifests — the index points at child digests that 404. Any fresh pull fails. This is the root cause of #562 to [Bug][Prod][Registry] 14 images have no :latest manifest (manifest unknown / 404) — any reschedule onto a fresh node will fail to start 2026-07-14 21:07:07 +00:00
spikerj commented 2026-07-14 21:26:54 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — ROOT CAUSE FOUND, and it is NOT registry corruption. The images are all fine.

I probed each dangling image for its architecture tags as well as :latest. The result is decisive:

                                 :latest   :amd64   :arm64v8
spikersoft-backend               MISSING   ok       ok
spikersoft-docker-monitor        MISSING   ok       ok
spikersoft-influx-dashboard      MISSING   ok       ok
spikersoft-metadata-extractor    MISSING   ok       ok
spikersoft-node-agent            MISSING   ok       ok
spikersoft-notifications         MISSING   ok       ok
spikersoft-security-monitor      MISSING   ok       ok
spikersoft-security-scanner      MISSING   ok       ok
spikersoft-system-remediation    MISSING   ok       ok
spikersoft-upload-coordinator    MISSING   ok       ok

Both architecture builds succeeded and pushed. Only the multi-arch index (:latest) was never created. Nothing is corrupt, nothing was garbage-collected, and no registry surgery is needed. The layers are sitting in the registry, healthy and pullable by digest.

The mechanism — this is #590, and it does more damage than that ticket says

Every affected workflow has:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
...
create_manifest:
  needs: [build_and_publish]     # <-- the job that stitches :latest

build_and_publish pushes :amd64 and :arm64v8. create_manifest then runs docker manifest create ... :latest from those tags. Because cancel-in-progress: true is set, a back-to-back push to master cancels the in-flight run. If the cancellation lands after the arch builds have pushed but before create_manifest executes, you get exactly the state above: healthy arch tags, no :latest.

#590 describes this concurrency setting killing deploy jobs. It also silently kills manifest stitching — which is a distinct and arguably worse failure, because a skipped deploy is a no-op, whereas a skipped create_manifest leaves :latest pointing at nothing and breaks every future pull on every node. #590 should be widened to cover this.

Corroboration

Two images on the original list have since recovered on their own — spikersoft-gameserver and spikersoft-keycloak-events now resolve :latest fine. Neither was repaired by hand. They simply got rebuilt by a workflow run that was not cancelled, and create_manifest completed. Same as spikersoft-quiz-generation earlier today. That is the mechanism confirming itself.

Two entries are not part of this at all — jetson-tx2-influxdb-grafana and sonar-scanner-datacenter have no tags whatsoever (:latest, :amd64, :arm64v8 all missing). They are third-party/externally-sourced images that were never pushed to this registry. They should be dropped from this ticket's scope.

Fix

Cheap and immediate: re-run each affected workflow. create_manifest will stitch :latest from the arch tags that are already in the registry — no rebuild of the layers is strictly required, so these runs should be fast.

Durable — fix the cause, or this recurs indefinitely: address #590. Either drop cancel-in-progress: true on master, or (better) exclude the publish/manifest/deploy jobs from cancellation so that a superseded run still finishes stitching and shipping what it already built. Cancelling a build is fine; cancelling after a partial push is what corrupts the tag state.

Related: #548 (closed) — the --amend against mutable :amd64/:arm64v8 tags. That fix never actually landed; all 23 of 23 multi-arch workflows still use the mutable pattern. Filed separately. It compounds this: because the arch tags are mutable, a cancelled run can leave :amd64 and :arm64v8 pointing at different commits, so a later create_manifest can stitch a mixed-commit :latest that looks healthy but is not.

**QA Team — ROOT CAUSE FOUND, and it is NOT registry corruption. The images are all fine.** I probed each dangling image for its **architecture** tags as well as `:latest`. The result is decisive: ``` :latest :amd64 :arm64v8 spikersoft-backend MISSING ok ok spikersoft-docker-monitor MISSING ok ok spikersoft-influx-dashboard MISSING ok ok spikersoft-metadata-extractor MISSING ok ok spikersoft-node-agent MISSING ok ok spikersoft-notifications MISSING ok ok spikersoft-security-monitor MISSING ok ok spikersoft-security-scanner MISSING ok ok spikersoft-system-remediation MISSING ok ok spikersoft-upload-coordinator MISSING ok ok ``` **Both architecture builds succeeded and pushed. Only the multi-arch index (`:latest`) was never created.** Nothing is corrupt, nothing was garbage-collected, and no registry surgery is needed. The layers are sitting in the registry, healthy and pullable by digest. ## The mechanism — this is #590, and it does more damage than that ticket says Every affected workflow has: ```yaml concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true ... create_manifest: needs: [build_and_publish] # <-- the job that stitches :latest ``` `build_and_publish` pushes `:amd64` and `:arm64v8`. `create_manifest` then runs `docker manifest create ... :latest` from those tags. Because `cancel-in-progress: true` is set, a back-to-back push to master **cancels the in-flight run**. If the cancellation lands *after* the arch builds have pushed but *before* `create_manifest` executes, you get exactly the state above: **healthy arch tags, no `:latest`.** **#590** describes this concurrency setting killing *deploy* jobs. It also silently kills **manifest stitching** — which is a distinct and arguably worse failure, because a skipped deploy is a no-op, whereas a skipped `create_manifest` leaves `:latest` pointing at nothing and **breaks every future pull on every node**. #590 should be widened to cover this. ## Corroboration Two images on the original list have since **recovered on their own** — `spikersoft-gameserver` and `spikersoft-keycloak-events` now resolve `:latest` fine. Neither was repaired by hand. They simply got rebuilt by a workflow run that was *not* cancelled, and `create_manifest` completed. Same as `spikersoft-quiz-generation` earlier today. That is the mechanism confirming itself. Two entries are **not** part of this at all — `jetson-tx2-influxdb-grafana` and `sonar-scanner-datacenter` have **no tags whatsoever** (`:latest`, `:amd64`, `:arm64v8` all missing). They are third-party/externally-sourced images that were never pushed to this registry. They should be dropped from this ticket's scope. ## Fix **Cheap and immediate:** re-run each affected workflow. `create_manifest` will stitch `:latest` from the arch tags that are *already in the registry* — no rebuild of the layers is strictly required, so these runs should be fast. **Durable — fix the cause, or this recurs indefinitely:** address **#590**. Either drop `cancel-in-progress: true` on `master`, or (better) exclude the publish/manifest/deploy jobs from cancellation so that a superseded run still finishes stitching and shipping what it already built. Cancelling a *build* is fine; cancelling *after* a partial push is what corrupts the tag state. Related: **#548** (closed) — the `--amend` against mutable `:amd64`/`:arm64v8` tags. That fix never actually landed; **all 23 of 23 multi-arch workflows still use the mutable pattern**. Filed separately. It compounds this: because the arch tags are mutable, a cancelled run can leave `:amd64` and `:arm64v8` pointing at *different commits*, so a later `create_manifest` can stitch a mixed-commit `:latest` that looks healthy but is not.
spikerj commented 2026-07-14 22:36:55 +00:00
Author
Owner
Copy Link
Copy Source

QA Team — live swarm evidence, and this is materially worse than the ticket title suggests.

We swept every service on the swarm. The :latest tag problem is actively causing outages right now, not just a latent reschedule risk.

Confirmed from a swarm manager, with a control

ref result
spikersoft-coderunner:amd64 valid manifest returned
spikersoft-coderunner:latest manifest unknown
spikersoft-coderunner:arm64 manifest unknown
spikersoft-node-agent:latest manifest unknown
spikersoft-backend:latest manifest unknown

:amd64 resolves from the same client with the same credentials, so this is a genuine tag-specific 404 — not an auth artifact. CI appears to be publishing only :amd64: no :latest, and no arm64 image at all, while 7 of 10 nodes are aarch64.

It is already breaking things

  • spikersoft-coderunner is 3/4. Replica .1 has been bouncing between dreamstream1/4/6 for 15+ hours, never starting: Rejected: "No such image: git.spikersoft.com/spikerj/spikersoft-coderunner:latest". RestartPolicy: any, so it never gives up. (We tested and discarded an arch-mismatch theory — all 7 dreamstreams are aarch64 and coderunner runs fine on ds2/3/5.)
  • spikersoft-node-agent reports a green 7/7 but has ZERO coverage on dreamstream1, dreamstream4 and dreamstream6 — all Rejected: "No such image: ...node-agent:latest" (18h / 11h / 14h). Global-mode services only count nodes with placed tasks, so docker service ls shows 7/7 while 30% of the fleet is unmonitored and the dashboard is green. Filing separately.

The scale of the exposure

24 of 32 spikersoft-* services are deployed with a bare :latest tag and no digest pin. They are running only because each node still holds a stale locally-cached :latest from when the tag existed (~3 days ago). Nodes without that cache — ds1/4/6 — reject instantly.

Any node reboot, docker image prune, or reschedule onto a cold node turns into an unrecoverable outage. This is a loaded gun pointed at most of the platform, and the trigger is routine maintenance.

Corroborating signal from Jaeger

traefik error spans in the last hour include registry manifest 404s for spikersoft-artpipe-processor, spikersoft-coderunner/manifests/arm64, and spikersoft-backend itself (trace ids 249b8c91f1e4ed6cddde19c8521f274b, 28cbe31162c6e2b3358f94b5081e0221).

This is also the most likely mechanism behind the recurring "merged but never deployed" pattern (#553, #582, and #588 — which we just re-opened as #602 after confirming its fix never reached a running container). If :latest cannot be pulled, a merged fix simply never becomes a running process.

Suggested priority

P1. We would treat this as the highest-severity open item on the board: it is both an active partial outage (coderunner, node-agent) and the root cause of fixes silently not shipping.

Related: #596 (mutable arch tags / --amend), #548, #590 (cancelled deploy jobs), #602.

**QA Team** — live swarm evidence, and this is materially worse than the ticket title suggests. We swept every service on the swarm. The `:latest` tag problem is **actively causing outages right now**, not just a latent reschedule risk. ## Confirmed from a swarm manager, with a control | ref | result | |---|---| | `spikersoft-coderunner:amd64` | **valid manifest returned** | | `spikersoft-coderunner:latest` | `manifest unknown` | | `spikersoft-coderunner:arm64` | `manifest unknown` | | `spikersoft-node-agent:latest` | `manifest unknown` | | `spikersoft-backend:latest` | `manifest unknown` | `:amd64` resolves from the same client with the same credentials, so this is a genuine tag-specific 404 — **not** an auth artifact. CI appears to be publishing only `:amd64`: no `:latest`, and **no arm64 image at all**, while **7 of 10 nodes are aarch64**. ## It is already breaking things - **`spikersoft-coderunner` is 3/4.** Replica `.1` has been bouncing between dreamstream1/4/6 for **15+ hours**, never starting: `Rejected: "No such image: git.spikersoft.com/spikerj/spikersoft-coderunner:latest"`. `RestartPolicy: any`, so it never gives up. (We tested and *discarded* an arch-mismatch theory — all 7 dreamstreams are aarch64 and coderunner runs fine on ds2/3/5.) - **`spikersoft-node-agent` reports a green `7/7` but has ZERO coverage on dreamstream1, dreamstream4 and dreamstream6** — all `Rejected: "No such image: ...node-agent:latest"` (18h / 11h / 14h). Global-mode services only count nodes with placed tasks, so `docker service ls` shows `7/7` while **30% of the fleet is unmonitored and the dashboard is green.** Filing separately. ## The scale of the exposure **24 of 32 `spikersoft-*` services are deployed with a bare `:latest` tag and no digest pin.** They are running *only* because each node still holds a stale locally-cached `:latest` from when the tag existed (~3 days ago). Nodes without that cache — ds1/4/6 — reject instantly. Any node reboot, `docker image prune`, or reschedule onto a cold node turns into an unrecoverable outage. This is a loaded gun pointed at most of the platform, and the trigger is routine maintenance. ## Corroborating signal from Jaeger traefik error spans in the last hour include registry manifest 404s for `spikersoft-artpipe-processor`, `spikersoft-coderunner/manifests/arm64`, and **`spikersoft-backend`** itself (trace ids `249b8c91f1e4ed6cddde19c8521f274b`, `28cbe31162c6e2b3358f94b5081e0221`). This is also the most likely **mechanism** behind the recurring "merged but never deployed" pattern (#553, #582, and #588 — which we just re-opened as #602 after confirming its fix never reached a running container). If `:latest` cannot be pulled, a merged fix simply never becomes a running process. ## Suggested priority **P1.** We would treat this as the highest-severity open item on the board: it is both an active partial outage (coderunner, node-agent) and the root cause of fixes silently not shipping. Related: #596 (mutable arch tags / `--amend`), #548, #590 (cancelled deploy jobs), #602.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#587