[Bug][Infra][CI] ModelEnvImages rebuilds all 10 model images from scratch every push — a provenance LABEL above the expensive layers busts the cache (3h build, hits the runner cap) #877

New Issue

2026-07-27T18:50:42Z

spikerj commented

2026-07-27 18:50:42 +00:00

Problem

ModelEnvImages rebuilds every model env image from scratch on every push, because a provenance LABEL sits above the expensive layers and invalidates them.

docker/Dockerfile.model-env:43-45:

ARG ARTPIPE_SHA=unknown
LABEL com.spikersoft.artpipe.sha="${ARTPIPE_SHA}" \
      com.spikersoft.artpipe.models="${MODELS}"

ARTPIPE_SHA changes on every commit, so that layer's cache key changes on every commit — and every layer below it misses, including:

RUN ... venv_setup --no-weights (per-model venvs, cloned repos, nvcc CUDA extension builds)
RUN --mount=type=secret,id=hf_token ... venv_setup --weights-only (weight downloads, up to ~19 GB)

The careful layer split 20 lines further down is completely defeated. Its own comment states the intent that no longer holds:

# Copying ONLY these means edits elsewhere under src/ don't invalidate the expensive layers below.

Evidence — run 18629

Triggering commit dbbfb69 changed exactly two files: src/artpipe/models/sdxl_lightning.py and tests/test_concept_run_params.py. Neither is in any early COPY layer, so the venv/weight layers should have been pure cache hits.

They weren't. From the job log:

artpipe-base unchanged (file-850cd15bd484 already in registry) — skipping build → the base is not the invalidator.
importing cache manifest from ...artpipe-model-env-<img>:latest → the registry cache was found and imported, for all 10 images.
48 CACHED entries total — all of them layers above the LABEL.
[stage-0 7/11] RUN (venv setup) and [stage-0 8/11] RUN (weights) executed for all ten images. Never CACHED.

Result: 10 sequential from-scratch builds, ~3 hours, then the runner's default 3-hour job cap killed it mid-push on the tenth image (textto3d). 02:17:56 → 05:17:57.

Nine images built and pushed; textto3d did not, so the tier-3 cascade to spikersoft-backend never fired for it.

Not the cause (checked and ruled out)

Disk — flat at 1.5 TB free for the whole run, never near the 60 GB preflight floor.
The docker daemon — the context deadline exceeded / container does not exist errors at 05:17 are symptoms of the job being killed, not the cause.
Per-model build errors — the nvdiffrast compile failure (instantmesh), No module named 'torch' (sf3d texture_baker) and the 401 on the gated stabilityai/stable-fast-3d repo all recovered via the --no-build-isolation retry / are logged as warnings. They did not fail the build.

Reproduction (minimal, verified)

FROM busybox:latest
ARG SHA=unknown
LABEL test.sha="${SHA}"
RUN echo "EXPENSIVE LAYER RAN" > /expensive.txt

DOCKER_BUILDKIT=1 docker build --build-arg SHA=ccc .   # RUN executes
DOCKER_BUILDKIT=1 docker build --build-arg SHA=ccc .   # #5 CACHED    (same arg)
DOCKER_BUILDKIT=1 docker build --build-arg SHA=ddd .   # #5 DONE 0.2s (changed arg → re-runs)

Changing only the ARG consumed by the LABEL re-runs every layer beneath it.

Fix

Move ARG ARTPIPE_SHA + the LABEL to the end of the Dockerfile, below the expensive layers. Labels are image metadata — position affects only the cache, not the resulting image, and spikersoft-backend's tier-3 workflow reads them off the finished image exactly as before.

Layers 9-11 (COPY . /opt/art_pipe/, root venv, manifest smoke check) still rebuild each commit — that's correct and cheap (~30s), since the full source tree genuinely changed.

Impact

Routine pushes should drop from ~3 hours to minutes, since only the volatile tail rebuilds.
The 3-hour runner cap stops being a live constraint (worth still setting an explicit timeout-minutes, but as a backstop rather than a thing routinely hit).
Registry churn drops correspondingly — currently every push re-pushes ~10 multi-GB images that are byte-identical apart from the label.

Notes

Related to the #839 fragility work on this job (per-image failures are collected rather than fatal). This is a different failure mode: nothing errors, it's just silently doing 10x the work it should.

Filed by Opus 5 Agent.

## Problem `ModelEnvImages` rebuilds **every model env image from scratch on every push**, because a provenance `LABEL` sits above the expensive layers and invalidates them. `docker/Dockerfile.model-env:43-45`: ```dockerfile ARG ARTPIPE_SHA=unknown LABEL com.spikersoft.artpipe.sha="${ARTPIPE_SHA}" \ com.spikersoft.artpipe.models="${MODELS}" ``` `ARTPIPE_SHA` changes on every commit, so that layer's cache key changes on every commit — and **every layer below it misses**, including: - `RUN ... venv_setup --no-weights` (per-model venvs, cloned repos, nvcc CUDA extension builds) - `RUN --mount=type=secret,id=hf_token ... venv_setup --weights-only` (weight downloads, up to ~19 GB) The careful layer split 20 lines further down is completely defeated. Its own comment states the intent that no longer holds: > `# Copying ONLY these means edits elsewhere under src/ don't invalidate the expensive layers below.` ## Evidence — run 18629 Triggering commit `dbbfb69` changed exactly two files: `src/artpipe/models/sdxl_lightning.py` and `tests/test_concept_run_params.py`. **Neither is in any early COPY layer**, so the venv/weight layers should have been pure cache hits. They weren't. From the job log: - `artpipe-base unchanged (file-850cd15bd484 already in registry) — skipping build` → the base is not the invalidator. - `importing cache manifest from ...artpipe-model-env-<img>:latest` → the registry cache **was** found and imported, for all 10 images. - 48 `CACHED` entries total — all of them layers **above** the LABEL. - `[stage-0 7/11] RUN` (venv setup) and `[stage-0 8/11] RUN` (weights) **executed for all ten images**. Never `CACHED`. Result: 10 sequential from-scratch builds, ~3 hours, then the runner's default 3-hour job cap killed it mid-push on the tenth image (`textto3d`). 02:17:56 → 05:17:57. Nine images built and pushed; `textto3d` did not, so the tier-3 cascade to spikersoft-backend never fired for it. ### Not the cause (checked and ruled out) - **Disk** — flat at 1.5 TB free for the whole run, never near the 60 GB preflight floor. - **The docker daemon** — the `context deadline exceeded` / `container does not exist` errors at 05:17 are *symptoms of the job being killed*, not the cause. - **Per-model build errors** — the nvdiffrast compile failure (instantmesh), `No module named 'torch'` (sf3d texture_baker) and the `401` on the gated `stabilityai/stable-fast-3d` repo all **recovered** via the `--no-build-isolation` retry / are logged as warnings. They did not fail the build. ## Reproduction (minimal, verified) ```dockerfile FROM busybox:latest ARG SHA=unknown LABEL test.sha="${SHA}" RUN echo "EXPENSIVE LAYER RAN" > /expensive.txt ``` ``` DOCKER_BUILDKIT=1 docker build --build-arg SHA=ccc . # RUN executes DOCKER_BUILDKIT=1 docker build --build-arg SHA=ccc . # #5 CACHED (same arg) DOCKER_BUILDKIT=1 docker build --build-arg SHA=ddd . # #5 DONE 0.2s (changed arg → re-runs) ``` Changing only the ARG consumed by the `LABEL` re-runs every layer beneath it. ## Fix Move `ARG ARTPIPE_SHA` + the `LABEL` to the **end** of the Dockerfile, below the expensive layers. Labels are image metadata — position affects only the cache, not the resulting image, and spikersoft-backend's tier-3 workflow reads them off the finished image exactly as before. Layers 9-11 (`COPY . /opt/art_pipe/`, root venv, manifest smoke check) still rebuild each commit — that's correct and cheap (~30s), since the full source tree genuinely changed. ## Impact - Routine pushes should drop from ~3 hours to minutes, since only the volatile tail rebuilds. - The 3-hour runner cap stops being a live constraint (worth still setting an explicit `timeout-minutes`, but as a backstop rather than a thing routinely hit). - Registry churn drops correspondingly — currently every push re-pushes ~10 multi-GB images that are byte-identical apart from the label. ## Notes Related to the `#839` fragility work on this job (per-image failures are collected rather than fatal). This is a different failure mode: nothing errors, it's just silently doing 10x the work it should. Filed by Opus 5 Agent.

spikerj referenced this issue

2026-07-28 14:24:32 +00:00

artpipe-model-blender:latest missing from registry — service Reject-loops every ~5s since the 04:11Z artpipe redeploy #887

spikerj commented

2026-07-28 15:57:20 +00:00

More fallout 2026-07-28: the on-merge ModelEnvImages run for artpipe#44 (run 18849, 02:47–05:47Z) died at the ~3h runner cap again (act-runner context-deadline/container-archive errors) — it was rebuilding the whole default set and never published the NEW qrmonster env image, so the #535 QR lane silently had no images and the first user QR request today hung (no worker, no queue; the fire-and-forget publish was dropped unrouted). Recovered via the #839 escape hatch: dispatched model-env-images with images=qrmonster only (run 18932, in flight). The single-image dispatch path keeps working; the full-set on-push build keeps being the thing that breaks. — Opus 5 Agent

spikerj closed this issue

2026-07-29 01:30:52 +00:00

spikerj referenced this issue

2026-07-29 16:56:50 +00:00

artpipe env images: two RUN loops produce 17 GB and 16 GB single layers — split per model #858

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#877