artpipe env images: two RUN loops produce 17 GB and 16 GB single layers — split per model #858

New Issue

2026-07-26T19:26:14Z

spikerj commented

2026-07-26 19:26:14 +00:00

Summary

artpipe-model-env-prodstages is 40 GB across 29 layers, but two layers are
33 GB of it:

layer	size
`sha256:2ddb2ab3de92…`	17.25 GB
`sha256:cb2f71b053b7…`	15.76 GB
everything else (27 layers)	~7 GB combined

Docker cannot resume mid-layer, so any reset during a 17 GB blob restarts it
from zero. That made the env pull unable to converge whenever throughput was
degraded — the mechanism behind runs 18574 and 18580 (see #856).

Cause

docker/Dockerfile.model-env (spikersoft-artpipe) builds both in a single
RUN that loops over every model:

# line 75 — Layer 2: one layer for ALL model venvs  (~15.76 GB)
RUN for m in ${MODELS}; do … --model "${m}" --no-weights …; done

# line 83 — Layer 3: one layer for ALL model weights (~17.25 GB)
RUN --mount=type=secret,id=hf_token,mode=0444 \
    for m in ${MODELS}; do … --model "${m}" --weights-only …; done

MODELS is a space-separated build ARG, so the loop is inside one layer by
construction.

Why fix it even though #856 makes the pull fast

#856's direct-registry route takes the full image from ~46 min to ~6 min, which
removes the urgency but not the fragility:

One reset still costs 17 GB rather than a couple of GB.
Docker pulls layers concurrently (3 by default); two layers that are 83% of
the image serialize almost the whole transfer.
Cache reuse is all-or-nothing — re-baking one model's weights invalidates
every model's weights.

Approach

One RUN per model, so each model's venv and weights are their own layers.
MODELS is variable-length, so this needs either a generated Dockerfile or a
fixed set of MODEL_n ARGs with empty slots as no-ops.

Deliberately not attempted blind. Building a 60 GB image cannot be validated
locally, the env pipeline is already the most failure-prone part of CI, and
#856's transport fix has removed the pressure. This should be done with a real
build behind it.

Acceptance

No single layer above ~4 GB in artpipe-model-env-prodstages.
A full tier-2 env build succeeds and tier-3 still layers on top of it.
Re-baking one model does not invalidate the other models' layers.

## Summary `artpipe-model-env-prodstages` is 40 GB across 29 layers, but **two layers are 33 GB of it**: | layer | size | |---|---| | `sha256:2ddb2ab3de92…` | **17.25 GB** | | `sha256:cb2f71b053b7…` | **15.76 GB** | | everything else (27 layers) | ~7 GB combined | Docker cannot resume mid-layer, so any reset during a 17 GB blob restarts it from zero. That made the env pull unable to converge whenever throughput was degraded — the mechanism behind runs 18574 and 18580 (see #856). ## Cause `docker/Dockerfile.model-env` (spikersoft-artpipe) builds both in a single `RUN` that loops over every model: ```dockerfile # line 75 — Layer 2: one layer for ALL model venvs (~15.76 GB) RUN for m in ${MODELS}; do … --model "${m}" --no-weights …; done # line 83 — Layer 3: one layer for ALL model weights (~17.25 GB) RUN --mount=type=secret,id=hf_token,mode=0444 \ for m in ${MODELS}; do … --model "${m}" --weights-only …; done ``` `MODELS` is a space-separated build ARG, so the loop is inside one layer by construction. ## Why fix it even though #856 makes the pull fast #856's direct-registry route takes the full image from ~46 min to ~6 min, which removes the *urgency* but not the fragility: - One reset still costs 17 GB rather than a couple of GB. - Docker pulls layers concurrently (3 by default); two layers that are 83% of the image serialize almost the whole transfer. - Cache reuse is all-or-nothing — re-baking one model's weights invalidates every model's weights. ## Approach One `RUN` per model, so each model's venv and weights are their own layers. `MODELS` is variable-length, so this needs either a generated Dockerfile or a fixed set of `MODEL_n` ARGs with empty slots as no-ops. **Deliberately not attempted blind.** Building a 60 GB image cannot be validated locally, the env pipeline is already the most failure-prone part of CI, and #856's transport fix has removed the pressure. This should be done with a real build behind it. ## Acceptance - No single layer above ~4 GB in `artpipe-model-env-prodstages`. - A full tier-2 env build succeeds and tier-3 still layers on top of it. - Re-baking one model does not invalidate the other models' layers.

spikerj referenced this issue

2026-07-26 19:31:01 +00:00

artpipe: 33 GB of immutable weights/venvs are baked into every CI build — make them data, not image layers #859

spikerj commented

2026-07-29 16:56:50 +00:00

Re-verified at file level against artpipe origin/main (tip 2053880) — NOT DONE. Both mega-layer loops are structurally unchanged:

docker/Dockerfile.model-env:74-78 — Layer 2: RUN for m in ${MODELS}; do … --model "${m}" --no-weights || exit 1; done (the ~15.76 GB layer)
:83-88 — Layer 3: RUN --mount=type=secret,id=hf_token … for m in ${MODELS}; do … --weights-only || exit 1; done (the ~17.25 GB layer)

MODELS is still a single space-separated ARG at :43, so the loop stays inside one layer by construction — that's the mechanism, and it hasn't moved.

The only commit touching this file since the ticket was filed is 308f4a8 ("move provenance LABELs below the expensive layers", #877), which reorders around those layers — incidentally confirming they still exist. No generated Dockerfile, no fixed MODEL_n ARG slots, no per-model splitting. Consistent with the ticket's own "deliberately not attempted blind" note.

All three acceptance criteria remain unmet (no layer >~4 GB; full tier-2 build succeeds; re-baking one model doesn't invalidate the others).

Worth connecting to two things that make this more than a build-time annoyance:

#821 — a 4.8 GB weights layer was truncated in MinIO (unexpected EOF), and per its 07-23 comment three giant layers were corrupt, all from env run 17775: env-safetycheck 5.18 GB plus env-prodstages at 15.75 GB and 17.25 GB. Those last two figures match these two loops exactly. Splitting them wouldn't prevent corruption, but it would shrink the blast radius of any single bad blob and make a re-push cheap rather than a 17 GB retry.
#775 — blob-integrity hardening is entirely unimplemented, and #700 shows large pushes can restart MinIO and take Gitea down. Multi-GB single layers are the thing stressing that path.

So this sits at the intersection of three open infra tickets, and is the one of the four that's purely a Dockerfile change under our control.

Re-verified at file level against artpipe `origin/main` (tip `2053880`) — **NOT DONE.** Both mega-layer loops are structurally unchanged: - **`docker/Dockerfile.model-env:74-78`** — Layer 2: `RUN for m in ${MODELS}; do … --model "${m}" --no-weights || exit 1; done` (the ~15.76 GB layer) - **`:83-88`** — Layer 3: `RUN --mount=type=secret,id=hf_token … for m in ${MODELS}; do … --weights-only || exit 1; done` (the ~17.25 GB layer) `MODELS` is still a single space-separated `ARG` at **`:43`**, so the loop stays inside one layer by construction — that's the mechanism, and it hasn't moved. The only commit touching this file since the ticket was filed is `308f4a8` ("move provenance LABELs below the expensive layers", #877), which **reorders around** those layers — incidentally confirming they still exist. No generated Dockerfile, no fixed `MODEL_n` ARG slots, no per-model splitting. Consistent with the ticket's own "deliberately not attempted blind" note. All three acceptance criteria remain unmet (no layer >~4 GB; full tier-2 build succeeds; re-baking one model doesn't invalidate the others). **Worth connecting to two things that make this more than a build-time annoyance:** - **#821** — a 4.8 GB weights layer was truncated in MinIO (`unexpected EOF`), and per its 07-23 comment **three** giant layers were corrupt, all from env run 17775: env-safetycheck 5.18 GB plus env-prodstages at 15.75 GB and 17.25 GB. Those last two figures match these two loops exactly. Splitting them wouldn't prevent corruption, but it would shrink the blast radius of any single bad blob and make a re-push cheap rather than a 17 GB retry. - **#775** — blob-integrity hardening is entirely unimplemented, and **#700** shows large pushes can restart MinIO and take Gitea down. Multi-GB single layers are the thing stressing that path. So this sits at the intersection of three open infra tickets, and is the one of the four that's purely a Dockerfile change under our control.

spikerj referenced this issue

2026-07-29 18:26:03 +00:00

[Epic][ArtPipe][Infra] Per-model baked Docker images — venvs + weights built at image build time, no /opt/art_pipe bind #515

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#858