artpipe: 33 GB of immutable weights/venvs are baked into every CI build — make them data, not image layers #859

New Issue

2026-07-26T19:31:01Z

spikerj commented

2026-07-26 19:31:01 +00:00

The complaint, and it is correct

The model weights never change. The thing that changes is a ~100 MB .NET
wrapper. Yet every backend merge touching SpikerSoft.Data, Common,
Contracts.Workers or EventHandlers.Infrastructure — i.e. most merges —
rebuilds and re-pushes 40-60 GB per model image, ten of them.

SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile.model is the whole story:

FROM ${ENV_IMAGE} AS final          # 40 GB, immutable
WORKDIR /app
COPY --from=publish /app/publish .  # ~100 MB, changes every merge
RUN mkdir -p /app/work && chown -R app:app /app

What the 40 GB actually is

Measured on artpipe-model-env-prodstages (29 layers, 40.0 GB):

content	size	changes when
model weights (Dockerfile.model-env line 83)	17.25 GB	never — they are upstream release artifacts
per-model python venvs, torch + CUDA (line 75)	15.76 GB	only when a dependency pins move
base + source + everything else	~7 GB	occasionally
the .NET wrapper	~0.1 GB	every merge

99.75% of the image is content that does not change with the thing being built.

Why layer caching does not already save us

It partly does — FROM ${ENV_IMAGE} means the tier-3 image reuses the env
image's layer digests, so a push should cross-repo-mount them rather than
re-upload. Two things defeat it:

The build needs the 40 GB base materialized locally. docker build on
the runner must have ENV_IMAGE present to layer onto it. #534 pins tier-2
and tier-3 to the same runner so it is usually already there — but any
tier-2 rebuild, tag change or cache eviction turns that into a 40 GB pull.
That is a cache miss with a 40 GB penalty, and it is what killed runs 18574
and 18580.
Any tier-2 rebuild invalidates all ten tier-3 images at once, so the
penalty is paid ten times over.

Proposal: weights are data, not image layers

docker-stack-gpu.yml records # NO /opt/art_pipe bind (#515/#519) — the pipeline tree is baked into the image, replacing an earlier /mnt/fusionio
bind. Baking fixed node-portability; the cost was putting 33 GB of immutable
content on the CI critical path.

The middle path keeps portability without baking — a named volume per GPU
node, not a shared filesystem:

Weights (17.25 GB) move to object storage (MinIO already holds art
artifacts). A one-shot per-node provisioning job populates a named volume from
MinIO; the runtime container mounts it at HF_HOME. Fetched once per node,
ever. HF_HUB_OFFLINE=1 still holds — the volume is provisioned, not
downloaded at runtime, so #455's failure mode stays loud.
Venvs (15.76 GB) are the same shape: rebuilt only when dependency pins
change, so they belong in a versioned volume too, keyed by a manifest hash.
The runtime image becomes base + .NET wrapper: order ~1 GB. CI builds and
pushes that in seconds, and a deploy is a normal image pull.

Staging: weights first (image 40 GB -> ~23 GB, and the never-changing part is
off the critical path), venvs second if the win justifies it.

Cheaper interim wins

#858 — split the two mega-layers so a cache miss costs a few GB per layer
instead of 17 GB, and layers pull concurrently.
#856 / PR #488 — direct registry transport, 14.9 -> 108.5 MB/s.
Do not rebuild all ten on every merge. A SpikerSoft.Data change does
require relinking the wrapper, but with a ~1 GB runtime image that becomes
cheap rather than something worth avoiding.

Neither interim fix addresses the root problem: 33 GB of immutable content is
being treated as a build output.

Acceptance

A wrapper-only change publishes an image measured in hundreds of MB.
Model weights transfer at most: upstream -> our storage once, our storage ->
each node once.
A tier-2 dependency bump does not force ten multi-GB rebuilds.

## The complaint, and it is correct The model weights never change. The thing that changes is a ~100 MB .NET wrapper. Yet every backend merge touching `SpikerSoft.Data`, `Common`, `Contracts.Workers` or `EventHandlers.Infrastructure` — i.e. most merges — rebuilds and re-pushes 40-60 GB **per model image**, ten of them. `SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile.model` is the whole story: ```dockerfile FROM ${ENV_IMAGE} AS final # 40 GB, immutable WORKDIR /app COPY --from=publish /app/publish . # ~100 MB, changes every merge RUN mkdir -p /app/work && chown -R app:app /app ``` ## What the 40 GB actually is Measured on `artpipe-model-env-prodstages` (29 layers, 40.0 GB): | content | size | changes when | |---|---|---| | model weights (Dockerfile.model-env line 83) | **17.25 GB** | never — they are upstream release artifacts | | per-model python venvs, torch + CUDA (line 75) | **15.76 GB** | only when a dependency pins move | | base + source + everything else | ~7 GB | occasionally | | **the .NET wrapper** | **~0.1 GB** | **every merge** | 99.75% of the image is content that does not change with the thing being built. ## Why layer caching does not already save us It partly does — `FROM ${ENV_IMAGE}` means the tier-3 image reuses the env image's layer digests, so a push should cross-repo-mount them rather than re-upload. Two things defeat it: 1. **The build needs the 40 GB base materialized locally.** `docker build` on the runner must have `ENV_IMAGE` present to layer onto it. #534 pins tier-2 and tier-3 to the same runner so it is usually already there — but any tier-2 rebuild, tag change or cache eviction turns that into a 40 GB pull. That is a cache miss with a 40 GB penalty, and it is what killed runs 18574 and 18580. 2. **Any tier-2 rebuild invalidates all ten tier-3 images at once**, so the penalty is paid ten times over. ## Proposal: weights are data, not image layers `docker-stack-gpu.yml` records `# NO /opt/art_pipe bind (#515/#519) — the pipeline tree is baked into the image`, replacing an earlier `/mnt/fusionio` bind. Baking fixed node-portability; the cost was putting 33 GB of immutable content on the CI critical path. The middle path keeps portability without baking — a **named volume per GPU node**, not a shared filesystem: - **Weights (17.25 GB)** move to object storage (MinIO already holds art artifacts). A one-shot per-node provisioning job populates a named volume from MinIO; the runtime container mounts it at `HF_HOME`. Fetched once per node, ever. `HF_HUB_OFFLINE=1` still holds — the volume is provisioned, not downloaded at runtime, so #455's failure mode stays loud. - **Venvs (15.76 GB)** are the same shape: rebuilt only when dependency pins change, so they belong in a versioned volume too, keyed by a manifest hash. - **The runtime image** becomes base + .NET wrapper: order ~1 GB. CI builds and pushes that in seconds, and a deploy is a normal image pull. Staging: weights first (image 40 GB -> ~23 GB, and the never-changing part is off the critical path), venvs second if the win justifies it. ## Cheaper interim wins - **#858** — split the two mega-layers so a cache miss costs a few GB per layer instead of 17 GB, and layers pull concurrently. - **#856 / PR #488** — direct registry transport, 14.9 -> 108.5 MB/s. - **Do not rebuild all ten on every merge.** A `SpikerSoft.Data` change does require relinking the wrapper, but with a ~1 GB runtime image that becomes cheap rather than something worth avoiding. Neither interim fix addresses the root problem: 33 GB of immutable content is being treated as a build output. ## Acceptance - A wrapper-only change publishes an image measured in hundreds of MB. - Model weights transfer at most: upstream -> our storage once, our storage -> each node once. - A tier-2 dependency bump does not force ten multi-GB rebuilds.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#859