spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[ArtPipe] Baked images Phase 0: build scaffolding (Dockerfiles, CI, setup flags) + disk/limits go-no-go gates (epic #515) #516

New Issue
Closed
opened 2026-07-13 00:40:18 +00:00 by spikerj · 5 comments
spikerj commented 2026-07-13 00:40:18 +00:00
Owner
Copy Link
Copy Source

Parent epic: #515. No-prod-impact scaffolding.

spikersoft-artpipe:

  • src/artpipe/venv_setup.py: add --no-weights / --weights-only to the setup CLI (thread weights: str = "both" through setup_model_venv) for clean Docker layer separation. Local dev/bootstrap.py behavior unchanged.
  • .dockerignore: .git, models/*/venv, .venv, .deps, .cache, output/, logs/, weight globs.
  • docker/Dockerfile.base: FROM nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04 (devel, not runtime — CUDA exts need nvcc at build; nvdiffrast JITs at first use), apt block from the existing processor Dockerfile (blender + GL/X, git, build tools), deadsnakes py3.10 AND py3.11, .NET 10 runtime (aspnetcore), app user (#495), tiny /opt/setup-venv with huggingface_hub for outer-interpreter weight downloads.
  • docker/Dockerfile.model-env: ONE parameterized file, ARG MODELS. Layers: (1) setup-code+manifests only → (2) venvs via setup --model $m --no-weights (nvcc, NO GPU needed) → (3) weights via --weights-only with BuildKit --mount=type=secret,id=hf_token → (4) full source COPY → root .venv for env-less manifests (Blender). Build as app user. Keep every layer <~8 GB (split Hunyuan snapshot downloads if needed).
  • .gitea/workflows/model-env-images.yml: push-to-main paths filter (src/, models/, docker/**, pyproject.toml) + workflow_dispatch(models); sequential builds, df -h preflight + docker builder prune between models (#514); cascade-dispatch the backend workflow on success.
  • Repo secrets to provision: DOCKER_PASSWORD, HF_TOKEN (deferrable — no Phase 1-3 model is gated), GITEA_PAT.

spikersoft-backend:

  • SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile.model: ARG ENV_IMAGE / FROM ${ENV_IMAGE} AS final, copy dotnet publish output, ENTRYPOINT. Existing Dockerfile untouched (backout path).
  • .gitea/workflows/spikersoft-artpipe-processor.yml: publish once, loop the per-model finals with --build-arg ENV_IMAGE=..., keep the #511 manifest gate, keep building legacy :latest through Phase 3.

Gates before Phase 1:

  • df -h /mnt/fusionio + mc du on gitea-packages — budget 80-150 GB of new blobs or STOP
  • Verify BuildKit --secret works on the laptop-server runner (Engine 24.0.2, DOCKER_BUILDKIT=1)
  • Verify Traefik/Gitea large-upload limits (LIMIT_SIZE_CONTAINER, responding timeouts) for multi-GB blob pushes
  • Gitea package cleanup rules keep-last-3 for artpipe-* images
Parent epic: #515. No-prod-impact scaffolding. **spikersoft-artpipe:** - `src/artpipe/venv_setup.py`: add `--no-weights` / `--weights-only` to the `setup` CLI (thread `weights: str = "both"` through `setup_model_venv`) for clean Docker layer separation. Local dev/bootstrap.py behavior unchanged. - `.dockerignore`: .git, models/*/venv, .venv, .deps, .cache, output/, logs/, weight globs. - `docker/Dockerfile.base`: FROM nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04 (devel, not runtime — CUDA exts need nvcc at build; nvdiffrast JITs at first use), apt block from the existing processor Dockerfile (blender + GL/X, git, build tools), deadsnakes py3.10 AND py3.11, .NET 10 **runtime** (aspnetcore), `app` user (#495), tiny /opt/setup-venv with huggingface_hub for outer-interpreter weight downloads. - `docker/Dockerfile.model-env`: ONE parameterized file, `ARG MODELS`. Layers: (1) setup-code+manifests only → (2) venvs via `setup --model $m --no-weights` (nvcc, NO GPU needed) → (3) weights via `--weights-only` with BuildKit `--mount=type=secret,id=hf_token` → (4) full source COPY → root `.venv` for env-less manifests (Blender). Build as `app` user. Keep every layer <~8 GB (split Hunyuan snapshot downloads if needed). - `.gitea/workflows/model-env-images.yml`: push-to-main paths filter (src/**, models/**, docker/**, pyproject.toml) + `workflow_dispatch(models)`; sequential builds, `df -h` preflight + `docker builder prune` between models (#514); cascade-dispatch the backend workflow on success. - Repo secrets to provision: `DOCKER_PASSWORD`, `HF_TOKEN` (deferrable — no Phase 1-3 model is gated), `GITEA_PAT`. **spikersoft-backend:** - `SpikerSoft.EventHandlers.ArtPipeProcessor/Dockerfile.model`: `ARG ENV_IMAGE` / `FROM ${ENV_IMAGE} AS final`, copy dotnet publish output, ENTRYPOINT. Existing Dockerfile untouched (backout path). - `.gitea/workflows/spikersoft-artpipe-processor.yml`: publish once, loop the per-model finals with `--build-arg ENV_IMAGE=...`, keep the #511 manifest gate, keep building legacy `:latest` through Phase 3. **Gates before Phase 1:** - [ ] `df -h /mnt/fusionio` + `mc du` on `gitea-packages` — budget 80-150 GB of new blobs or STOP - [ ] Verify BuildKit `--secret` works on the laptop-server runner (Engine 24.0.2, DOCKER_BUILDKIT=1) - [ ] Verify Traefik/Gitea large-upload limits (`LIMIT_SIZE_CONTAINER`, responding timeouts) for multi-GB blob pushes - [ ] Gitea package cleanup rules keep-last-3 for artpipe-* images
spikerj commented 2026-07-13 00:59:55 +00:00
Author
Owner
Copy Link
Copy Source

Phase 0 scaffolding landed/opened (2026-07-12):

  • spikersoft-artpipe PR #11 (MERGED): setup --no-weights/--weights-only flags, .dockerignore, docker/Dockerfile.base + Dockerfile.model-env, .gitea/workflows/model-env-images.yml. Follow-up PR #12 (open): ARTPIPE_SHA provenance labels.
  • spikersoft-backend PR #250 (open): Dockerfile.model + per-model finals loop in the ArtPipeProcessor workflow (legacy image kept as backout path).
  • spikersoft-infrastructure PR #51 (open): skopeo mirror workflow + docs notes (the #521 half riding in this phase).

Remaining Phase 0 gates — MANUAL (no SSH access from this session):

  1. Disk go/no-go on SERVER: df -h /mnt/fusionio and mc du spiker/gitea-packages (or the console) — budget 80-150 GB of new registry blobs across all phases, ~5-8 GB for the SafetyCheck pilot alone. Record the numbers here.
  2. Provision spikersoft-artpipe repo Actions secrets: DOCKER_PASSWORD (registry push), GITEA_PAT (cascade dispatch); HF_TOKEN optional (no Phase 1-3 model is gated).
  3. Verify BuildKit works on the laptop-server runner (DOCKER_BUILDKIT=1 build with --secret; Engine 24.0.2 ships BuildKit but confirm).
  4. Verify Gitea/Traefik large-upload limits for multi-GB blob pushes (gitea app.ini LIMIT_SIZE_CONTAINER unset, Traefik responding timeouts) — the SafetyCheck pilot push is itself the practical test.
  5. Gitea package cleanup rules: keep-last-3 on artpipe-base / artpipe-model-env-* / artpipe-model-*.

Note: git.spikersoft.com returned 404 fleet-wide for ~3.5 min at ~20:55 UTC today (gitea container on laptop-server; Traefik+MinIO stayed healthy). Recovered on its own — flagging in case it correlates with the #498 packages→MinIO migration.

Phase 0 scaffolding landed/opened (2026-07-12): - spikersoft-artpipe PR #11 (MERGED): setup --no-weights/--weights-only flags, .dockerignore, docker/Dockerfile.base + Dockerfile.model-env, .gitea/workflows/model-env-images.yml. Follow-up PR #12 (open): ARTPIPE_SHA provenance labels. - spikersoft-backend PR #250 (open): Dockerfile.model + per-model finals loop in the ArtPipeProcessor workflow (legacy image kept as backout path). - spikersoft-infrastructure PR #51 (open): skopeo mirror workflow + docs notes (the #521 half riding in this phase). **Remaining Phase 0 gates — MANUAL (no SSH access from this session):** 1. Disk go/no-go on SERVER: `df -h /mnt/fusionio` and `mc du spiker/gitea-packages` (or the console) — budget 80-150 GB of new registry blobs across all phases, ~5-8 GB for the SafetyCheck pilot alone. Record the numbers here. 2. Provision spikersoft-artpipe repo Actions secrets: `DOCKER_PASSWORD` (registry push), `GITEA_PAT` (cascade dispatch); `HF_TOKEN` optional (no Phase 1-3 model is gated). 3. Verify BuildKit works on the laptop-server runner (DOCKER_BUILDKIT=1 build with --secret; Engine 24.0.2 ships BuildKit but confirm). 4. Verify Gitea/Traefik large-upload limits for multi-GB blob pushes (gitea app.ini LIMIT_SIZE_CONTAINER unset, Traefik responding timeouts) — the SafetyCheck pilot push is itself the practical test. 5. Gitea package cleanup rules: keep-last-3 on artpipe-base / artpipe-model-env-* / artpipe-model-*. Note: git.spikersoft.com returned 404 fleet-wide for ~3.5 min at ~20:55 UTC today (gitea container on laptop-server; Traefik+MinIO stayed healthy). Recovered on its own — flagging in case it correlates with the #498 packages→MinIO migration.
spikerj commented 2026-07-13 01:09:23 +00:00
Author
Owner
Copy Link
Copy Source

Build-chain status (2026-07-13 ~01:35Z): tier 3 is red because tier 2 never published.

  • backend ArtPipeProcessor run for the tier-3 merge (79482d8f, 'Dockerfile.model + CI finals loop') FAILED at 01:03:24Z, deploy+notify correctly skipped.
  • Deterministic cause, not outage flake: Dockerfile.model:44 = FROM ${ENV_IMAGE}, workflow sets ENV_IMAGE=git.spikersoft.com/spikerj/artpipe-model-env-${IMG}:latest — and no artpipe-model-env-* image exists in the registry because both artpipe ModelEnvImages runs failed (00:48Z/01:02Z, those DO look like laptop-server-outage collateral; runners healthy again since ~01:04Z — see my #517 comment).
  • Unblock order: re-run ModelEnvImages in spikersoft-artpipe first (its success also cascade-dispatches tier 3), then #517's gate (artpipe-model-safetycheck:latest pullable) can be re-checked before the /mnt/infrastructure sync.
Build-chain status (2026-07-13 ~01:35Z): **tier 3 is red because tier 2 never published.** - backend ArtPipeProcessor run for the tier-3 merge (79482d8f, 'Dockerfile.model + CI finals loop') FAILED at 01:03:24Z, deploy+notify correctly skipped. - Deterministic cause, not outage flake: `Dockerfile.model:44` = `FROM ${ENV_IMAGE}`, workflow sets `ENV_IMAGE=git.spikersoft.com/spikerj/artpipe-model-env-${IMG}:latest` — and no `artpipe-model-env-*` image exists in the registry because both artpipe ModelEnvImages runs failed (00:48Z/01:02Z, those DO look like laptop-server-outage collateral; runners healthy again since ~01:04Z — see my #517 comment). - Unblock order: re-run **ModelEnvImages** in spikersoft-artpipe first (its success also cascade-dispatches tier 3), then #517's gate (`artpipe-model-safetycheck:latest` pullable) can be re-checked before the /mnt/infrastructure sync.
spikerj commented 2026-07-13 01:20:57 +00:00
Author
Owner
Copy Link
Copy Source

Disk go/no-go: GO (measured on SERVER 2026-07-12):

/dev/fioa  2.9T total, 81G used, 2.7T avail (3%) on /mnt/fusionio
/mnt/fusionio/minio (MinIO data root incl. gitea-packages): 52G

The full fully-baked budget (80-150 GB images + ~20 GB mirror tier) fits ~15x over. Gate 1 cleared; remaining gates: artpipe repo secrets (DOCKER_PASSWORD/GITEA_PAT — the secrets API doesn't expose a list, so unverified from here), BuildKit sanity on the laptop-server runner (the pilot build is the practical test), package cleanup rules.

Disk go/no-go: **GO** (measured on SERVER 2026-07-12): /dev/fioa 2.9T total, 81G used, 2.7T avail (3%) on /mnt/fusionio /mnt/fusionio/minio (MinIO data root incl. gitea-packages): 52G The full fully-baked budget (80-150 GB images + ~20 GB mirror tier) fits ~15x over. Gate 1 cleared; remaining gates: artpipe repo secrets (DOCKER_PASSWORD/GITEA_PAT — the secrets API doesn't expose a list, so unverified from here), BuildKit sanity on the laptop-server runner (the pilot build is the practical test), package cleanup rules.
spikerj commented 2026-07-13 18:58:51 +00:00
Author
Owner
Copy Link
Copy Source

QA watch 2026-07-13 ~18:58Z — RECOMMEND CLOSE: scaffolding proven end-to-end TWICE. (1) Phase 1: safetycheck env → cascade → final → deployed 1/1 on the 4090, consumers verified (#517, 07-13 ~02:40Z). (2) Phase 2 today: ModelEnvImages 10821 green on the 4090 → all 3 env images published (incl. the 65 GB sdxl push) → cascade auto-dispatched backend finals 10822 → all four artpipe-model-* finals published 18:53Z tagged 14556a4-746b1b6. Dockerfile.base/model-env/model + the CI finals loop + cascade secret all demonstrably work. Residual issues tracked separately: #537 image bloat, #536 concurrency guard, #534 finals routing, #518/#519 gated cutovers. (Leaving the close itself to the team.)

QA watch 2026-07-13 ~18:58Z — RECOMMEND CLOSE: scaffolding proven end-to-end TWICE. (1) Phase 1: safetycheck env → cascade → final → deployed 1/1 on the 4090, consumers verified (#517, 07-13 ~02:40Z). (2) Phase 2 today: ModelEnvImages 10821 green on the 4090 → all 3 env images published (incl. the 65 GB sdxl push) → cascade auto-dispatched backend finals 10822 → all four artpipe-model-* finals published 18:53Z tagged 14556a4-746b1b6. Dockerfile.base/model-env/model + the CI finals loop + cascade secret all demonstrably work. Residual issues tracked separately: #537 image bloat, #536 concurrency guard, #534 finals routing, #518/#519 gated cutovers. (Leaving the close itself to the team.)
spikerj commented 2026-07-22 02:57:50 +00:00
Author
Owner
Copy Link
Copy Source

Board-sweep verified complete: Dockerfile.model in tree (PR #250) and the ticket's last comment is a QA RECOMMEND CLOSE — scaffolding proven end-to-end twice (Phase 1 + Phase 2 cascade green). Applying the close.

Board-sweep verified complete: Dockerfile.model in tree (PR #250) and the ticket's last comment is a QA RECOMMEND CLOSE — scaffolding proven end-to-end twice (Phase 1 + Phase 2 cascade green). Applying the close.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#516