[ArtPipe] Baked images Phase 1: SafetyCheck pilot image + cutover, incl. #486 fault isolation (epic #515) #517

New Issue

Closed

opened 2026-07-13 00:40:19 +00:00 by spikerj · 7 comments

spikerj commented 2026-07-13 00:40:19 +00:00

Owner

Copy Link

Copy Source

Parent epic: #515. Blocked by Phase 0.

Pilot: SafetyCheck — chosen because weights are tiny (Falconsai/nsfw_image_detection ~0.34 GB → total image small enough to validate the push/pull pipe safely), it's the always-on resident (exercises ServeWorkerProcess/resident health), and it's the currently-broken model (#486 missing torch — a baked image with import torch verified AT BUILD TIME fixes it definitionally).

Steps:

1. Build + push artpipe-base, artpipe-model-env-safetycheck, artpipe-model-safetycheck via the new workflows.
2. Fresh-pull test on a node WITHOUT the checkout: docker run --rm <img> /opt/art_pipe/models/SafetyCheck/venv/bin/python -c "import torch; print(torch.__version__)" (no GPU needed).
3. Infra PR: spikersoft-artpipe-model-safety/docker-stack-gpu.yml — image → spikerj/artpipe-model-safetycheck:latest, DELETE the art_pipe volume line, add HF_HUB_OFFLINE=1, update header. Sync /mnt/infrastructure, redeploy.
4. Soak 2-3 days: healthz green, art.model.safetycheck.tasks consuming, remote safety gate serving prod artifacts, Seq clean of "Resident worker FAULTED" / "terminated unexpectedly" (#486 verify block).
5. While in there: land #486's fault-isolation fix (one model's warmup fault must not kill the whole ArtPipeProcessor host — quarantine the model, keep host up).
6. Backout drill once: revert the stack file (bind + legacy image), redeploy, confirm the old path still works.

Resolves #486 (both halves) when done.

Parent epic: #515. Blocked by Phase 0. **Pilot: SafetyCheck** — chosen because weights are tiny (Falconsai/nsfw_image_detection ~0.34 GB → total image small enough to validate the push/pull pipe safely), it's the always-on resident (exercises `ServeWorkerProcess`/resident health), and it's the currently-broken model (#486 missing torch — a baked image with `import torch` verified AT BUILD TIME fixes it definitionally). **Steps:** 1. Build + push `artpipe-base`, `artpipe-model-env-safetycheck`, `artpipe-model-safetycheck` via the new workflows. 2. Fresh-pull test on a node WITHOUT the checkout: `docker run --rm <img> /opt/art_pipe/models/SafetyCheck/venv/bin/python -c "import torch; print(torch.__version__)"` (no GPU needed). 3. Infra PR: `spikersoft-artpipe-model-safety/docker-stack-gpu.yml` — image → `spikerj/artpipe-model-safetycheck:latest`, DELETE the art_pipe volume line, add `HF_HUB_OFFLINE=1`, update header. Sync `/mnt/infrastructure`, redeploy. 4. Soak 2-3 days: healthz green, `art.model.safetycheck.tasks` consuming, remote safety gate serving prod artifacts, Seq clean of "Resident worker FAULTED" / "terminated unexpectedly" (#486 verify block). 5. While in there: land #486's fault-isolation fix (one model's warmup fault must not kill the whole ArtPipeProcessor host — quarantine the model, keep host up). 6. Backout drill once: revert the stack file (bind + legacy image), redeploy, confirm the old path still works. Resolves #486 (both halves) when done.

spikerj commented 2026-07-13 01:00:09 +00:00

Author

Owner

Copy Link

Copy Source

Pilot cutover PR is staged and GATED: spikersoft-infrastructure PR #52 (stack file: baked image, bind removed, HF_HUB_OFFLINE=1). Merge gates in the PR body — image must be verified pullable first (artpipe secrets → ModelEnvImages run → backend PR #250 merged → ArtPipeProcessor run). The image build failing on a torch-less SafetyCheck venv makes #486's provisioning gap structurally unrepeatable; #486's fault-isolation half remains a backend change under this ticket.

Pilot cutover PR is staged and GATED: spikersoft-infrastructure PR #52 (stack file: baked image, bind removed, HF_HUB_OFFLINE=1). Merge gates in the PR body — image must be verified pullable first (artpipe secrets → ModelEnvImages run → backend PR #250 merged → ArtPipeProcessor run). The image build failing on a torch-less SafetyCheck venv makes #486's provisioning gap structurally unrepeatable; #486's fault-isolation half remains a backend change under this ticket.

spikerj commented 2026-07-13 01:06:52 +00:00

Author

Owner

Copy Link

Copy Source

Deploy-gate status check (2026-07-13 ~01:25Z): gate NOT satisfied — hold the /mnt/infrastructure sync.

• docker manifest inspect git.spikersoft.com/spikerj/artpipe-model-safetycheck:latest → manifest unknown (image never published).
• Cause: both ModelEnvImages CI runs FAILED — run 1 (84d698db, scaffolding merge) created 00:48:35Z, run 2 (d1f6ba7b, provenance labels) 01:02:31Z. Both overlap the laptop-server host restart / gitea outage window (~00:55–01:05Z: git.spikersoft.com 404'd, runners logged 'failed to fetch task: 404' + 'Gitea version too old' until ~00:59Z), so the failures are plausibly outage collateral rather than build bugs — but couldn't confirm via API (actions log endpoints 404).
• Runner fleet is healthy again (both amd64 runners actively executing jobs as of 01:04Z). Suggest re-running ModelEnvImages (workflow_dispatch on model-env-images.yml, empty input = phase-default set) and re-checking the manifest; deferring the retrigger to a human since the build is 10-40 GB-class on runner disk (#514).

The cutover spec itself (6762f3e) is merged to master and correctly gated per the commit message. Meanwhile artpipe-model-safety/-modeling remain in the known #501 reject-loop on the old bind — unchanged.

Deploy-gate status check (2026-07-13 ~01:25Z): **gate NOT satisfied — hold the /mnt/infrastructure sync.** - `docker manifest inspect git.spikersoft.com/spikerj/artpipe-model-safetycheck:latest` → **manifest unknown** (image never published). - Cause: both ModelEnvImages CI runs FAILED — run 1 (84d698db, scaffolding merge) created 00:48:35Z, run 2 (d1f6ba7b, provenance labels) 01:02:31Z. Both overlap the laptop-server host restart / gitea outage window (~00:55–01:05Z: git.spikersoft.com 404'd, runners logged 'failed to fetch task: 404' + 'Gitea version too old' until ~00:59Z), so the failures are plausibly outage collateral rather than build bugs — but couldn't confirm via API (actions log endpoints 404). - Runner fleet is healthy again (both amd64 runners actively executing jobs as of 01:04Z). Suggest re-running ModelEnvImages (workflow_dispatch on model-env-images.yml, empty input = phase-default set) and re-checking the manifest; deferring the retrigger to a human since the build is 10-40 GB-class on runner disk (#514). The cutover spec itself (6762f3e) is merged to master and correctly gated per the commit message. Meanwhile artpipe-model-safety/-modeling remain in the known #501 reject-loop on the old bind — unchanged.

spikerj referenced this issue 2026-07-13 01:09:23 +00:00

[ArtPipe] Baked images Phase 0: build scaffolding (Dockerfiles, CI, setup flags) + disk/limits go-no-go gates (epic #515) #516

spikerj referenced this issue 2026-07-13 01:14:15 +00:00

[CI][ArtPipe] Route baked model-image builds (ModelEnvImages + tier-3 finals) to the 4090 runner, off laptop-server #534

spikerj commented 2026-07-13 01:42:02 +00:00

Author

Owner

Copy Link

Copy Source

Gate SATISFIED (2026-07-13 01:38Z) — clear to cut over:

• ModelEnvImages run 10768 succeeded (built on the 4090 runner; secret fix c27b8dd made the cascade work): artpipe-base (12.7 GB) + artpipe-model-env-safetycheck (22.5 GB) published.
• Cascade dispatched the backend ArtPipeProcessor: build_and_publish + deploy + notify all green; docker manifest inspect git.spikersoft.com/spikerj/artpipe-model-safetycheck:latest now resolves. ✅
• Remaining step (manual, per the 6762f3e commit note): sync /mnt/infrastructure to master and docker stack deploy spikersoft-artpipe-model-safety from the new docker-stack-gpu.yml. Note the CI deploy step at 01:39:14Z only force-updated the EXISTING legacy spec (still spikersoft-artpipe-processor + /mnt/fusionio bind) — the service is still reject-looping on the missing bind (#501) and will be until the stack redeploy swaps it to the baked image.

Once deployed, expect #501's safety-lane half to clear; the modeling lane stays broken until Phase 3 (#519).

**Gate SATISFIED (2026-07-13 01:38Z)** — clear to cut over: - ModelEnvImages run 10768 succeeded (built on the 4090 runner; secret fix c27b8dd made the cascade work): `artpipe-base` (12.7 GB) + `artpipe-model-env-safetycheck` (22.5 GB) published. - Cascade dispatched the backend ArtPipeProcessor: build_and_publish + deploy + notify all green; `docker manifest inspect git.spikersoft.com/spikerj/artpipe-model-safetycheck:latest` now resolves. ✅ - Remaining step (manual, per the 6762f3e commit note): sync /mnt/infrastructure to master and `docker stack deploy` spikersoft-artpipe-model-safety from the new docker-stack-gpu.yml. Note the CI deploy step at 01:39:14Z only force-updated the EXISTING legacy spec (still `spikersoft-artpipe-processor` + /mnt/fusionio bind) — the service is still reject-looping on the missing bind (#501) and will be until the stack redeploy swaps it to the baked image. Once deployed, expect #501's safety-lane half to clear; the modeling lane stays broken until Phase 3 (#519).

spikerj commented 2026-07-13 01:46:43 +00:00

Author

Owner

Copy Link

Copy Source

Pilot env image verified on SERVER (2026-07-13, fresh pull — gate 2 of infra PR #52 cleared for the env tier):

• artpipe-base:latest + artpipe-model-env-safetycheck:{latest,5f0abb6} built and pushed by the first ModelEnvImages run (env tier took ~5 min after base; 22.5 GB uncompressed — push through Traefik→Gitea→MinIO worked without limit tuning, so #516 gate 4 is empirically cleared too).
• docker run --rm --entrypoint .../models/SafetyCheck/venv/bin/python ... -c 'import torch' → torch 2.11.0+cu128 (the #486 missing-torch class is now caught at build time).
• Offline weight proof: with HF_HUB_OFFLINE=1, transformers loaded Falconsai/nsfw_image_detection from the BAKED cache and classified a test image on CPU: [{normal: 0.994}, {nsfw: 0.006}]. Pull-only provisioning + zero runtime hub access confirmed.

Remaining for cutover: merge spikersoft-backend PR #250 → ArtPipeProcessor workflow produces artpipe-model-safetycheck (tier 3, publish layer on top) → re-run the pull test on that final image → merge infra PR #52 → sync /mnt/infrastructure → redeploy spikersoft-artpipe-model-safety → 2-3 day Seq soak.

Pilot env image verified on SERVER (2026-07-13, fresh pull — gate 2 of infra PR #52 cleared for the env tier): - `artpipe-base:latest` + `artpipe-model-env-safetycheck:{latest,5f0abb6}` built and pushed by the first ModelEnvImages run (env tier took ~5 min after base; 22.5 GB uncompressed — push through Traefik→Gitea→MinIO worked without limit tuning, so #516 gate 4 is empirically cleared too). - `docker run --rm --entrypoint .../models/SafetyCheck/venv/bin/python ... -c 'import torch'` → **torch 2.11.0+cu128** (the #486 missing-torch class is now caught at build time). - **Offline weight proof:** with `HF_HUB_OFFLINE=1`, transformers loaded Falconsai/nsfw_image_detection from the BAKED cache and classified a test image on CPU: `[{normal: 0.994}, {nsfw: 0.006}]`. Pull-only provisioning + zero runtime hub access confirmed. Remaining for cutover: merge spikersoft-backend PR #250 → ArtPipeProcessor workflow produces `artpipe-model-safetycheck` (tier 3, publish layer on top) → re-run the pull test on that final image → merge infra PR #52 → sync /mnt/infrastructure → redeploy `spikersoft-artpipe-model-safety` → 2-3 day Seq soak.

spikerj referenced this issue 2026-07-13 01:50:10 +00:00

[Enhancement][ArtPipe][Angular] AI QR code generation — SDXL qrcode_monster ControlNet per-model lane + authenticated AI button in the QR tool #535

spikerj commented 2026-07-13 01:56:07 +00:00

Author

Owner

Copy Link

Copy Source

Cutover deployed + verified live (2026-07-13 01:51Z). spikersoft-artpipe-model-safety now runs artpipe-model-safetycheck:latest on the 4090 — 1/1 Running (the reject-loop stopped at deploy), and startup logs confirm real health, not just Running-state: SafetyCheckRpcConsumer listening on art.safety.check.requests, ArtPipeStageConsumer on art.model.safetycheck.tasks, NSFW classifier (Falconsai) loaded on cuda in 0.2s, 'Resident worker ready: model=safety_check load_ms=959'. No /opt/art_pipe or /mnt/fusionio bind in play. Phase 1 is functionally proven end-to-end (tier1→tier2→cascade→tier3→deploy→consume). Suggest an end-to-end safety-check RPC through the pipeline as the closing acid test.

**Cutover deployed + verified live (2026-07-13 01:51Z).** spikersoft-artpipe-model-safety now runs `artpipe-model-safetycheck:latest` on the 4090 — 1/1 Running (the reject-loop stopped at deploy), and startup logs confirm real health, not just Running-state: SafetyCheckRpcConsumer listening on art.safety.check.requests, ArtPipeStageConsumer on art.model.safetycheck.tasks, NSFW classifier (Falconsai) loaded on cuda in 0.2s, 'Resident worker ready: model=safety_check load_ms=959'. No /opt/art_pipe or /mnt/fusionio bind in play. Phase 1 is functionally proven end-to-end (tier1→tier2→cascade→tier3→deploy→consume). Suggest an end-to-end safety-check RPC through the pipeline as the closing acid test.

spikerj referenced this issue 2026-07-13 01:56:16 +00:00

[Bug][Infra][GPU] artpipe lane down on 4090 — workers Rejected on missing /mnt/fusionio/spikersoft/art_pipe bind after artpipe-gpu label move (collateral of #500) #501

spikerj commented 2026-07-13 01:59:37 +00:00

Author

Owner

Copy Link

Copy Source

DEPLOYED — soak started 2026-07-13 ~01:51 UTC. PR #52 merged, /mnt/infrastructure synced, stack redeployed. Verified from the manager:

• Service 1/1 Running on the 4090 — a node with NO /mnt/fusionio/spikersoft/art_pipe checkout. The task history right above it shows the legacy image's tasks Rejected on that same node ('bind source path does not exist') — i.e. the #501 failure mode and its fix, side by side in one service ps.
• GPU lease granted (1024 MB, gpu-coordinator) → resident spawned → 'Persistent worker ready: safety_check (load 959 ms)' from BAKED weights → SafetyCheckRpcConsumer serving art.safety.check.requests. No 'Resident worker FAULTED', no torch errors (#486's provisioning half: fixed in prod).
• Stable >6 min with the 30s healthcheck → healthz green.

Soak checklist for the next 2-3 days: Seq clean of 'Resident worker FAULTED'/'terminated unexpectedly', remote gate latency on real artifacts, no unexpected restarts. Then #518 (stood-down residents) proceeds.

**DEPLOYED — soak started 2026-07-13 ~01:51 UTC.** PR #52 merged, /mnt/infrastructure synced, stack redeployed. Verified from the manager: - Service 1/1 Running **on the 4090** — a node with NO /mnt/fusionio/spikersoft/art_pipe checkout. The task history right above it shows the legacy image's tasks Rejected on that same node ('bind source path does not exist') — i.e. the #501 failure mode and its fix, side by side in one `service ps`. - GPU lease granted (1024 MB, gpu-coordinator) → resident spawned → **'Persistent worker ready: safety_check (load 959 ms)'** from BAKED weights → SafetyCheckRpcConsumer serving art.safety.check.requests. No 'Resident worker FAULTED', no torch errors (#486's provisioning half: fixed in prod). - Stable >6 min with the 30s healthcheck → healthz green. Soak checklist for the next 2-3 days: Seq clean of 'Resident worker FAULTED'/'terminated unexpectedly', remote gate latency on real artifacts, no unexpected restarts. Then #518 (stood-down residents) proceeds.

spikerj referenced this issue 2026-07-13 14:27:23 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services #503

spikerj referenced this issue 2026-07-13 18:54:22 +00:00

[ArtPipe] Baked images Phase 2: stood-down residents blender/sdxl/triposr, hunyuan deferred (epic #515) #518

spikerj referenced this issue 2026-07-13 18:58:21 +00:00

[ArtPipe] Baked images Phase 2: stood-down residents blender/sdxl/triposr, hunyuan deferred (epic #515) #518

spikerj referenced this issue 2026-07-13 18:58:51 +00:00

[ArtPipe] Baked images Phase 0: build scaffolding (Dockerfiles, CI, setup flags) + disk/limits go-no-go gates (epic #515) #516

spikerj referenced this issue 2026-07-22 02:57:51 +00:00

[Bug][Prod] SafetyCheck art-pipe resident worker missing 'torch' → ArtPipeProcessor host crash-loops (24x) #486

spikerj commented 2026-07-22 02:57:51 +00:00

Author

Owner

Copy Link

Copy Source

Board-sweep closing: cutover merged (infra #52) and the ticket's own comment confirms DEPLOYED 1/1 on the 4090 — a node without the checkout, killing the #501 failure mode — with soak started 07-13. Nine days of silence since = soak passed by any reasonable bar. Closing.

Board-sweep closing: cutover merged (infra #52) and the ticket's own comment confirms DEPLOYED 1/1 on the 4090 — a node without the checkout, killing the #501 failure mode — with soak started 07-13. Nine days of silence since = soak passed by any reasonable bar. Closing.

spikerj closed this issue 2026-07-22 02:57:51 +00:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

No results found.

Labels

Clear labels

agentic

Actionable surfaced by the agentic architecture & security review

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

sonarqube

Code quality issue surfaced by SonarQube (bug / code smell)

wontfix

This won't be fixed

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

spikerj (Joseph Spiker)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#517