[Bug][Prod] SafetyCheck art-pipe resident worker missing 'torch' → ArtPipeProcessor host crash-loops (24x) #486

New Issue

2026-07-12T00:26:57Z

spikerj commented

2026-07-12 00:26:57 +00:00

Summary

The SafetyCheck resident art-pipe worker fails to start with ModuleNotFoundError: No module named 'torch', and because a warmup fault is treated as a non-recoverable "pre-ready config error," it takes down the entire ArtPipeProcessor host, which then crash-loops (observed 24× Fatal: terminated unexpectedly + 23× Hosting failed to start in a 24h window). This also produces a burst of TaskCanceledExceptions as the host is torn down mid-startup.

Evidence (Seq)

[Error] Resident worker FAULTED (no automatic restart — pre-ready failures are config errors):
        Worker for model 'SafetyCheck' failed during startup: Traceback (most recent call last):
  File "/opt/art_pipe/src/artpipe/worker.py", line 402, in _serve_main
    backend.warmup(on_progress=lambda p, m, **kw: log.info(
  File "/opt/art_pipe/src/artpipe/models/safety_check.py", line 155, in warmup
    self._ensure_model(on_progress)
  File "/opt/art_pipe/src/artpipe/models/safety_check.py", line 130, in _ensure_model
    import torch
ModuleNotFoundError: No module named 'torch'

Source: SpikerSoft.EventHandlers.ArtPipeProcessor.Services.ResidentArtPipeStageExecutor
Followed by [Fatal] SpikerSoft.EventHandlers.ArtPipeProcessor terminated unexpectedly + [Error] Hosting failed to start.

Two problems here

Provisioning gap: the SafetyCheck model's resident venv has no torch. This is the same family as #432 (art_pipe model venvs missing requirements) but a different model + different missing package + the Resident executor path — so #432's fixes (install cloned-repo requirements, python3.10) don't obviously cover it. safety_check.py:_ensure_model imports torch directly, so its venv (or the worker base image) must provide torch. Not a CUDA-version issue like #426 — torch is entirely absent.
Reliability / blast radius: a single model's pre-ready fault crashes the whole ArtPipeProcessor host and it crash-loops with no isolation. One misprovisioned model should not take down the host / all other resident models. Consider quarantining a faulted model (mark it unavailable, keep the host up) instead of terminating the process.

Impact

The SafetyCheck stage — the content-safety gate — cannot run. Depending on pipeline wiring this either blocks assets or lets them through ungated.
ArtPipeProcessor crash-loops, disrupting any resident-model stages it hosts.

Proposed fix

Ensure torch is installed in the SafetyCheck resident venv / worker image (align with the cu128 torch index used elsewhere per #426). Verify python -c "import torch" succeeds in that venv during provisioning.
Add fault isolation so a single model warmup failure degrades that model only, not the whole host (log + mark unavailable, don't terminate).

Verify

# in the SafetyCheck worker env
python -c "import torch; print(torch.__version__)"
# Seq: no new 'Resident worker FAULTED' / 'ArtPipeProcessor terminated unexpectedly' after redeploy

Related: #432, #426, #455 (art_pipe model provisioning family).

Filed proactively by automated health check (Seq centralized error audit).

## Summary The **SafetyCheck** resident art-pipe worker fails to start with `ModuleNotFoundError: No module named 'torch'`, and because a warmup fault is treated as a non-recoverable "pre-ready config error," it **takes down the entire `ArtPipeProcessor` host, which then crash-loops** (observed 24× `Fatal: terminated unexpectedly` + 23× `Hosting failed to start` in a 24h window). This also produces a burst of `TaskCanceledException`s as the host is torn down mid-startup. ## Evidence (Seq) ``` [Error] Resident worker FAULTED (no automatic restart — pre-ready failures are config errors): Worker for model 'SafetyCheck' failed during startup: Traceback (most recent call last): File "/opt/art_pipe/src/artpipe/worker.py", line 402, in _serve_main backend.warmup(on_progress=lambda p, m, **kw: log.info( File "/opt/art_pipe/src/artpipe/models/safety_check.py", line 155, in warmup self._ensure_model(on_progress) File "/opt/art_pipe/src/artpipe/models/safety_check.py", line 130, in _ensure_model import torch ModuleNotFoundError: No module named 'torch' ``` Source: `SpikerSoft.EventHandlers.ArtPipeProcessor.Services.ResidentArtPipeStageExecutor` Followed by `[Fatal] SpikerSoft.EventHandlers.ArtPipeProcessor terminated unexpectedly` + `[Error] Hosting failed to start`. ## Two problems here 1. **Provisioning gap:** the SafetyCheck model's resident venv has no `torch`. This is the same *family* as #432 (art_pipe model venvs missing requirements) but a different model + different missing package + the **Resident** executor path — so #432's fixes (install cloned-repo requirements, python3.10) don't obviously cover it. `safety_check.py:_ensure_model` imports `torch` directly, so its venv (or the worker base image) must provide torch. Not a CUDA-version issue like #426 — torch is entirely absent. 2. **Reliability / blast radius:** a single model's pre-ready fault crashes the **whole ArtPipeProcessor host** and it crash-loops with no isolation. One misprovisioned model should not take down the host / all other resident models. Consider quarantining a faulted model (mark it unavailable, keep the host up) instead of terminating the process. ## Impact - The **SafetyCheck** stage — the content-safety gate — cannot run. Depending on pipeline wiring this either blocks assets or lets them through ungated. - ArtPipeProcessor crash-loops, disrupting any resident-model stages it hosts. ## Proposed fix - Ensure `torch` is installed in the SafetyCheck resident venv / worker image (align with the cu128 torch index used elsewhere per #426). Verify `python -c "import torch"` succeeds in that venv during provisioning. - Add fault isolation so a single model warmup failure degrades that model only, not the whole host (log + mark unavailable, don't terminate). ## Verify ``` # in the SafetyCheck worker env python -c "import torch; print(torch.__version__)" # Seq: no new 'Resident worker FAULTED' / 'ArtPipeProcessor terminated unexpectedly' after redeploy ``` Related: #432, #426, #455 (art_pipe model provisioning family). --- _Filed proactively by automated health check (Seq centralized error audit)._

spikerj commented

2026-07-12 00:32:20 +00:00

Upstream root cause confirmed from Seq Warnings: ServeWorkerProcessFactory logs 'Model venv python not found at /opt/art_pipe/models/SafetyCheck/venv/bin/python; falling back to python3. Bootstrap the venv (#357).' So the SafetyCheck resident venv was never bootstrapped -> the worker runs under the system python3, which has no torch -> the ModuleNotFoundError. Fix ties to #357 (venv bootstrap): ensure the SafetyCheck model venv is created/provisioned (with torch) rather than silently falling back to system python3.

Upstream root cause confirmed from Seq Warnings: `ServeWorkerProcessFactory` logs 'Model venv python not found at /opt/art_pipe/models/SafetyCheck/venv/bin/python; falling back to python3. Bootstrap the venv (#357).' So the SafetyCheck resident venv was never bootstrapped -> the worker runs under the system python3, which has no torch -> the ModuleNotFoundError. Fix ties to #357 (venv bootstrap): ensure the SafetyCheck model venv is created/provisioned (with torch) rather than silently falling back to system python3.

spikerj commented

2026-07-12 00:33:53 +00:00

Provisioning half fixed in spikersoft-artpipe PR #10: SafetyCheck's install.pip never listed torch — transformers treats it as optional and, uniquely among the models, SafetyCheck vendors no repo whose requirements pull it in, so the venv legitimately had no torch despite the manifest's cu128 torch_index_url. torch is now declared (regression-asserted in test_manifest_is_valid). OPS after merge: re-run the SafetyCheck venv bootstrap on the GPU host — merging alone doesn't heal the existing venv. — Reliability half (blast radius) findings, backend-side: the executor's eager warmup deliberately swallows the fault (SetFault, jobs fail fast) and ResidentWorkerHealthCheck reports it, so the crash-loop is NOT the warmup itself. Suspects for the 'Hosting failed to start' + Fatal pairs each boot: (a) BackgroundService default StopHost behavior — SafetyCheckRpcConsumer.ExecuteAsync rethrows fatal Rabbit wiring errors, which stops the whole host; (b) ResidentGpuLeaseHolder.StartAsync's blocking AcquireAsync on restart racing the crashed instance's still-held resident lease (released on shutdown only) — an acquire failure during host start produces exactly 'Hosting failed to start'. Confirming needs the Seq exception detail on the Hosting-failed events; suggested fixes once confirmed: HostOptions.BackgroundServiceExceptionBehavior/retry loop in the RPC consumer, and a bounded retry-with-backoff (or takeover of the stale lease by holder identity) in the lease holder. Leaving open for the reliability half + venv re-bootstrap.

spikerj referenced this issue

2026-07-12 00:57:52 +00:00

art_pipe observability: install OpenTelemetry into model venvs so worker spans (#430) actually export (#428) #460

spikerj referenced this issue

2026-07-12 02:22:04 +00:00

[Bug][Prod] ArtPipeProcessor image: /home/app not writable + no HF_HOME → HuggingFace cache PermissionError blocks model weight downloads #495

spikerj referenced this issue

2026-07-12 04:24:46 +00:00

[Bug][Prod] /mnt/fusionio/spikersoft/ai missing embedding model — book pipeline dies at embeddings (nomic-embed-text-v2-moe.f16.gguf not found) #497

spikerj commented

2026-07-12 18:07:53 +00:00

Reliability half fixed in spikersoft-backend PR #232 — both triage suspects confirmed and addressed: (a) SafetyCheckRpcConsumer now retries transport wiring with backoff instead of rethrowing into BackgroundServiceExceptionBehavior.StopHost; (b) ResidentGpuLeaseHolder waits-and-retries the resident acquire inside StartAsync instead of crash-looping host start against a crashed predecessor's still-held lease. 3 regression tests incl. the load-bearing 'ExecuteTask completes unfaulted under persistent wiring failure'. Remaining on this ticket after merge: the ops verify that the SafetyCheck venv re-bootstrap took on the GPU host (provisioning half, artpipe PR #10).

Reliability half fixed in **spikersoft-backend PR #232** — both triage suspects confirmed and addressed: (a) SafetyCheckRpcConsumer now retries transport wiring with backoff instead of rethrowing into BackgroundServiceExceptionBehavior.StopHost; (b) ResidentGpuLeaseHolder waits-and-retries the resident acquire inside StartAsync instead of crash-looping host start against a crashed predecessor's still-held lease. 3 regression tests incl. the load-bearing 'ExecuteTask completes unfaulted under persistent wiring failure'. Remaining on this ticket after merge: the ops verify that the SafetyCheck venv re-bootstrap took on the GPU host (provisioning half, artpipe PR #10).

spikerj referenced this issue

2026-07-13 00:40:01 +00:00

[Epic][ArtPipe][Infra] Per-model baked Docker images — venvs + weights built at image build time, no /opt/art_pipe bind #515

spikerj referenced this issue

2026-07-13 00:40:19 +00:00

[ArtPipe] Baked images Phase 1: SafetyCheck pilot image + cutover, incl. #486 fault isolation (epic #515) #517

spikerj referenced this issue

2026-07-13 01:00:09 +00:00

[ArtPipe] Baked images Phase 1: SafetyCheck pilot image + cutover, incl. #486 fault isolation (epic #515) #517

spikerj referenced this issue

2026-07-13 01:46:43 +00:00

[ArtPipe] Baked images Phase 1: SafetyCheck pilot image + cutover, incl. #486 fault isolation (epic #515) #517

spikerj referenced this issue

2026-07-13 01:59:37 +00:00

[ArtPipe] Baked images Phase 1: SafetyCheck pilot image + cutover, incl. #486 fault isolation (epic #515) #517

spikerj commented

2026-07-22 02:57:51 +00:00

Board-sweep closing on merged evidence: reliability half in backend #232 (consumer retry + lease-holder wait); the missing-torch half is fixed definitionally by the baked SafetyCheck image cutover (infra #52, deployed 1/1 per #517). Reopen if the crash-loop signature reappears.