[ArtStudio] Stage requests for unconsumed stages are silently dropped — asset hangs InProgress forever #360

New Issue

2026-07-04T12:53:07Z

spikerj commented

2026-07-04 12:53:07 +00:00

Found during the #359 end-to-end run. Epic #346.

Symptom: if a stage in an asset's plan is not in any running worker's ArtPipe:Stages set, the art.asset.stage.requested.<stage> message is published to the topic exchange with no bound queue and vanishes. The stage run stays Pending and the asset stays InProgress forever — no error, no timeout, nothing for the student or staff to see. (Contrast: a consumed stage with missing StageSettings fails cleanly with events.)

Repro: run ArtPipeProcessor with ArtPipe:Stages=["concept","modeling"], submit a Prop (plan includes texturing) → texturing request published, dropped; asset never terminal.

Fix directions (pick in implementation):

Alternate-exchange / mandatory-publish dead-letter on art.asset.lifecycle so unroutable stage requests land somewhere observable and fail the run, and/or
a watchdog (API hosted service or worker) that fails stage runs stuck Pending/Queued beyond a config TTL, and/or
submit-time validation that every stage in the plan has a registered consumer (requires a stage-registry heartbeat).

Watchdog + alternate-exchange together is the belt-and-suspenders answer; at minimum the watchdog, since it also covers worker crashes mid-stage (lease released, message acked, run stuck Running).

Acceptance: an asset whose plan contains an unconsumed stage reaches a terminal Failed state with a clear error within the TTL, visible via the API and lifecycle events; no asset can stay InProgress indefinitely without a live stage run.

Found during the #359 end-to-end run. Epic #346. **Symptom:** if a stage in an asset's plan is not in any running worker's `ArtPipe:Stages` set, the `art.asset.stage.requested.<stage>` message is published to the topic exchange with no bound queue and vanishes. The stage run stays Pending and the asset stays InProgress forever — no error, no timeout, nothing for the student or staff to see. (Contrast: a *consumed* stage with missing StageSettings fails cleanly with events.) **Repro:** run ArtPipeProcessor with `ArtPipe:Stages=["concept","modeling"]`, submit a Prop (plan includes texturing) → texturing request published, dropped; asset never terminal. **Fix directions (pick in implementation):** - Alternate-exchange / mandatory-publish dead-letter on `art.asset.lifecycle` so unroutable stage requests land somewhere observable and fail the run, and/or - a watchdog (API hosted service or worker) that fails stage runs stuck Pending/Queued beyond a config TTL, and/or - submit-time validation that every stage in the plan has a registered consumer (requires a stage-registry heartbeat). Watchdog + alternate-exchange together is the belt-and-suspenders answer; at minimum the watchdog, since it also covers worker crashes mid-stage (lease released, message acked, run stuck Running). **Acceptance:** an asset whose plan contains an unconsumed stage reaches a terminal Failed state with a clear error within the TTL, visible via the API and lifecycle events; no asset can stay InProgress indefinitely without a live stage run.

spikerj added the bug label 2026-07-04 12:53:07 +00:00

spikerj commented

2026-07-04 16:22:15 +00:00

Resolved in spikersoft-backend PR #65 (merged to master). Watchdog hosted service fails stage runs stuck Pending/Queued (10 min default) or Running (2 h default) — covering silently-dropped requests and worker crashes — plus an additive art.asset.unrouted fanout+queue for immediate unrouted-message failure. Full alternate-exchange capture activates via broker policy (documented in the contracts file; belongs in infra provisioning). Closing.

Resolved in spikersoft-backend PR #65 (merged to `master`). Watchdog hosted service fails stage runs stuck Pending/Queued (10 min default) or Running (2 h default) — covering silently-dropped requests and worker crashes — plus an additive `art.asset.unrouted` fanout+queue for immediate unrouted-message failure. Full alternate-exchange capture activates via broker policy (documented in the contracts file; belongs in infra provisioning). Closing.

spikerj closed this issue

2026-07-04 16:22:31 +00:00

spikerj referenced this issue

2026-07-17 14:43:41 +00:00

[Reader][Perf] EPUB page prefetch/LRU + fix dead ngsw api-books cache group #650

spikerj referenced this issue

2026-07-25 01:39:55 +00:00

ModelEnvImages: triposg venv unbuildable (fvdb needs Python >=3.11, manifest pins 3.10) — aborts the serial loop and silently blocks 4 later images #839

Sign in to join this conversation.