Art Studio: artifacts only appear when the whole asset finishes, and the progress percentages are not honest #863

New Issue

2026-07-27T01:57:46Z

spikerj commented

2026-07-27 01:57:46 +00:00

The artifact panel says "No artifacts yet — they appear as stages complete." That is
false: nothing appears until the whole asset finishes. The percentages are also
untrustworthy — most are hand-placed milestones, one (Blender) is openly synthetic, and
the overall bar weights a 3-second enrichment the same as a 6-minute modeling stage.

No MinIO bucket triggers are needed. art_pipe knows nothing about S3 — the .NET side does
all uploading, so the upload call site already knows the instant an artifact lands.
Bucket notifications would be the wrong layer.

What is actually wrong

ArtAssetStageProgressEvent carries no artifact signal, and the Angular SignalR source
only re-fetches the asset on the terminal status — so asset.artifacts cannot
change mid-run. (The polling fallback re-reads both endpoints every tick, which is why
this looked intermittent rather than simply broken.)
art_pipe's progress message ("Extracting mesh") already crosses the worker protocol
and both .NET executors parse it — then ArtPipeStageOrchestrator.OnProgress drops it.
The one honest signal never reaches the UI.
Overall progress counts completed stages plus one running stage at uniform weight,
so retries, two running rows and cheap-vs-expensive stages all mis-report.

Approach

Event-triggered refetch, not event-carried URLs. The event gains a credential-free
artifactCount hint and the client refetches GET api/artstudio/{id} on stage-completed.
Presigned URLs stay out of the message: they are bearer credentials in a query string, the
same message bodies are read by the #389 metrics projection queue (a DLQ dump or Seq log
would become a credential store), and a worker-minted URL is a different string than the
controller's bucket-memoized one, so the browser would download the bytes twice and the
#804 cache-across-visits property would break.

Contract — Message, Measured, ArtifactCount on ArtAssetStageProgressEvent,
all [JsonIgnore(WhenWritingNull)] so absent-value payloads stay byte-identical.
Backend — carry the message through the orchestrator; split the progress throttles
(publish 2 %, persist 5 %) so hub frames are finer than Mongo writes, keeping the
DB-cancel poll on the persist cadence.
Frontend — refresh artifacts on stage-completed with dedupe + debounce + in-flight
guards; weighted overall progress capped below 100 until every stage is done; measured
percentages render determinate, milestones render as a buffer bar with an estimate
marker; art_pipe's phase text goes in the stage console.
artpipe — a real per-step callback_on_step_end for the concept stage
(sdxl_lightning), reporting measured=True. Blender's synthetic creep is deliberately
left alone: with the flag opt-in, it renders honestly as an estimate for free.

Follow-up to #349 (the event plumbing this builds on).

The artifact panel says *"No artifacts yet — they appear as stages complete."* That is false: nothing appears until the **whole asset** finishes. The percentages are also untrustworthy — most are hand-placed milestones, one (Blender) is openly synthetic, and the overall bar weights a 3-second enrichment the same as a 6-minute modeling stage. No MinIO bucket triggers are needed. art_pipe knows nothing about S3 — the .NET side does all uploading, so the upload call site already knows the instant an artifact lands. Bucket notifications would be the wrong layer. ## What is actually wrong 1. `ArtAssetStageProgressEvent` carries no artifact signal, and the Angular SignalR source only re-fetches the asset on the **terminal** status — so `asset.artifacts` cannot change mid-run. (The polling fallback re-reads both endpoints every tick, which is why this looked intermittent rather than simply broken.) 2. art_pipe's progress `message` ("Extracting mesh") already crosses the worker protocol and both .NET executors parse it — then `ArtPipeStageOrchestrator.OnProgress` drops it. The one honest signal never reaches the UI. 3. Overall progress counts completed stages plus **one** running stage at uniform weight, so retries, two running rows and cheap-vs-expensive stages all mis-report. ## Approach Event-**triggered refetch**, not event-carried URLs. The event gains a credential-free `artifactCount` hint and the client refetches `GET api/artstudio/{id}` on stage-completed. Presigned URLs stay out of the message: they are bearer credentials in a query string, the same message bodies are read by the #389 metrics projection queue (a DLQ dump or Seq log would become a credential store), and a worker-minted URL is a different string than the controller's bucket-memoized one, so the browser would download the bytes twice and the #804 cache-across-visits property would break. - **Contract** — `Message`, `Measured`, `ArtifactCount` on `ArtAssetStageProgressEvent`, all `[JsonIgnore(WhenWritingNull)]` so absent-value payloads stay byte-identical. - **Backend** — carry the message through the orchestrator; split the progress throttles (publish 2 %, persist 5 %) so hub frames are finer than Mongo writes, keeping the DB-cancel poll on the persist cadence. - **Frontend** — refresh artifacts on stage-completed with dedupe + debounce + in-flight guards; weighted overall progress capped below 100 until every stage is done; measured percentages render determinate, milestones render as a buffer bar with an estimate marker; art_pipe's phase text goes in the stage console. - **artpipe** — a real per-step `callback_on_step_end` for the concept stage (`sdxl_lightning`), reporting `measured=True`. Blender's synthetic creep is deliberately left alone: with the flag opt-in, it renders honestly as an estimate for free. Follow-up to #349 (the event plumbing this builds on).

spikerj commented

2026-07-27 02:02:41 +00:00

PRs open:

spikersoft-backend #491 — the wire contract (message / measured / artifactCount, all omitted when null) and the publish/persist throttle split.
spikersoft-angular #585 — the headline fix: artifacts refetched on stage-completed, weighted overall progress capped below 100 until done, measured-vs-milestone rendering, phase text in the stage console. Also wraps the timeline/console effects in untracked() (the #860 mechanism).
spikersoft-artpipe #42 — real per-step callback_on_step_end for the concept stage, reporting measured=True.

Sequencing: the backend + Angular PRs fix the artifact bug against the current worker image, because the message field already crossed the worker protocol and was simply dropped before reaching the wire. Only the artpipe PR needs the 36 GiB prodstages rebuild, and it only upgrades concept from milestone to measured.

PRs open: - spikersoft-backend #491 — the wire contract (`message` / `measured` / `artifactCount`, all omitted when null) and the publish/persist throttle split. - spikersoft-angular #585 — the headline fix: artifacts refetched on stage-completed, weighted overall progress capped below 100 until done, measured-vs-milestone rendering, phase text in the stage console. Also wraps the timeline/console effects in `untracked()` (the #860 mechanism). - spikersoft-artpipe #42 — real per-step `callback_on_step_end` for the concept stage, reporting `measured=True`. Sequencing: the backend + Angular PRs fix the artifact bug against the **current** worker image, because the `message` field already crossed the worker protocol and was simply dropped before reaching the wire. Only the artpipe PR needs the 36 GiB `prodstages` rebuild, and it only upgrades concept from milestone to measured.

spikerj referenced this issue

2026-07-27 02:36:20 +00:00

spikersoft-artpipe has no PR CI — its whole pytest suite is ungated #864

spikerj commented

2026-07-29 15:28:15 +00:00

Resolved across all three repos — spikersoft-angular PR #585 (f361c20f), spikersoft-backend PR #491, spikersoft-artpipe PR #42 (e535f35). Verified against origin/master / origin/main:

Contract — SpikerSoft.Contracts.Workers/ArtStudio/ArtAssetLifecycleEvents.cs: Message (:88), Measured (:102), ArtifactCount (:119), each with [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] (:87,101,118) so old consumers are unaffected; MessageMaxLength at :122.

Backend — the orchestrator no longer drops the message on the floor: ArtPipeStageOrchestrator.cs:319-333 splits publishThrottle from persistThrottle and passes progress.Message / progress.Measured through PublishStageProgressAsync (:332-333), while the DB-cancel poll stays on the slower persist cadence (:335-350).

Angular — refetch-on-stage-completed driven by the artifactCount hint (signalr-art-asset-progress-source.ts:39,243,355), so artifacts appear per stage rather than only at asset completion. Honest progress via weighted stages: ART_STAGE_WEIGHTS (art-studio.models.ts:1501), DEFAULT_ART_STAGE_WEIGHT (:1514), rollup at :1538-1550; measured-vs-milestone rendering distinguished at art-stage-timeline.component.html:69,74.

artpipe — real per-step progress from SDXL Lightning: src/artpipe/models/sdxl_lightning.py:217-234,386 callback_on_step_end emitting measured=True; test tests/test_concept_run_params.py:200.

⚠️ Deploy check before you trust this in prod. Per comment 13152, the 2026-07-26 prodstages rebuild predates artpipe PR #42. If no artpipe image rebuild has run since, the live worker will still emit milestone-only ticks and the measured=True path won't be exercised — the code is right, the running image may not be. Same rebuild gates #861's target_faces relaxation. Worth confirming the deployed image digest.

Closing on code completeness.

Resolved across all three repos — spikersoft-angular PR #585 (`f361c20f`), spikersoft-backend PR #491, spikersoft-artpipe PR #42 (`e535f35`). Verified against `origin/master` / `origin/main`: **Contract** — `SpikerSoft.Contracts.Workers/ArtStudio/ArtAssetLifecycleEvents.cs`: `Message` (`:88`), `Measured` (`:102`), `ArtifactCount` (`:119`), each with `[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]` (`:87,101,118`) so old consumers are unaffected; `MessageMaxLength` at `:122`. **Backend** — the orchestrator no longer drops the message on the floor: `ArtPipeStageOrchestrator.cs:319-333` splits `publishThrottle` from `persistThrottle` and passes `progress.Message` / `progress.Measured` through `PublishStageProgressAsync` (`:332-333`), while the DB-cancel poll stays on the slower persist cadence (`:335-350`). **Angular** — refetch-on-stage-completed driven by the `artifactCount` hint (`signalr-art-asset-progress-source.ts:39,243,355`), so artifacts appear per stage rather than only at asset completion. Honest progress via weighted stages: `ART_STAGE_WEIGHTS` (`art-studio.models.ts:1501`), `DEFAULT_ART_STAGE_WEIGHT` (`:1514`), rollup at `:1538-1550`; measured-vs-milestone rendering distinguished at `art-stage-timeline.component.html:69,74`. **artpipe** — real per-step progress from SDXL Lightning: `src/artpipe/models/sdxl_lightning.py:217-234,386` `callback_on_step_end` emitting `measured=True`; test `tests/test_concept_run_params.py:200`. ⚠️ **Deploy check before you trust this in prod.** Per comment 13152, the 2026-07-26 `prodstages` rebuild predates artpipe PR #42. If no artpipe image rebuild has run since, the live worker will still emit milestone-only ticks and the `measured=True` path won't be exercised — the code is right, the running image may not be. Same rebuild gates #861's `target_faces` relaxation. Worth confirming the deployed image digest. Closing on code completeness.

spikerj closed this issue

2026-07-29 15:28:18 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#863