spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[infra drift] image-description & trellis3d deploy to /mnt/infrastructure folders that don't exist #403

New Issue
Closed
opened 2026-07-05 20:18:44 +00:00 by spikerj · 4 comments
spikerj commented 2026-07-05 20:18:44 +00:00
Owner
Copy Link
Copy Source

Verified finding (surfaced while fixing the Art Studio Metrics deploy gap, infra PR #11)

Two backend CI workflows have a Deploy to production step that cds into an infra folder that does not exist in spikersoft-infrastructure master, so the step errors on every triggering push:

Workflow (spikersoft-backend .gitea/workflows/) Deploy step
spikersoft-image-description.yml cd /mnt/infrastructure/spikersoft-image-description → docker stack deploy -c docker-stack-gpu.yml spikersoft-gpu
spikersoft-trellis3d.yml cd /mnt/infrastructure/spikersoft-trellis3d → docker stack deploy -c docker-stack-gpu.yml spikersoft-gpu

Verified

  • Both workflows trigger on push to master (paths-filtered to each service's project) + workflow_dispatch.
  • Neither spikersoft-image-description/ nor spikersoft-trellis3d/ exists in infra master → the cd fails and the docker stack deploy runs in the wrong dir with no docker-stack-gpu.yml present. The deploy step cannot succeed.
  • Neither is in the infra README's drift sections: not §6 (live-but-no-folder), not §7 (dormant folders — that lists spikersoft-embeddings/spikersoft-quiz-generation), not §8 (anomalies). So this is undocumented drift.
  • Both backend projects still exist (SpikerSoft.EventHandlers.{ImageDescription,Trellis3D}.Python/Dockerfile). Trellis3D also carries a Dockerfile.old (reworked).
  • Both deploy to a shared spikersoft-gpu stack — a pattern no current infra folder provides; the live GPU workers are the spikersoft-artpipe-model-* stacks.

Why this is likely superseded (not just missing)

The ProArt/Art Studio epic (#346) explicitly generalizes the single-stage Trellis3D.Python/trellis-3d-generator vertical into the multi-stage artpipe-modeling/imaging/rigging services ("don't build a parallel one"). So these two workflows may be legacy and their missing folders intentional.

Two resolutions — needs the GPU-topology owner's call (not auto-fixing)

(a) Superseded → disable/remove spikersoft-image-description.yml + spikersoft-trellis3d.yml (and archive the backend projects) if the artpipe-model-* stacks replaced them.
(b) Still live → commit spikersoft-{image-description,trellis3d}/docker-stack-gpu.yml folders (modeled on the spikersoft-artpipe-model-* GPU stacks) so the deploy resolves. I did not author these — GPU device reservations, model/venv volume mounts (/mnt/fusionio), and the shared-spikersoft-gpu-stack intent require someone who knows the GPU topology; a mechanical CPU-stack copy (like the metrics fix) would be wrong here.

Contrast: the Art Studio Metrics gap was auto-fixable (clean CPU consumer, sibling keycloak-events template) → done in infra PR #11. These two are GPU + possibly-dead, so they get a ticket instead.

Severity: Medium (broken deploy step, but low-frequency — paths-filtered — and possibly dead paths; confirm before acting). Related: #346, infra PR #11.

## Verified finding (surfaced while fixing the Art Studio Metrics deploy gap, infra PR #11) Two backend CI workflows have a **Deploy to production** step that `cd`s into an infra folder that **does not exist** in `spikersoft-infrastructure` master, so the step errors on every triggering push: | Workflow (spikersoft-backend `.gitea/workflows/`) | Deploy step | |---|---| | `spikersoft-image-description.yml` | `cd /mnt/infrastructure/spikersoft-image-description` → `docker stack deploy -c docker-stack-gpu.yml spikersoft-gpu` | | `spikersoft-trellis3d.yml` | `cd /mnt/infrastructure/spikersoft-trellis3d` → `docker stack deploy -c docker-stack-gpu.yml spikersoft-gpu` | ### Verified - Both workflows **trigger on push to `master`** (paths-filtered to each service's project) + `workflow_dispatch`. - Neither `spikersoft-image-description/` nor `spikersoft-trellis3d/` exists in infra master → the `cd` fails and the `docker stack deploy` runs in the wrong dir with no `docker-stack-gpu.yml` present. **The deploy step cannot succeed.** - Neither is in the infra README's drift sections: not §6 (live-but-no-folder), not §7 (dormant folders — that lists `spikersoft-embeddings`/`spikersoft-quiz-generation`), not §8 (anomalies). So this is **undocumented drift**. - Both backend projects still exist (`SpikerSoft.EventHandlers.{ImageDescription,Trellis3D}.Python/Dockerfile`). Trellis3D also carries a `Dockerfile.old` (reworked). - Both deploy to a **shared `spikersoft-gpu` stack** — a pattern no current infra folder provides; the live GPU workers are the `spikersoft-artpipe-model-*` stacks. ### Why this is likely superseded (not just missing) The ProArt/Art Studio epic (#346) explicitly **generalizes the single-stage `Trellis3D.Python`/`trellis-3d-generator` vertical** into the multi-stage `artpipe-modeling`/`imaging`/`rigging` services ("don't build a parallel one"). So these two workflows may be **legacy** and their missing folders **intentional**. ## Two resolutions — needs the GPU-topology owner's call (not auto-fixing) **(a) Superseded** → disable/remove `spikersoft-image-description.yml` + `spikersoft-trellis3d.yml` (and archive the backend projects) if the artpipe-model-* stacks replaced them. **(b) Still live** → commit `spikersoft-{image-description,trellis3d}/docker-stack-gpu.yml` folders (modeled on the `spikersoft-artpipe-model-*` GPU stacks) so the deploy resolves. I did **not** author these — GPU device reservations, model/venv volume mounts (`/mnt/fusionio`), and the shared-`spikersoft-gpu`-stack intent require someone who knows the GPU topology; a mechanical CPU-stack copy (like the metrics fix) would be wrong here. Contrast: the Art Studio Metrics gap was auto-fixable (clean CPU consumer, sibling `keycloak-events` template) → done in infra PR #11. These two are GPU + possibly-dead, so they get a ticket instead. **Severity:** Medium (broken deploy step, but low-frequency — paths-filtered — and possibly dead paths; confirm before acting). **Related:** #346, infra PR #11.
spikerj commented 2026-07-12 04:55:32 +00:00
Author
Owner
Copy Link
Copy Source

Production evidence: image-description is still live (resolves the (a) superseded vs (b) still-live question) — and its absence is a user-facing book-upload failure

Surfaced today (2026-07-12) while investigating book-upload failures on the SERVER swarm. The open question in this issue — "are these legacy/superseded, or still live?" — is answered decisively by the running pipeline: the book-upload pipeline still hard-depends on the image-description service, and because it isn't deployed (the deploy drift in this issue), every book that contains images hangs.

What happens

upload-coordinator (currently running) fans out one image.description.requested event per extracted image and advances the workflow to Stage=ImageDescriptions (55%). Example from a 36-image book upload today:

04:49:17 INF  UploadCoordinator  📸 Found 36 images for book 6a531ccb…, validating file links
04:49:17 DBG  UploadCoordinator  Published event to image.extraction/image.description.requested  (×36)
04:49:17 DBG  UploadCoordinator  Sent book processing notification: Stage=GeneratingImageDescriptions, Progress=55%

Nothing consumes those events

  • No image-description / vision service exists in docker service ls (consistent with the failed deploy in this issue).
  • spikersoft-backend consumer registrations at startup: cluster.events, art.asset.*, SignalR notifications, code-execution — no image-description consumer.
  • gpu-coordinator grants leases to Embeddings, QuizGeneration, and ArtPipe (artpipe-safetycheck) — never an image-description worker requesting a lease.

Result: books with images hang until staleness

04:26:03 WRN  UploadCoordinator  Staleness detected: UploadId=edca9d69-5d14-44ff-86c3-06e9b08f704c,
              Stage=ImageDescriptions, Elapsed=31.0min

Books without images complete fine — the coordinator logs 📝 No images to process … triggering embeddings directly and skips the stage. So the failure is specific to image-bearing books.

Recommendation

This is resolution path (b) still live in this issue: commit the spikersoft-image-description/docker-stack-gpu.yml deploy folder (modeled on the spikersoft-artpipe-model-* GPU stacks) so the service deploys and consumes image.extraction/image.description.requested. It is not safe to disable/remove per path (a) — the book pipeline has no fallback and silently hangs at 55% without it.

(Trellis3D — the other service in this issue — is a separate 3D/art-pipeline concern and isn't implicated by this book-pipeline evidence.)

## Production evidence: image-description is **still live** (resolves the (a) superseded vs (b) still-live question) — and its absence is a user-facing book-upload failure Surfaced today (2026-07-12) while investigating book-upload failures on the SERVER swarm. The open question in this issue — "are these legacy/superseded, or still live?" — is answered decisively by the running pipeline: **the book-upload pipeline still hard-depends on the `image-description` service, and because it isn't deployed (the deploy drift in this issue), every book that contains images hangs.** ### What happens `upload-coordinator` (currently running) fans out one `image.description.requested` event per extracted image and advances the workflow to `Stage=ImageDescriptions (55%)`. Example from a 36-image book upload today: ``` 04:49:17 INF UploadCoordinator 📸 Found 36 images for book 6a531ccb…, validating file links 04:49:17 DBG UploadCoordinator Published event to image.extraction/image.description.requested (×36) 04:49:17 DBG UploadCoordinator Sent book processing notification: Stage=GeneratingImageDescriptions, Progress=55% ``` ### Nothing consumes those events - No `image-description` / vision service exists in `docker service ls` (consistent with the failed deploy in this issue). - `spikersoft-backend` consumer registrations at startup: `cluster.events`, `art.asset.*`, SignalR notifications, code-execution — **no image-description consumer**. - `gpu-coordinator` grants leases to Embeddings, QuizGeneration, and ArtPipe (`artpipe-safetycheck`) — **never** an image-description worker requesting a lease. ### Result: books with images hang until staleness ``` 04:26:03 WRN UploadCoordinator Staleness detected: UploadId=edca9d69-5d14-44ff-86c3-06e9b08f704c, Stage=ImageDescriptions, Elapsed=31.0min ``` Books **without** images complete fine — the coordinator logs `📝 No images to process … triggering embeddings directly` and skips the stage. So the failure is specific to image-bearing books. ### Recommendation This is resolution path **(b) still live** in this issue: commit the `spikersoft-image-description/docker-stack-gpu.yml` deploy folder (modeled on the `spikersoft-artpipe-model-*` GPU stacks) so the service deploys and consumes `image.extraction/image.description.requested`. It is **not** safe to disable/remove per path (a) — the book pipeline has no fallback and silently hangs at 55% without it. (Trellis3D — the other service in this issue — is a separate 3D/art-pipeline concern and isn't implicated by this book-pipeline evidence.)
spikerj commented 2026-07-12 05:04:36 +00:00
Author
Owner
Copy Link
Copy Source

Image-description half resolved as (b) — and it's definitively LIVE, not superseded: tonight's book upload proved the UploadOrchestrator actively publishes image.description.requested (36 messages) and the whole post-approval tail (descriptions → embeddings → quiz) is gated on this consumer. With the phantom deploy, nothing consumed the queue and the book stuck exactly as this ticket predicted.

Fix PRs:

  • spikersoft-infrastructure #36 — the missing spikersoft-image-description/ stack, committed. artpipe-gpu placement (16 GB Qwen3-VL lease ⇒ effectively the 4090, which currently holds the label), GPU_SCHEDULING_MODE=shared, node-local model-cache volume, no fusionio bind.
  • spikersoft-backend #225 — (1) S3 image fetch: off-SERVER the worker GETs books/<ISBN>/images/… from the MinIO ebooks bucket (the #493 dual-write output — its first consumer), filesystem-first, hardened key mapping; (2) the deploy step now targets the real folder/stack with the empty-secret guard.

This also resolves the data-gravity blocker that made the old hand-rolled spikersoft-gpu deploy fragile: the worker no longer needs SERVER's disks at all.

Trellis3D half stays open — likely superseded per #346; needs Joey's (a)-vs-(b) call. Recommend (a): disable/remove spikersoft-trellis3d.yml's deploy or the whole workflow.

Activation: merge both → pull /mnt/infrastructure → backend merge auto-triggers build+deploy → task schedules on the 4090 → stuck book resumes.

**Image-description half resolved as (b) — and it's definitively LIVE, not superseded:** tonight's book upload proved the UploadOrchestrator actively publishes `image.description.requested` (36 messages) and the whole post-approval tail (descriptions → embeddings → quiz) is gated on this consumer. With the phantom deploy, nothing consumed the queue and the book stuck exactly as this ticket predicted. Fix PRs: - **spikersoft-infrastructure #36** — the missing `spikersoft-image-description/` stack, committed. `artpipe-gpu` placement (16 GB Qwen3-VL lease ⇒ effectively the 4090, which currently holds the label), `GPU_SCHEDULING_MODE=shared`, node-local model-cache volume, **no fusionio bind**. - **spikersoft-backend #225** — (1) S3 image fetch: off-SERVER the worker GETs `books/<ISBN>/images/…` from the MinIO `ebooks` bucket (the #493 dual-write output — its first consumer), filesystem-first, hardened key mapping; (2) the deploy step now targets the real folder/stack with the empty-secret guard. This also resolves the data-gravity blocker that made the old hand-rolled `spikersoft-gpu` deploy fragile: the worker no longer needs SERVER's disks at all. **Trellis3D half stays open** — likely superseded per #346; needs Joey's (a)-vs-(b) call. Recommend (a): disable/remove `spikersoft-trellis3d.yml`'s deploy or the whole workflow. Activation: merge both → pull `/mnt/infrastructure` → backend merge auto-triggers build+deploy → task schedules on the 4090 → stuck book resumes.
spikerj commented 2026-07-12 19:37:58 +00:00
Author
Owner
Copy Link
Copy Source

Trellis3d half: backend PR #243 makes the workflow workflow_dispatch-only. Every auto-triggered run was a guaranteed red X (the deploy step cds into /mnt/infrastructure/spikersoft-trellis3d, which does not exist), so auto-trigger was pure noise. The workflow header now documents both exits: restore push/PR triggers + create the infra folder if Trellis 3D is repaired (#412), or delete the workflow + project if retired. The #412 decision itself stays open and is unchanged by this.

With the image-description half already verified live in production (infra #36 + MinIO model sync; consumer drained the 36-message queue during the book-pipeline verification), #243 is the last actionable piece of this ticket — suggest closing on its merge, with the repair-vs-retire question living on in #412 where it belongs.

Trellis3d half: **backend PR #243** makes the workflow `workflow_dispatch`-only. Every auto-triggered run was a guaranteed red X (the deploy step `cd`s into `/mnt/infrastructure/spikersoft-trellis3d`, which does not exist), so auto-trigger was pure noise. The workflow header now documents both exits: restore push/PR triggers + create the infra folder if Trellis 3D is repaired (#412), or delete the workflow + project if retired. The #412 decision itself stays open and is unchanged by this. With the image-description half already verified live in production (infra #36 + MinIO model sync; consumer drained the 36-message queue during the book-pipeline verification), #243 is the last actionable piece of this ticket — suggest closing on its merge, with the repair-vs-retire question living on in #412 where it belongs.
spikerj commented 2026-07-12 19:43:31 +00:00
Author
Owner
Copy Link
Copy Source

Closing — both halves resolved:

  • image-description: infra #36 stack folder + CI deploy verified live in production (consumer drained the image.description.requested queue during the end-to-end book-pipeline verification; models/images now served from MinIO).
  • trellis3d: backend #243 merged — workflow is workflow_dispatch-only, so no more guaranteed-red auto-runs. The repair-vs-retire product decision lives on in #412 (the workflow header documents both exits).
Closing — both halves resolved: - **image-description**: infra #36 stack folder + CI deploy verified live in production (consumer drained the image.description.requested queue during the end-to-end book-pipeline verification; models/images now served from MinIO). - **trellis3d**: backend #243 merged — workflow is `workflow_dispatch`-only, so no more guaranteed-red auto-runs. The repair-vs-retire product decision lives on in #412 (the workflow header documents both exits).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#403