spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

EPIC: Pixal3D as an Art Studio modeling backend #833

New Issue
Open
opened 2026-07-25 00:53:13 +00:00 by spikerj · 1 comment
spikerj commented 2026-07-25 00:53:13 +00:00
Owner
Copy Link
Copy Source

Goal

Add TencentARC/Pixal3D (SIGGRAPH 2026, MIT) as a selectable modeling stage backend. Pixal3D back-projects pixel features into 3D for direct pixel-to-3D correspondence, giving near-reconstruction fidelity with PBR textures — a clear step up from TripoSG / TripoSR / InstantMesh.

Target shape: a deployed service that consumes a RabbitMQ message with W3C trace context, runs weights baked into the container, uploads the GLB to MinIO, and drives the UI over SignalR via the API.

Key constraints (established during planning)

  • CUDA/Linux only. The TRELLIS.2 base env, natten and flex_gemm are CUDA-compiled and the pipeline hardcodes .cuda(). Runs on the 4090 node (artpipe-gpu lane) only — not SERVER's 3070 Ti (8 GB), not Apple Silicon. No AppleSilicon method variant.
  • VRAM: 18,000 MB at the default 1536 resolution (~10-12 GB with low_vram). Lane arithmetic: 24,576 MB total, ~24,064 usable, minus SafetyCheck 1,024 + Blender 2,048 = 20,992 free. 18,000 is grantable. Note spikersoft-image-description holds 20,000 on the same lane, so the two serialize.
  • Image size ~35-45 GB — weights alone are ~27 GB (TencentARC/Pixal3D is 24.04 GB across three 1.3B bf16 cascade checkpoints; plus MoGe-2 1.31 GB, DINOv3 1.21 GB, BiRefNet 0.44 GB).
  • No new .NET worker project. SpikerSoft.EventHandlers.ArtPipeProcessor already does the whole loop (consume → trace context → subprocess → NSFW gate → MinIO → stage events → SignalR relay). A bespoke project would fork the safety gate, artifact rollback, failover chains, DLQ retry and telemetry.
  • WRAP strategy, mirroring trellis_mac (#358): inference.py sets os.environ before importing torch, exposes no progress callback, and hardcodes .cuda() — importing it in-process fights all three.

Notable finding: upstream's export is lossy

inference.py ends with glb.export(path, extension_webp=True). trimesh's exporter does img.save(f, format="WEBP") with no lossless=True and no quality=, so Pillow's default applies — lossy, quality 80 — silently re-compressing every 4096px PBR map. It also marks EXT_texture_webp as required, so readers lacking the extension must refuse the file.

Converting to PNG afterwards would preserve already-damaged pixels, so we intervene before the damage: our runner neutralises the flag at the trimesh boundary and exports lossless PNG by default. texture_format=webp reproduces upstream for A/B comparison.

Slices

  • PR-1 — artpipe: manifest + WRAP backend + runner + tests
  • PR-2 — artpipe: make the VRAM admission gate tunable (18000 is unadmittable at today's constants)
  • PR-3 — backend: decouple model-queue binding from resident mode + per-model routing allow-list
  • PR-4 — CI: bake the pixal3d env image
  • PR-5 — deploy stack + method option + routing cutover

Licensing

Pixal3D code is MIT; MoGe-2 and BiRefNet are MIT. Caveat: upstream pins DINOv3 to the third-party re-upload camenduru/dinov3-vitl16-pretrain-lvd1689m rather than Meta's gated facebook/ repo. Meta's DINOv3 licence still governs those weights — decision taken to keep upstream's mirror as-shipped for now; revisit before commercial distribution.

## Goal Add [TencentARC/Pixal3D](https://github.com/TencentARC/Pixal3D) (SIGGRAPH 2026, MIT) as a selectable **modeling** stage backend. Pixal3D back-projects pixel features into 3D for direct pixel-to-3D correspondence, giving near-reconstruction fidelity with PBR textures — a clear step up from TripoSG / TripoSR / InstantMesh. Target shape: a deployed service that consumes a RabbitMQ message with W3C trace context, runs weights baked into the container, uploads the GLB to MinIO, and drives the UI over SignalR via the API. ## Key constraints (established during planning) - **CUDA/Linux only.** The TRELLIS.2 base env, `natten` and `flex_gemm` are CUDA-compiled and the pipeline hardcodes `.cuda()`. Runs on the `4090` node (`artpipe-gpu` lane) only — not SERVER's 3070 Ti (8 GB), not Apple Silicon. No `AppleSilicon` method variant. - **VRAM: 18,000 MB** at the default 1536 resolution (~10-12 GB with `low_vram`). Lane arithmetic: 24,576 MB total, ~24,064 usable, minus SafetyCheck 1,024 + Blender 2,048 = **20,992 free**. 18,000 is grantable. Note `spikersoft-image-description` holds 20,000 on the same lane, so the two serialize. - **Image size ~35-45 GB** — weights alone are ~27 GB (`TencentARC/Pixal3D` is 24.04 GB across three 1.3B bf16 cascade checkpoints; plus MoGe-2 1.31 GB, DINOv3 1.21 GB, BiRefNet 0.44 GB). - **No new .NET worker project.** `SpikerSoft.EventHandlers.ArtPipeProcessor` already does the whole loop (consume → trace context → subprocess → NSFW gate → MinIO → stage events → SignalR relay). A bespoke project would fork the safety gate, artifact rollback, failover chains, DLQ retry and telemetry. - **WRAP strategy**, mirroring `trellis_mac` (#358): `inference.py` sets `os.environ` before importing torch, exposes no progress callback, and hardcodes `.cuda()` — importing it in-process fights all three. ## Notable finding: upstream's export is lossy `inference.py` ends with `glb.export(path, extension_webp=True)`. trimesh's exporter does `img.save(f, format="WEBP")` with **no `lossless=True` and no `quality=`**, so Pillow's default applies — **lossy, quality 80** — silently re-compressing every 4096px PBR map. It also marks `EXT_texture_webp` as *required*, so readers lacking the extension must refuse the file. Converting to PNG afterwards would preserve already-damaged pixels, so we intervene before the damage: our runner neutralises the flag at the trimesh boundary and exports lossless PNG by default. `texture_format=webp` reproduces upstream for A/B comparison. ## Slices - [ ] PR-1 — artpipe: manifest + WRAP backend + runner + tests - [ ] PR-2 — artpipe: make the VRAM admission gate tunable (18000 is unadmittable at today's constants) - [ ] PR-3 — backend: decouple model-queue binding from resident mode + per-model routing allow-list - [ ] PR-4 — CI: bake the `pixal3d` env image - [ ] PR-5 — deploy stack + method option + routing cutover ## Licensing Pixal3D code is MIT; MoGe-2 and BiRefNet are MIT. **Caveat:** upstream pins DINOv3 to the third-party re-upload `camenduru/dinov3-vitl16-pretrain-lvd1689m` rather than Meta's gated `facebook/` repo. Meta's DINOv3 licence still governs those weights — decision taken to keep upstream's mirror as-shipped for now; revisit before commercial distribution.
spikerj commented 2026-07-29 16:54:53 +00:00
Author
Owner
Copy Link
Copy Source

Epic status roll-up — audited all three children against origin/master / origin/main. Staying open: the code is done, but nothing can route to it yet.

Child State
#834 PR-1: artpipe manifest + WRAP backend + runner ✅ code complete; 4090 hardware runs unrun
#835 PR-2: tunable VRAM admission gate ✅ code complete; 4090 confirmation unrun
#838 PR-5: deploy stack, method option, routing cutover ⚠️ PARTIAL — the stack is inert

#834 / #835 are genuinely done in code. models/Pixal3D/artpipe.json (image_to_3d, vram_mb: 18000, py3.10/cu128, ATTN_BACKEND=sdpa, verify_imports for every compiled extension); run_pixal3d.py:64 force_png_textures(); src/artpipe/models/pixal3d.py Pixal3DBackend; config.py:130 TIMEOUT_PIXAL3D = 2700 and :404 pixal3d in the CUDA allowed_models["modeling"] — with :391 confirming the stated negative constraint holds (stage_defaults["modeling"] = "hunyuan3d", so Pixal3D is not silently the default). Both review fixes landed via PR #35: TRELLIS.2 setup.sh without --basic, and the WebP guarantee promoted to an artifact-level postcondition (pixal3d.py:85 _WEBP_EXTENSION, checked :248-262, covered by tests/test_pixal3d_backend.py:392-474). #835's tunables are at config.py:112/:117 with the deployed-env-var-name test at tests/test_config.py:447-468 — which closes the Jaeger:EndPoint-class typo gap rather than just asserting the constants.

#838 is why this epic can't close. The infra stack merged (spikersoft-artpipe-model-pixal3d/docker-stack-gpu.yml, infra PR #149) and the tier-2 build case exists (model-env-images.yml:269), but git grep -in "pixal" origin/master returns zero matches in spikersoft-backend — so:

  1. No pixal3d) case in .gitea/workflows/spikersoft-artpipe-processor.yml; it can't be built even on manual dispatch.
  2. spikersoft-artpipe-model-pixal3d is absent from the deploy loop at :548-553 — the merged stack file is inert.
  3. No Key = "pixal3d" in ArtStudioStageMethodOptions.CreateDefault(), so it can't be selected.

Net: the backend half of the cutover was never written, which is also why every verification checkbox across all three children is unrun — there is currently no path from a user request to the Pixal3D queue.

Remaining, in order: the three backend items on #838 (small, mechanical), then the 4090 runs that #834/#835 are waiting on — artpipe setup --model Pixal3D, a real POST /jobs timing/VRAM run with the overrides exported, and the Blender 5.1 GLB PBR round-trip.

One caveat worth carrying: #615 means an infra deploy can silently ship a stale stack file, so once #838's deploy-loop entry lands, confirm the running service directly rather than trusting a green deploy.

**Epic status roll-up** — audited all three children against `origin/master` / `origin/main`. Staying open: the code is done, but nothing can route to it yet. | Child | State | |---|---| | #834 PR-1: artpipe manifest + WRAP backend + runner | ✅ code complete; 4090 hardware runs unrun | | #835 PR-2: tunable VRAM admission gate | ✅ code complete; 4090 confirmation unrun | | #838 PR-5: deploy stack, method option, routing cutover | ⚠️ **PARTIAL — the stack is inert** | **#834 / #835 are genuinely done in code.** `models/Pixal3D/artpipe.json` (image_to_3d, `vram_mb: 18000`, py3.10/cu128, `ATTN_BACKEND=sdpa`, `verify_imports` for every compiled extension); `run_pixal3d.py:64` `force_png_textures()`; `src/artpipe/models/pixal3d.py` `Pixal3DBackend`; `config.py:130` `TIMEOUT_PIXAL3D = 2700` and `:404` pixal3d in the CUDA `allowed_models["modeling"]` — with `:391` confirming the stated negative constraint holds (`stage_defaults["modeling"] = "hunyuan3d"`, so Pixal3D is not silently the default). Both review fixes landed via PR #35: TRELLIS.2 `setup.sh` without `--basic`, and the WebP guarantee promoted to an artifact-level postcondition (`pixal3d.py:85` `_WEBP_EXTENSION`, checked `:248-262`, covered by `tests/test_pixal3d_backend.py:392-474`). #835's tunables are at `config.py:112`/`:117` with the deployed-env-var-name test at `tests/test_config.py:447-468` — which closes the `Jaeger:EndPoint`-class typo gap rather than just asserting the constants. **#838 is why this epic can't close.** The infra stack merged (`spikersoft-artpipe-model-pixal3d/docker-stack-gpu.yml`, infra PR #149) and the tier-2 build case exists (`model-env-images.yml:269`), but `git grep -in "pixal" origin/master` returns **zero matches in spikersoft-backend** — so: 1. No `pixal3d)` case in `.gitea/workflows/spikersoft-artpipe-processor.yml`; it can't be built even on manual dispatch. 2. `spikersoft-artpipe-model-pixal3d` is absent from the deploy loop at `:548-553` — **the merged stack file is inert**. 3. No `Key = "pixal3d"` in `ArtStudioStageMethodOptions.CreateDefault()`, so it can't be selected. Net: the backend half of the cutover was never written, which is also why every verification checkbox across all three children is unrun — there is currently no path from a user request to the Pixal3D queue. **Remaining, in order:** the three backend items on #838 (small, mechanical), then the 4090 runs that #834/#835 are waiting on — `artpipe setup --model Pixal3D`, a real `POST /jobs` timing/VRAM run with the overrides exported, and the Blender 5.1 GLB PBR round-trip. One caveat worth carrying: **#615** means an infra deploy can silently ship a stale stack file, so once #838's deploy-loop entry lands, confirm the running service directly rather than trusting a green deploy.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#833