spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[ArtStudio M3] Metal extension reuse — TRELLIS via trellis-mac, then mtldiffrast/mtlgemm for InstantMesh, Hunyuan paint, UniRig #358

New Issue
Open
opened 2026-07-04 05:28:07 +00:00 by spikerj · 4 comments
spikerj commented 2026-07-04 05:28:07 +00:00
Owner
Copy Link
Copy Source

Sub-ticket of epic #346, Mac workstream. Depends on #356 (device seam); upgrades the #357 profile. Absorbs the implementation half of spike #355.

What: bring the CUDA-extension-blocked backends to Apple Silicon by reusing trellis-mac's Metal drop-ins. trellis-mac already reimplemented exactly the kernels that block them: mtldiffrast ← nvdiffrast, mtlgemm ← flex_gemm/spconv, mtlmesh ← cumesh, mtlbvh ← cubvh (all zero-copy on unified memory, autograd-compatible, API drop-ins under trellis-mac/deps/).

Scope (in value order):

  • TRELLIS modeling on Mac — the flagship: register trellis-mac as an art_pipe backend (own venv + artpipe.json manifest with Darwin overrides, per art_pipe's per-model isolation). Decide wrap (trellis-generate CLI behind the worker protocol) vs port (apply patches/mps_compat.py recipe inside art_pipe's trellis backend) — wrap first, port later. Reconcile python versions (art_pipe cu-venvs 3.10 vs trellis-mac 3.11 — fine, venvs are per-model)
  • Package the four Metal deps as installable wheels (or a shared source checkout) consumable by other backend venvs
  • InstantMesh: swap nvdiffrast → mtldiffrast; fp16 fix from #356
  • Hunyuan3D texture paint (hy3dgen.texgen custom_rasterizer + DifferentiableRenderer → mtldiffrast) — unlocks Hunyuan3D-2 full shape+paint on Mac, the biggest texture-quality jump over SD-Turbo-Tex
  • UniRig: spconv → mtlgemm (+ flash_attn already auto-falls back; sdpa attention route per trellis-mac patch_sparse_attention) — real auto-rigging on Mac instead of smart_rig-only
  • Out of scope, document why: TripoSG (fvdb has no Metal port — no drop-in path)
  • Upstream conversation with Eric: where do Metal deps live long-term (ProArt optional extra? separate repo?)

Note: each backend is independently shippable — land TRELLIS first and update the #357 macos-mps profile's modeling default when it proves out. Benchmark each on the M5 (128 GB) and post to the #354 matrix.

Acceptance: TRELLIS image→3D produces a textured GLB on the M5 through the normal art_pipe worker protocol; at least one of InstantMesh / Hunyuan-paint / UniRig also runs on Metal; the macos-mps profile is updated with measured defaults; TripoSG documented as CUDA-only.

Sub-ticket of epic #346, Mac workstream. Depends on #356 (device seam); upgrades the #357 profile. Absorbs the implementation half of spike #355. **What:** bring the CUDA-extension-blocked backends to Apple Silicon by reusing trellis-mac's Metal drop-ins. trellis-mac already reimplemented exactly the kernels that block them: `mtldiffrast` ← nvdiffrast, `mtlgemm` ← flex_gemm/spconv, `mtlmesh` ← cumesh, `mtlbvh` ← cubvh (all zero-copy on unified memory, autograd-compatible, API drop-ins under `trellis-mac/deps/`). **Scope (in value order):** - [ ] **TRELLIS modeling on Mac** — the flagship: register trellis-mac as an art_pipe backend (own venv + `artpipe.json` manifest with Darwin overrides, per art_pipe's per-model isolation). Decide wrap (`trellis-generate` CLI behind the worker protocol) vs port (apply `patches/mps_compat.py` recipe inside art_pipe's trellis backend) — wrap first, port later. Reconcile python versions (art_pipe cu-venvs 3.10 vs trellis-mac 3.11 — fine, venvs are per-model) - [ ] Package the four Metal deps as installable wheels (or a shared source checkout) consumable by other backend venvs - [ ] **InstantMesh**: swap nvdiffrast → mtldiffrast; fp16 fix from #356 - [ ] **Hunyuan3D texture paint** (`hy3dgen.texgen` custom_rasterizer + DifferentiableRenderer → mtldiffrast) — unlocks Hunyuan3D-2 full shape+paint on Mac, the biggest texture-quality jump over SD-Turbo-Tex - [ ] **UniRig**: spconv → mtlgemm (+ flash_attn already auto-falls back; sdpa attention route per trellis-mac `patch_sparse_attention`) — real auto-rigging on Mac instead of smart_rig-only - [ ] Out of scope, document why: TripoSG (fvdb has no Metal port — no drop-in path) - [ ] Upstream conversation with Eric: where do Metal deps live long-term (ProArt optional extra? separate repo?) **Note:** each backend is independently shippable — land TRELLIS first and update the #357 macos-mps profile's modeling default when it proves out. Benchmark each on the M5 (128 GB) and post to the #354 matrix. **Acceptance:** TRELLIS image→3D produces a textured GLB on the M5 through the normal art_pipe worker protocol; at least one of InstantMesh / Hunyuan-paint / UniRig also runs on Metal; the macos-mps profile is updated with measured defaults; TripoSG documented as CUDA-only.
trellis_mac_dragon.glb.zip
6.1 MiB
spikerj added the enhancement label 2026-07-04 05:28:07 +00:00
spikerj commented 2026-07-04 21:23:11 +00:00
Author
Owner
Copy Link
Copy Source

Provenance note (Joey, 2026-07-04): our trellis-mac checkout is a fork of shivampkumar/trellis-mac — the upstream CUDA→Apple-Silicon (PyTorch MPS) port of Microsoft TRELLIS.2, including the Metal reimplementations (mtldiffrast/mtlgemm/mtlmesh/mtlbvh) this ticket plans to reuse. The in-flight slice-1 work (TrellisMac art_pipe backend wrap) has been instructed to: verify the fork's LICENSE terms before wrapping, carry explicit attribution (Microsoft TRELLIS.2 → shivampkumar/trellis-mac → our fork) in the backend module, manifest, and validation docs, and report any upstream README/setup drift relative to our fork. Anything we improve at the wrap seam should be considered for upstreaming to that project as well as to Eric's ProArt.

**Provenance note (Joey, 2026-07-04):** our `trellis-mac` checkout is a fork of [shivampkumar/trellis-mac](https://github.com/shivampkumar/trellis-mac) — the upstream CUDA→Apple-Silicon (PyTorch MPS) port of Microsoft TRELLIS.2, including the Metal reimplementations (`mtldiffrast`/`mtlgemm`/`mtlmesh`/`mtlbvh`) this ticket plans to reuse. The in-flight slice-1 work (TrellisMac art_pipe backend wrap) has been instructed to: verify the fork's LICENSE terms before wrapping, carry explicit attribution (Microsoft TRELLIS.2 → shivampkumar/trellis-mac → our fork) in the backend module, manifest, and validation docs, and report any upstream README/setup drift relative to our fork. Anything we improve at the wrap seam should be considered for upstreaming to that project as well as to Eric's ProArt.
spikerj commented 2026-07-04 22:11:22 +00:00
Author
Owner
Copy Link
Copy Source

First TRELLIS.2 generation on the M5 through the platform seam — output attached: trellis_mac_dragon.glb.zip (9.0 MB GLB zipped to 6.1 MB; Gitea's attachment allowlist rejects bare .glb, hence the zip). Generated from the dragon concept image via the new TrellisMac art_pipe backend wrap (trellis-mac MPS port, upstream shivampkumar/trellis-mac). Drop it into any glTF viewer to compare against the TripoSR output of the same image. The implementing agent's full validation numbers (wall time, verts, quality notes) land with its report shortly.

**First TRELLIS.2 generation on the M5 through the platform seam** — output attached: [trellis_mac_dragon.glb.zip](https://git.spikersoft.com/attachments/3258fc8b-8dc3-4713-bc2e-c1bafb110e66) (9.0 MB GLB zipped to 6.1 MB; Gitea's attachment allowlist rejects bare `.glb`, hence the zip). Generated from the dragon concept image via the new TrellisMac art_pipe backend wrap (trellis-mac MPS port, upstream shivampkumar/trellis-mac). Drop it into any glTF viewer to compare against the TripoSR output of the same image. The implementing agent's full validation numbers (wall time, verts, quality notes) land with its report shortly.
spikerj commented 2026-07-04 22:16:38 +00:00
Author
Owner
Copy Link
Copy Source

Slice 1 delivered (art_pipe commit a552717 on the device branch; 682 pytest passing, independently re-verified). Validation numbers for the attached GLB:

Run Wall Output
seed 42 155.4 s 8.98 MB GLB — 142k verts, UV-mapped, 1024² PBR (baseColor + metallicRoughness)
seed 123 170.8 s 9.54 MB GLB

Peak RSS 18.6 GB (matches upstream's ~18 GB claim), ~2× faster than the upstream M4 Pro reference. Quality vs TripoSR on the same concept: decisive — instantly reads as the chibi dragon with materials, vs TripoSR's vertex-color blob — at a 156 s vs 11 s trade-off. trellis_mac added to macos-mps allowed_models; triposr stays modeling default pending the quality/latency call. Attribution chain (Microsoft TRELLIS.2 MIT → shivampkumar/trellis-mac MIT → our fork) embedded in module/manifest/docs; our fork is byte-identical with upstream origin/main — no drift, nothing needing upstreaming.

⚠️ Licensing flags for the platform gate (Joey):

  • RMBG-2.0 (background removal) is CC BY-NC 4.0 — commercial use requires a BRIA license. This sits in the trellis_mac path; needs a decision before the backend serves paying-customer traffic (options: BRIA license, swap the rembg step, or keep trellis_mac staff-only).
  • DINOv3 weights are Meta-gated (access granted on our HF account).

Ops notes: fresh Macs need xcodebuild -downloadComponent MetalToolchain before setup.sh's Metal rebuilds (prebuilt wheels carried us here); known upstream mesh flaws (small shell holes — no cumesh hole-fill on Metal yet). Deferred: warm-worker (~80 s/job reload), cancel-orphan gap (shared with the Blender pseudo-backend), the PORT half of this ticket, InstantMesh/Hunyuan-paint/UniRig Metal reuse.

Slice 1 delivered (art_pipe commit `a552717` on the device branch; **682 pytest passing**, independently re-verified). Validation numbers for the attached GLB: | Run | Wall | Output | |---|---|---| | seed 42 | **155.4 s** | 8.98 MB GLB — 142k verts, UV-mapped, 1024² PBR (baseColor + metallicRoughness) | | seed 123 | 170.8 s | 9.54 MB GLB | Peak RSS 18.6 GB (matches upstream's ~18 GB claim), **~2× faster than the upstream M4 Pro reference**. Quality vs TripoSR on the same concept: decisive — instantly reads as the chibi dragon with materials, vs TripoSR's vertex-color blob — at a 156 s vs 11 s trade-off. `trellis_mac` added to macos-mps `allowed_models`; **triposr stays modeling default** pending the quality/latency call. Attribution chain (Microsoft TRELLIS.2 MIT → shivampkumar/trellis-mac MIT → our fork) embedded in module/manifest/docs; our fork is byte-identical with upstream `origin/main` — no drift, nothing needing upstreaming. **⚠️ Licensing flags for the platform gate (Joey):** - **RMBG-2.0 (background removal) is CC BY-NC 4.0 — commercial use requires a BRIA license.** This sits in the trellis_mac path; needs a decision before the backend serves paying-customer traffic (options: BRIA license, swap the rembg step, or keep trellis_mac staff-only). - DINOv3 weights are Meta-gated (access granted on our HF account). Ops notes: fresh Macs need `xcodebuild -downloadComponent MetalToolchain` before setup.sh's Metal rebuilds (prebuilt wheels carried us here); known upstream mesh flaws (small shell holes — no cumesh hole-fill on Metal yet). Deferred: warm-worker (~80 s/job reload), cancel-orphan gap (shared with the Blender pseudo-backend), the PORT half of this ticket, InstantMesh/Hunyuan-paint/UniRig Metal reuse.
spikerj commented 2026-07-04 23:43:27 +00:00
Author
Owner
Copy Link
Copy Source

Licensing resolution (Joey, 2026-07-05): SpikerSoft is a non-profit — no one pays; it exists for the benefit of kids learning. RMBG-2.0's non-commercial restriction is therefore satisfied for our use, and the trellis_mac backend is cleared for student-facing use on that basis (attribution retained per the license and our embedded credit chain). Standing caveat recorded: if any revenue model ever appears (e.g. charging schools), this specific dependency must be re-evaluated / BRIA-licensed / swapped — the flag lives here so it's findable. DINOv3 gated access remains tied to our HF account.

With licensing cleared, the remaining modeling-default question (triposr 11 s draft vs trellis_mac ~156 s quality) is purely a UX/queue-budget call — a "quality mode" toggle in the composer (draft default, TRELLIS for the final pass) may serve kids better than either extreme. Remaining ticket scope: that decision, the PORT half, and InstantMesh/Hunyuan-paint/UniRig Metal reuse.

**Licensing resolution (Joey, 2026-07-05):** SpikerSoft is a non-profit — no one pays; it exists for the benefit of kids learning. RMBG-2.0's non-commercial restriction is therefore satisfied for our use, and the trellis_mac backend is **cleared for student-facing use** on that basis (attribution retained per the license and our embedded credit chain). Standing caveat recorded: if any revenue model ever appears (e.g. charging schools), this specific dependency must be re-evaluated / BRIA-licensed / swapped — the flag lives here so it's findable. DINOv3 gated access remains tied to our HF account. With licensing cleared, the remaining modeling-default question (`triposr` 11 s draft vs `trellis_mac` ~156 s quality) is purely a UX/queue-budget call — a "quality mode" toggle in the composer (draft default, TRELLIS for the final pass) may serve kids better than either extreme. Remaining ticket scope: that decision, the PORT half, and InstantMesh/Hunyuan-paint/UniRig Metal reuse.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#358