CraftsMan：透過3D本機生成和互動幾何細化進行高保真度網格生成

CraftsMan: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refiner

May 23, 2024

作者: Weiyu Li, Jiarui Liu, Rui Chen, Yixun Liang, Xuelin Chen, Ping Tan, Xiaoxiao Long

cs.AI

摘要

我們提出了一種新穎的生成式3D建模系統，名為CraftsMan，可以生成具有高保真度的3D幾何形狀，具有高度多樣化的形狀、規則網格拓撲和詳細表面，並且特別是可以通過互動方式來精緻幾何形狀。儘管在3D生成方面取得了顯著進展，但現有方法仍然在優化過程冗長、網格拓撲不規則、表面帶有噪點以及難以適應用戶編輯等方面存在困難，因此阻礙了它們在3D建模軟件中的廣泛應用和實施。我們的工作受到工匠的啟發，通常先粗略勾勒出作品的整體輪廓，然後再細緻呈現表面細節。具體來說，我們採用了一種3D本地擴散模型，它在從潛在集合式3D表示學習的潛在空間上運行，以在幾秒鐘內生成具有規則網格拓撲的粗略幾何形狀。特別是，此過程以文本提示或參考圖像作為輸入，並利用強大的多視圖（MV）擴散模型生成粗略幾何形狀的多個視圖，這些視圖被輸入到我們的MV條件3D擴散模型中，用於生成3D幾何形狀，從而顯著提高了韌性和泛化能力。隨後，使用基於法線的幾何細化器來顯著增強表面細節。這種細化可以自動執行，也可以與用戶提供的編輯進行互動。大量實驗表明，與現有方法相比，我們的方法在生成優質3D資產方面取得了高效果。首頁：https://craftsman3d.github.io/，代碼：https://github.com/wyysf-98/CraftsMan

English

We present a novel generative 3D modeling system, coined CraftsMan, which can generate high-fidelity 3D geometries with highly varied shapes, regular mesh topologies, and detailed surfaces, and, notably, allows for refining the geometry in an interactive manner. Despite the significant advancements in 3D generation, existing methods still struggle with lengthy optimization processes, irregular mesh topologies, noisy surfaces, and difficulties in accommodating user edits, consequently impeding their widespread adoption and implementation in 3D modeling software. Our work is inspired by the craftsman, who usually roughs out the holistic figure of the work first and elaborates the surface details subsequently. Specifically, we employ a 3D native diffusion model, which operates on latent space learned from latent set-based 3D representations, to generate coarse geometries with regular mesh topology in seconds. In particular, this process takes as input a text prompt or a reference image and leverages a powerful multi-view (MV) diffusion model to generate multiple views of the coarse geometry, which are fed into our MV-conditioned 3D diffusion model for generating the 3D geometry, significantly improving robustness and generalizability. Following that, a normal-based geometry refiner is used to significantly enhance the surface details. This refinement can be performed automatically, or interactively with user-supplied edits. Extensive experiments demonstrate that our method achieves high efficacy in producing superior-quality 3D assets compared to existing methods. HomePage: https://craftsman3d.github.io/, Code: https://github.com/wyysf-98/CraftsMan