CraftsMan：通过3D本地生成和交互式几何细化实现高保真度网格生成

CraftsMan: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refiner

May 23, 2024

作者: Weiyu Li, Jiarui Liu, Rui Chen, Yixun Liang, Xuelin Chen, Ping Tan, Xiaoxiao Long

cs.AI

摘要

我们提出了一种新颖的生成式3D建模系统，命名为CraftsMan，能够生成具有高保真度、高度多样形状、规则网格拓扑和详细表面的3D几何体，并且显著地允许以交互方式对几何体进行细化。尽管在3D生成方面取得了显著进展，但现有方法仍然面临着优化过程冗长、网格拓扑不规则、表面嘈杂以及难以适应用户编辑等问题，从而阻碍了它们在3D建模软件中的广泛应用和实施。我们的工作受到工匠的启发，通常先勾勒出作品的整体轮廓，然后再详细阐述表面细节。具体而言，我们采用了一种3D本地扩散模型，该模型在从潜在集合式3D表示学习的潜在空间上运行，以在几秒钟内生成具有规则网格拓扑的粗略几何体。特别是，该过程以文本提示或参考图像作为输入，并利用强大的多视角（MV）扩散模型生成粗略几何体的多个视图，这些视图被馈送到我们的MV条件的3D扩散模型中，用于生成3D几何体，显著提高了鲁棒性和泛化能力。随后，使用基于法线的几何体细化器显著增强表面细节。这种细化可以自动进行，也可以与用户提供的编辑交互进行。大量实验证明，与现有方法相比，我们的方法在生成优质3D资产方面表现出高效性。主页：https://craftsman3d.github.io/，代码：https://github.com/wyysf-98/CraftsMan

English

We present a novel generative 3D modeling system, coined CraftsMan, which can generate high-fidelity 3D geometries with highly varied shapes, regular mesh topologies, and detailed surfaces, and, notably, allows for refining the geometry in an interactive manner. Despite the significant advancements in 3D generation, existing methods still struggle with lengthy optimization processes, irregular mesh topologies, noisy surfaces, and difficulties in accommodating user edits, consequently impeding their widespread adoption and implementation in 3D modeling software. Our work is inspired by the craftsman, who usually roughs out the holistic figure of the work first and elaborates the surface details subsequently. Specifically, we employ a 3D native diffusion model, which operates on latent space learned from latent set-based 3D representations, to generate coarse geometries with regular mesh topology in seconds. In particular, this process takes as input a text prompt or a reference image and leverages a powerful multi-view (MV) diffusion model to generate multiple views of the coarse geometry, which are fed into our MV-conditioned 3D diffusion model for generating the 3D geometry, significantly improving robustness and generalizability. Following that, a normal-based geometry refiner is used to significantly enhance the surface details. This refinement can be performed automatically, or interactively with user-supplied edits. Extensive experiments demonstrate that our method achieves high efficacy in producing superior-quality 3D assets compared to existing methods. HomePage: https://craftsman3d.github.io/, Code: https://github.com/wyysf-98/CraftsMan