MeshCraft：探索基於流式擴散變換器的高效可控網格生成

MeshCraft: Exploring Efficient and Controllable Mesh Generation with Flow-based DiTs

March 29, 2025

作者: Xianglong He, Junyi Chen, Di Huang, Zexiang Liu, Xiaoshui Huang, Wanli Ouyang, Chun Yuan, Yangguang Li

cs.AI

摘要

在3D內容創作領域，通過AI模型實現最佳網格拓撲結構一直是3D藝術家們追求的目標。先前的方法，如MeshGPT，已探索了通過網格自迴歸技術生成即用型3D物體。雖然這些方法產生了視覺上令人印象深刻的結果，但由於其在自迴歸過程中依賴於逐個標記的預測，導致了幾個顯著的限制。這些限制包括極慢的生成速度和無法控制的網格面數。本文中，我們介紹了MeshCraft，這是一個用於高效且可控網格生成的新框架，它利用連續空間擴散來生成離散的三角形面。具體而言，MeshCraft由兩個核心組件構成：1）一個基於變換器的VAE，它將原始網格編碼為連續的面級標記並將其解碼回原始網格；2）一個基於流的擴散變換器，該變換器以面數為條件，能夠生成具有預定義面數的高質量3D網格。通過利用擴散模型同時生成整個網格拓撲結構，MeshCraft在顯著快於自迴歸方法的速度下實現了高保真網格生成。具體來說，MeshCraft能在僅3.2秒內生成一個800面的網格（比現有基線快35倍）。大量實驗表明，MeshCraft在ShapeNet數據集上的質量和數量評估中均優於最先進的技術，並在Objaverse數據集上展示了卓越的性能。此外，它與現有的條件指導策略無縫集成，展示了其潛力，能夠減輕藝術家在網格創建中耗時的手動工作。

English

In the domain of 3D content creation, achieving optimal mesh topology through AI models has long been a pursuit for 3D artists. Previous methods, such as MeshGPT, have explored the generation of ready-to-use 3D objects via mesh auto-regressive techniques. While these methods produce visually impressive results, their reliance on token-by-token predictions in the auto-regressive process leads to several significant limitations. These include extremely slow generation speeds and an uncontrollable number of mesh faces. In this paper, we introduce MeshCraft, a novel framework for efficient and controllable mesh generation, which leverages continuous spatial diffusion to generate discrete triangle faces. Specifically, MeshCraft consists of two core components: 1) a transformer-based VAE that encodes raw meshes into continuous face-level tokens and decodes them back to the original meshes, and 2) a flow-based diffusion transformer conditioned on the number of faces, enabling the generation of high-quality 3D meshes with a predefined number of faces. By utilizing the diffusion model for the simultaneous generation of the entire mesh topology, MeshCraft achieves high-fidelity mesh generation at significantly faster speeds compared to auto-regressive methods. Specifically, MeshCraft can generate an 800-face mesh in just 3.2 seconds (35times faster than existing baselines). Extensive experiments demonstrate that MeshCraft outperforms state-of-the-art techniques in both qualitative and quantitative evaluations on ShapeNet dataset and demonstrates superior performance on Objaverse dataset. Moreover, it integrates seamlessly with existing conditional guidance strategies, showcasing its potential to relieve artists from the time-consuming manual work involved in mesh creation.