MeshAnything：藝術家創建的網格生成與自回歸Transformer

MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers

June 14, 2024

作者: Yiwen Chen, Tong He, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, Guosheng Lin, Chi Zhang

cs.AI

摘要

最近，通過重建和生成創建的3D資產已經達到了手工製作資產的質量水平，突顯了它們作為替代品的潛力。然而，這種潛力主要尚未實現，因為這些資產總是需要轉換為網格以供3D行業應用，而當前的網格提取方法生成的網格明顯遠遜於藝術家創建的網格（AMs），即由人類藝術家創建的網格。具體來說，當前的網格提取方法依賴於密集的面並忽略幾何特徵，導致效率低下、後處理複雜且表示質量較低。為了解決這些問題，我們引入了MeshAnything，這是一個將網格提取視為生成問題的模型，生成與指定形狀對齊的AMs。通過將任何3D表示中的3D資產轉換為AMs，MeshAnything可以與各種3D資產生產方法集成，從而增強它們在3D行業中的應用。MeshAnything的架構包括一個VQ-VAE和一個僅解碼器的形狀條件化Transformer。我們首先使用VQ-VAE學習網格詞彙，然後在這個詞彙上訓練形狀條件化的解碼器Transformer，用於形狀條件化的自回歸網格生成。我們的大量實驗表明，我們的方法生成的AMs具有數百倍更少的面，顯著提高了存儲、渲染和模擬效率，同時實現了與先前方法可比的精度。

English

Recently, 3D assets created via reconstruction and generation have matched the quality of manually crafted assets, highlighting their potential for replacement. However, this potential is largely unrealized because these assets always need to be converted to meshes for 3D industry applications, and the meshes produced by current mesh extraction methods are significantly inferior to Artist-Created Meshes (AMs), i.e., meshes created by human artists. Specifically, current mesh extraction methods rely on dense faces and ignore geometric features, leading to inefficiencies, complicated post-processing, and lower representation quality. To address these issues, we introduce MeshAnything, a model that treats mesh extraction as a generation problem, producing AMs aligned with specified shapes. By converting 3D assets in any 3D representation into AMs, MeshAnything can be integrated with various 3D asset production methods, thereby enhancing their application across the 3D industry. The architecture of MeshAnything comprises a VQ-VAE and a shape-conditioned decoder-only transformer. We first learn a mesh vocabulary using the VQ-VAE, then train the shape-conditioned decoder-only transformer on this vocabulary for shape-conditioned autoregressive mesh generation. Our extensive experiments show that our method generates AMs with hundreds of times fewer faces, significantly improving storage, rendering, and simulation efficiencies, while achieving precision comparable to previous methods.