MeshAnything：使用自回归Transformer进行艺术家创建的网格生成

MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers

June 14, 2024

作者: Yiwen Chen, Tong He, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, Guosheng Lin, Chi Zhang

cs.AI

摘要

最近，通过重建和生成创建的3D资产已经达到了手工制作资产的质量，突显了它们作为替代品的潜力。然而，这种潜力在很大程度上尚未得到实现，因为这些资产总是需要转换为网格以用于3D行业应用，而当前网格提取方法生成的网格明显劣于由人类艺术家创建的网格（AMs）。具体而言，当前的网格提取方法依赖于密集面并忽略几何特征，导致低效率、复杂的后处理和较低的表示质量。为了解决这些问题，我们引入了MeshAnything，这是一个将网格提取视为生成问题的模型，可以生成与指定形状对齐的AMs。通过将任何3D表示中的3D资产转换为AMs，MeshAnything可以与各种3D资产生产方法集成，从而增强它们在整个3D行业中的应用。MeshAnything的架构包括一个VQ-VAE和一个形状条件的仅解码器变压器。我们首先使用VQ-VAE学习网格词汇，然后在此词汇上训练形状条件的仅解码器变压器，用于形状条件的自回归网格生成。我们广泛的实验证明，我们的方法生成的AMs面数少了数百倍，显著提高了存储、渲染和模拟效率，同时实现了与先前方法可比的精度。

English

Recently, 3D assets created via reconstruction and generation have matched the quality of manually crafted assets, highlighting their potential for replacement. However, this potential is largely unrealized because these assets always need to be converted to meshes for 3D industry applications, and the meshes produced by current mesh extraction methods are significantly inferior to Artist-Created Meshes (AMs), i.e., meshes created by human artists. Specifically, current mesh extraction methods rely on dense faces and ignore geometric features, leading to inefficiencies, complicated post-processing, and lower representation quality. To address these issues, we introduce MeshAnything, a model that treats mesh extraction as a generation problem, producing AMs aligned with specified shapes. By converting 3D assets in any 3D representation into AMs, MeshAnything can be integrated with various 3D asset production methods, thereby enhancing their application across the 3D industry. The architecture of MeshAnything comprises a VQ-VAE and a shape-conditioned decoder-only transformer. We first learn a mesh vocabulary using the VQ-VAE, then train the shape-conditioned decoder-only transformer on this vocabulary for shape-conditioned autoregressive mesh generation. Our extensive experiments show that our method generates AMs with hundreds of times fewer faces, significantly improving storage, rendering, and simulation efficiencies, while achieving precision comparable to previous methods.