MeshFormer：具有3D引导重建的高质量网格生成模型

摘要

最近，开放式世界的3D重建模型引起了广泛关注。然而，缺乏足够的3D归纳偏差，现有方法通常需要昂贵的训练成本，并且难以提取高质量的3D网格。在这项工作中，我们介绍了MeshFormer，一种稀疏视图重建模型，明确利用3D本地结构、输入指导和训练监督。具体而言，我们不使用三平面表示，而是将特征存储在3D稀疏体素中，并结合变换器和3D卷积，利用显式3D结构和投影偏差。除了稀疏视图的RGB输入，我们要求网络接收输入并生成相应的法线图。输入法线图可以由2D扩散模型预测，显著有助于几何学习的指导和细化。此外，通过将有符号距离函数（SDF）监督与表面渲染相结合，我们直接学习生成高质量网格，无需复杂的多阶段训练过程。通过结合这些显式的3D偏差，MeshFormer能够高效训练并生成具有细粒度几何细节的高质量纹理网格。它还可以与2D扩散模型集成，实现快速的单图像到3D和文本到3D任务。项目页面：https://meshformer3d.github.io

English

Open-world 3D reconstruction models have recently garnered significant attention. However, without sufficient 3D inductive bias, existing methods typically entail expensive training costs and struggle to extract high-quality 3D meshes. In this work, we introduce MeshFormer, a sparse-view reconstruction model that explicitly leverages 3D native structure, input guidance, and training supervision. Specifically, instead of using a triplane representation, we store features in 3D sparse voxels and combine transformers with 3D convolutions to leverage an explicit 3D structure and projective bias. In addition to sparse-view RGB input, we require the network to take input and generate corresponding normal maps. The input normal maps can be predicted by 2D diffusion models, significantly aiding in the guidance and refinement of the geometry's learning. Moreover, by combining Signed Distance Function (SDF) supervision with surface rendering, we directly learn to generate high-quality meshes without the need for complex multi-stage training processes. By incorporating these explicit 3D biases, MeshFormer can be trained efficiently and deliver high-quality textured meshes with fine-grained geometric details. It can also be integrated with 2D diffusion models to enable fast single-image-to-3D and text-to-3D tasks. Project page: https://meshformer3d.github.io

MeshFormer：具有3D引导重建的高质量网格生成模型

MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model

摘要

Summary

Support

Support