MeshFormer：具有3D引導重建模型的高質量網格生成

摘要

最近，開放式世界的3D重建模型引起了相當大的關注。然而，現有方法缺乏足夠的3D歸納偏差，通常需要昂貴的訓練成本，並且難以提取高質量的3D網格。在這項工作中，我們介紹了MeshFormer，一個稀疏視圖重建模型，明確利用3D本地結構、輸入引導和訓練監督。具體來說，我們不使用三平面表示，而是將特徵存儲在3D稀疏體素中，並結合變壓器和3D卷積，以利用明確的3D結構和投影偏差。除了稀疏視圖的RGB輸入外，我們要求網絡接受輸入並生成相應的法向圖。輸入的法向圖可以通過2D擴散模型預測，顯著有助於幫助幾何學習的引導和細化。此外，通過將有符號距離函數（SDF）監督與表面渲染相結合，我們直接學習生成高質量網格，無需複雜的多階段訓練過程。通過結合這些明確的3D偏差，MeshFormer可以高效訓練並生成具有精細幾何細節的高質量紋理網格。它還可以與2D擴散模型集成，實現快速的單圖像到3D和文本到3D任務。項目頁面：https://meshformer3d.github.io

English

Open-world 3D reconstruction models have recently garnered significant attention. However, without sufficient 3D inductive bias, existing methods typically entail expensive training costs and struggle to extract high-quality 3D meshes. In this work, we introduce MeshFormer, a sparse-view reconstruction model that explicitly leverages 3D native structure, input guidance, and training supervision. Specifically, instead of using a triplane representation, we store features in 3D sparse voxels and combine transformers with 3D convolutions to leverage an explicit 3D structure and projective bias. In addition to sparse-view RGB input, we require the network to take input and generate corresponding normal maps. The input normal maps can be predicted by 2D diffusion models, significantly aiding in the guidance and refinement of the geometry's learning. Moreover, by combining Signed Distance Function (SDF) supervision with surface rendering, we directly learn to generate high-quality meshes without the need for complex multi-stage training processes. By incorporating these explicit 3D biases, MeshFormer can be trained efficiently and deliver high-quality textured meshes with fine-grained geometric details. It can also be integrated with 2D diffusion models to enable fast single-image-to-3D and text-to-3D tasks. Project page: https://meshformer3d.github.io

MeshFormer：具有3D引導重建模型的高質量網格生成

MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model

摘要

Summary

Support

Support