MetaDreamer：使用几何和纹理解耦的高效文本生成3D模型

MetaDreamer: Efficient Text-to-3D Creation With Disentangling Geometry and Texture

November 16, 2023

作者: Lincong Feng, Muyu Wang, Maoyu Wang, Kuo Xu, Xiaoli Liu

cs.AI

摘要

通过从2D扩散模型中提炼的先验知识，生成模型在3D物体合成方面取得了显著进展。然而，在现有的3D合成框架中仍存在多视角几何不一致和生成速度缓慢的挑战。这可以归因于两个因素：首先，在优化中几何先验知识不足，其次是传统3D生成方法中几何和纹理之间的纠缠问题。作为回应，我们引入了MetaDreammer，这是一种利用丰富的2D和3D先验知识的两阶段优化方法。在第一阶段，我们重点优化几何表示，以确保3D物体的多视角一致性和准确性。在第二阶段，我们集中于微调几何和优化纹理，从而实现更精细的3D物体。通过分别利用两个阶段的2D和3D先验知识，我们有效地缓解了几何和纹理之间的相互依赖关系。MetaDreamer为每个阶段建立了明确的优化目标，从而在3D生成过程中节省了大量时间。最终，MetaDreamer可以基于文本提示在20分钟内生成高质量的3D物体，并据我们所知，这是最高效的文本到3D生成方法。此外，我们将图像控制引入到该过程中，增强了3D生成的可控性。大量经验证据证实，我们的方法不仅高效，而且达到了当前最先进的3D生成技术水平。

English

Generative models for 3D object synthesis have seen significant advancements with the incorporation of prior knowledge distilled from 2D diffusion models. Nevertheless, challenges persist in the form of multi-view geometric inconsistencies and slow generation speeds within the existing 3D synthesis frameworks. This can be attributed to two factors: firstly, the deficiency of abundant geometric a priori knowledge in optimization, and secondly, the entanglement issue between geometry and texture in conventional 3D generation methods.In response, we introduce MetaDreammer, a two-stage optimization approach that leverages rich 2D and 3D prior knowledge. In the first stage, our emphasis is on optimizing the geometric representation to ensure multi-view consistency and accuracy of 3D objects. In the second stage, we concentrate on fine-tuning the geometry and optimizing the texture, thereby achieving a more refined 3D object. Through leveraging 2D and 3D prior knowledge in two stages, respectively, we effectively mitigate the interdependence between geometry and texture. MetaDreamer establishes clear optimization objectives for each stage, resulting in significant time savings in the 3D generation process. Ultimately, MetaDreamer can generate high-quality 3D objects based on textual prompts within 20 minutes, and to the best of our knowledge, it is the most efficient text-to-3D generation method. Furthermore, we introduce image control into the process, enhancing the controllability of 3D generation. Extensive empirical evidence confirms that our method is not only highly efficient but also achieves a quality level that is at the forefront of current state-of-the-art 3D generation techniques.