Cycle3D：通过生成-重建循环实现高质量和一致的图像到三维生成

Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle

July 28, 2024

作者: Zhenyu Tang, Junwu Zhang, Xinhua Cheng, Wangbo Yu, Chaoran Feng, Yatian Pang, Bin Lin, Li Yuan

cs.AI

摘要

最近的3D大型重建模型通常采用两阶段过程，首先通过多视角扩散模型生成多视角图像，然后利用前馈模型将图像重建为3D内容。然而，多视角扩散模型经常会产生质量低下且不一致的图像，从而对最终3D重建的质量产生不利影响。为解决这一问题，我们提出了一个统一的3D生成框架称为Cycle3D，该框架在多步扩散过程中循环利用基于2D扩散的生成模块和前馈3D重建模块。具体而言，2D扩散模型用于生成高质量纹理，而重建模型则保证多视角一致性。此外，2D扩散模型可以进一步控制生成的内容，并为未见视角注入参考视角信息，从而增强在去噪过程中3D生成的多样性和纹理一致性。大量实验证明，与最先进的基准方法相比，我们的方法能够以高质量和一致性创建3D内容。

English

Recent 3D large reconstruction models typically employ a two-stage process, including first generate multi-view images by a multi-view diffusion model, and then utilize a feed-forward model to reconstruct images to 3D content.However, multi-view diffusion models often produce low-quality and inconsistent images, adversely affecting the quality of the final 3D reconstruction. To address this issue, we propose a unified 3D generation framework called Cycle3D, which cyclically utilizes a 2D diffusion-based generation module and a feed-forward 3D reconstruction module during the multi-step diffusion process. Concretely, 2D diffusion model is applied for generating high-quality texture, and the reconstruction model guarantees multi-view consistency.Moreover, 2D diffusion model can further control the generated content and inject reference-view information for unseen views, thereby enhancing the diversity and texture consistency of 3D generation during the denoising process. Extensive experiments demonstrate the superior ability of our method to create 3D content with high-quality and consistency compared with state-of-the-art baselines.