Cycle3D：透過生成-重建循環實現高質量且一致的圖像到3D生成

Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle

July 28, 2024

作者: Zhenyu Tang, Junwu Zhang, Xinhua Cheng, Wangbo Yu, Chaoran Feng, Yatian Pang, Bin Lin, Li Yuan

cs.AI

摘要

最近的3D大型重建模型通常採用兩階段過程，首先通過多視角擴散模型生成多視角影像，然後利用前饋模型將影像重建為3D內容。然而，多視角擴散模型通常會產生質量低且不一致的影像，對最終3D重建的質量產生不利影響。為解決這個問題，我們提出了一個統一的3D生成框架，稱為Cycle3D，它在多步驟擴散過程中循環利用2D基於擴散的生成模組和前饋3D重建模組。具體而言，2D擴散模型用於生成高質量紋理，而重建模型則確保多視角一致性。此外，2D擴散模型可以進一步控制生成的內容，並為未見視角注入參考視角信息，從而在去噪過程中增強3D生成的多樣性和紋理一致性。大量實驗證明，與最先進的基準方法相比，我們的方法在創建具有高質量和一致性的3D內容方面具有卓越能力。

English

Recent 3D large reconstruction models typically employ a two-stage process, including first generate multi-view images by a multi-view diffusion model, and then utilize a feed-forward model to reconstruct images to 3D content.However, multi-view diffusion models often produce low-quality and inconsistent images, adversely affecting the quality of the final 3D reconstruction. To address this issue, we propose a unified 3D generation framework called Cycle3D, which cyclically utilizes a 2D diffusion-based generation module and a feed-forward 3D reconstruction module during the multi-step diffusion process. Concretely, 2D diffusion model is applied for generating high-quality texture, and the reconstruction model guarantees multi-view consistency.Moreover, 2D diffusion model can further control the generated content and inject reference-view information for unseen views, thereby enhancing the diversity and texture consistency of 3D generation during the denoising process. Extensive experiments demonstrate the superior ability of our method to create 3D content with high-quality and consistency compared with state-of-the-art baselines.