魔法增強：使用多視角條件擴散提升3D生成

Magic-Boost: Boost 3D Generation with Mutli-View Conditioned Diffusion

April 9, 2024

作者: Fan Yang, Jianfeng Zhang, Yichun Shi, Bowen Chen, Chenxu Zhang, Huichao Zhang, Xiaofeng Yang, Jiashi Feng, Guosheng Lin

cs.AI

摘要

受惠於2D擴散模型的快速發展，最近3D內容創作取得了顯著進展。一種有前途的解決方案涉及微調預訓練的2D擴散模型，以利用其產生多視角影像的能力，然後通過快速NeRFs或大型重建模型等方法轉換為準確的3D模型。然而，由於仍存在不一致性和生成分辨率有限，這些方法生成的結果仍然缺乏精細紋理和複雜幾何形狀。為解決這個問題，我們提出了Magic-Boost，一種多視角條件擴散模型，通過短暫的SDS優化（約15分鐘）顯著改進粗糙的生成結果。與先前基於文本或單張圖像的擴散模型相比，Magic-Boost展現出強大的能力，能夠從虛擬合成的多視角影像中生成具有高一致性的圖像。它提供精確的SDS指導，與輸入圖像的特徵相吻合，豐富了初始生成結果的幾何和紋理的局部細節。大量實驗表明，Magic-Boost極大地增強了粗糙的輸入，生成具有豐富幾何和紋理細節的高質量3D資產。 （專案頁面：https://magic-research.github.io/magic-boost/)

English

Benefiting from the rapid development of 2D diffusion models, 3D content creation has made significant progress recently. One promising solution involves the fine-tuning of pre-trained 2D diffusion models to harness their capacity for producing multi-view images, which are then lifted into accurate 3D models via methods like fast-NeRFs or large reconstruction models. However, as inconsistency still exists and limited generated resolution, the generation results of such methods still lack intricate textures and complex geometries. To solve this problem, we propose Magic-Boost, a multi-view conditioned diffusion model that significantly refines coarse generative results through a brief period of SDS optimization (sim15min). Compared to the previous text or single image based diffusion models, Magic-Boost exhibits a robust capability to generate images with high consistency from pseudo synthesized multi-view images. It provides precise SDS guidance that well aligns with the identity of the input images, enriching the local detail in both geometry and texture of the initial generative results. Extensive experiments show Magic-Boost greatly enhances the coarse inputs and generates high-quality 3D assets with rich geometric and textural details. (Project Page: https://magic-research.github.io/magic-boost/)