魔法增强：利用多视角条件扩散提升3D生成

Magic-Boost: Boost 3D Generation with Mutli-View Conditioned Diffusion

April 9, 2024

作者: Fan Yang, Jianfeng Zhang, Yichun Shi, Bowen Chen, Chenxu Zhang, Huichao Zhang, Xiaofeng Yang, Jiashi Feng, Guosheng Lin

cs.AI

摘要

受益于2D扩散模型的快速发展，最近3D内容创建取得了显著进展。一种有前途的解决方案涉及微调预训练的2D扩散模型，以利用其生成多视角图像的能力，然后通过快速NeRFs或大型重建模型等方法将其提升为准确的3D模型。然而，由于仍然存在不一致性和生成分辨率有限，这些方法生成的结果仍然缺乏复杂纹理和复杂几何形状。为了解决这个问题，我们提出了Magic-Boost，这是一种多视角条件扩散模型，通过短暂的SDS优化（约15分钟）显著改进粗糙的生成结果。与先前基于文本或单个图像的扩散模型相比，Magic-Boost表现出强大的能力，能够从伪合成的多视角图像中生成具有高一致性的图像。它提供精确的SDS指导，与输入图像的特征相吻合，丰富了初始生成结果的几何和纹理的局部细节。大量实验证明Magic-Boost极大地增强了粗糙输入，并生成了具有丰富几何和纹理细节的高质量3D资产。（项目页面：https://magic-research.github.io/magic-boost/)

English

Benefiting from the rapid development of 2D diffusion models, 3D content creation has made significant progress recently. One promising solution involves the fine-tuning of pre-trained 2D diffusion models to harness their capacity for producing multi-view images, which are then lifted into accurate 3D models via methods like fast-NeRFs or large reconstruction models. However, as inconsistency still exists and limited generated resolution, the generation results of such methods still lack intricate textures and complex geometries. To solve this problem, we propose Magic-Boost, a multi-view conditioned diffusion model that significantly refines coarse generative results through a brief period of SDS optimization (sim15min). Compared to the previous text or single image based diffusion models, Magic-Boost exhibits a robust capability to generate images with high consistency from pseudo synthesized multi-view images. It provides precise SDS guidance that well aligns with the identity of the input images, enriching the local detail in both geometry and texture of the initial generative results. Extensive experiments show Magic-Boost greatly enhances the coarse inputs and generates high-quality 3D assets with rich geometric and textural details. (Project Page: https://magic-research.github.io/magic-boost/)