SV3D：利用潜在视频扩散从单个图像生成新颖的多视角合成和3D生成。

SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion

March 18, 2024

作者: Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, Varun Jampani

cs.AI

摘要

我们提出了稳定视频3D（SV3D）- 一种潜在视频扩散模型，用于围绕3D对象生成高分辨率的图像到多视角视频。最近关于3D生成的研究提出了技术，用于调整2D生成模型以进行新视角合成（NVS）和3D优化。然而，这些方法存在一些缺点，要么是由于视角有限，要么是由于NVS不一致，从而影响了3D对象生成的性能。在这项工作中，我们提出了SV3D，它调整了图像到视频扩散模型，用于新的多视角合成和3D生成，从而利用了视频模型的泛化和多视角一致性，同时进一步增加了用于NVS的显式相机控制。我们还提出了改进的3D优化技术，以利用SV3D及其NVS输出进行图像到3D生成。在多个数据集上进行的广泛实验结果，包括2D和3D指标以及用户研究，证明了SV3D在NVS和3D重建方面相对于先前工作具有最先进的性能。

English

We present Stable Video 3D (SV3D) -- a latent video diffusion model for high-resolution, image-to-multi-view generation of orbital videos around a 3D object. Recent work on 3D generation propose techniques to adapt 2D generative models for novel view synthesis (NVS) and 3D optimization. However, these methods have several disadvantages due to either limited views or inconsistent NVS, thereby affecting the performance of 3D object generation. In this work, we propose SV3D that adapts image-to-video diffusion model for novel multi-view synthesis and 3D generation, thereby leveraging the generalization and multi-view consistency of the video models, while further adding explicit camera control for NVS. We also propose improved 3D optimization techniques to use SV3D and its NVS outputs for image-to-3D generation. Extensive experimental results on multiple datasets with 2D and 3D metrics as well as user study demonstrate SV3D's state-of-the-art performance on NVS as well as 3D reconstruction compared to prior works.