4D扩散：多视角视频扩散模型用于4D生成

4Diffusion: Multi-view Video Diffusion Model for 4D Generation

May 31, 2024

作者: Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, Yu Qiao

cs.AI

摘要

当前的4D生成方法借助先进的扩散生成模型取得了显著的效果。然而，这些方法缺乏多视角时空建模，在整合来自多个扩散模型的不同先验知识方面遇到挑战，导致时间外观不一致和闪烁问题。在本文中，我们提出了一种新颖的4D生成流程，名为4Diffusion，旨在从单目视频中生成空间时间一致的4D内容。我们首先设计了一个针对多视角视频生成的统一扩散模型，通过将可学习的运动模块融入冻结的3D感知扩散模型中，以捕获多视角空间时间相关性。在经过精心筛选的数据集上训练后，我们的扩散模型获得了合理的时间一致性，并固有地保留了3D感知扩散模型的泛化能力和空间一致性。随后，我们提出了基于我们的多视角视频扩散模型的4D感知得分蒸馏采样损失，以优化由动态NeRF参数化的4D表示。这旨在消除由多个扩散模型引起的差异，从而实现生成空间时间一致的4D内容。此外，我们设计了一个锚定损失来增强外观细节，并促进动态NeRF的学习。大量定性和定量实验表明，我们的方法相比先前的方法实现了更优越的性能。

English

Current 4D generation methods have achieved noteworthy efficacy with the aid of advanced diffusion generative models. However, these methods lack multi-view spatial-temporal modeling and encounter challenges in integrating diverse prior knowledge from multiple diffusion models, resulting in inconsistent temporal appearance and flickers. In this paper, we propose a novel 4D generation pipeline, namely 4Diffusion aimed at generating spatial-temporally consistent 4D content from a monocular video. We first design a unified diffusion model tailored for multi-view video generation by incorporating a learnable motion module into a frozen 3D-aware diffusion model to capture multi-view spatial-temporal correlations. After training on a curated dataset, our diffusion model acquires reasonable temporal consistency and inherently preserves the generalizability and spatial consistency of the 3D-aware diffusion model. Subsequently, we propose 4D-aware Score Distillation Sampling loss, which is based on our multi-view video diffusion model, to optimize 4D representation parameterized by dynamic NeRF. This aims to eliminate discrepancies arising from multiple diffusion models, allowing for generating spatial-temporally consistent 4D content. Moreover, we devise an anchor loss to enhance the appearance details and facilitate the learning of dynamic NeRF. Extensive qualitative and quantitative experiments demonstrate that our method achieves superior performance compared to previous methods.