高斯流：用于4D内容创建的高斯动力学喷溅

GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation

March 19, 2024

作者: Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, Ulrich Neumann

cs.AI

摘要

从图像或视频创建高斯飞溅的4D场是一项具有挑战性的任务，因为它的约束不足。虽然优化可以从输入视频中提取光度参考或受到生成模型的调节，但直接监督高斯运动仍未得到充分探讨。在本文中，我们引入了一个新概念，高斯流，它连接了3D高斯和相邻帧之间的像素速度之间的动态。高斯流可以通过将高斯动态喷洒到图像空间中来高效获得。这种可微分的过程使得可以从光流中直接进行动态监督。我们的方法显著地有利于使用高斯飞溅进行4D动态内容生成和4D新视角合成，特别是对于那些难以通过现有方法处理的具有丰富运动的内容。在4D生成中常见的颜色漂移问题也通过改进的高斯动态得到解决。在广泛的实验中表现出卓越的视觉质量证明了我们方法的有效性。定量和定性评估表明我们的方法在4D生成和4D新视角合成两项任务上均取得了最先进的结果。项目页面：https://zerg-overmind.github.io/GaussianFlow.github.io/

English

Creating 4D fields of Gaussian Splatting from images or videos is a challenging task due to its under-constrained nature. While the optimization can draw photometric reference from the input videos or be regulated by generative models, directly supervising Gaussian motions remains underexplored. In this paper, we introduce a novel concept, Gaussian flow, which connects the dynamics of 3D Gaussians and pixel velocities between consecutive frames. The Gaussian flow can be efficiently obtained by splatting Gaussian dynamics into the image space. This differentiable process enables direct dynamic supervision from optical flow. Our method significantly benefits 4D dynamic content generation and 4D novel view synthesis with Gaussian Splatting, especially for contents with rich motions that are hard to be handled by existing methods. The common color drifting issue that happens in 4D generation is also resolved with improved Guassian dynamics. Superior visual quality on extensive experiments demonstrates our method's effectiveness. Quantitative and qualitative evaluations show that our method achieves state-of-the-art results on both tasks of 4D generation and 4D novel view synthesis. Project page: https://zerg-overmind.github.io/GaussianFlow.github.io/