GaussianFlow：將高斯動態塗抹應用於4D內容創建

GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation

March 19, 2024

作者: Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, Ulrich Neumann

cs.AI

摘要

從影像或視頻中創建高斯擴散的4D場是一項具有挑戰性的任務，因為它存在著不完全約束的特性。儘管優化可以從輸入視頻中獲取光度參考或受到生成模型的調節，但直接監督高斯運動仍然是一個未被充分探索的領域。在本文中，我們引入了一個新概念，高斯流，它將3D高斯和相鄰幀之間的像素速度的動態相連。高斯流可以通過將高斯動態傳播到圖像空間中來有效地獲得。這種可微分的過程使得可以從光流中直接進行動態監督。我們的方法顯著地有利於使用高斯擴散進行4D動態內容生成和4D新視角合成，特別適用於那些現有方法難以處理的具有豐富運動的內容。改進的高斯動態還解決了在4D生成中常見的顏色漂移問題。在廣泛實驗中展示出的卓越視覺質量證明了我們方法的有效性。定量和定性評估表明，我們的方法在4D生成和4D新視角合成的兩項任務上均取得了最先進的結果。專案頁面：https://zerg-overmind.github.io/GaussianFlow.github.io/

English

Creating 4D fields of Gaussian Splatting from images or videos is a challenging task due to its under-constrained nature. While the optimization can draw photometric reference from the input videos or be regulated by generative models, directly supervising Gaussian motions remains underexplored. In this paper, we introduce a novel concept, Gaussian flow, which connects the dynamics of 3D Gaussians and pixel velocities between consecutive frames. The Gaussian flow can be efficiently obtained by splatting Gaussian dynamics into the image space. This differentiable process enables direct dynamic supervision from optical flow. Our method significantly benefits 4D dynamic content generation and 4D novel view synthesis with Gaussian Splatting, especially for contents with rich motions that are hard to be handled by existing methods. The common color drifting issue that happens in 4D generation is also resolved with improved Guassian dynamics. Superior visual quality on extensive experiments demonstrates our method's effectiveness. Quantitative and qualitative evaluations show that our method achieves state-of-the-art results on both tasks of 4D generation and 4D novel view synthesis. Project page: https://zerg-overmind.github.io/GaussianFlow.github.io/