時空高斯特徵點塗抹以進行即時動態視圖合成

摘要

動態場景的新視角合成一直是一個引人入勝但具挑戰性的問題。儘管近期取得了一些進展，但同時實現高分辨率的逼真結果、實時渲染和緊湊存儲仍然是一項艱巨的任務。為應對這些挑戰，我們提出了時空高斯特徵塗抹作為一種新穎的動態場景表示，由三個關鍵組成部分組成。首先，我們通過增強3D高斯模型的時間不透明度和參數化運動/旋轉來制定具有表現力的時空高斯模型。這使得時空高斯模型能夠捕捉場景中的靜態、動態以及瞬態內容。其次，我們引入了特徵塗抹渲染，用神經特徵取代球面調和。這些特徵有助於建模視角和時間依賴性外觀，同時保持較小的尺寸。第三，我們利用訓練誤差和粗深度的指導，在現有管線難以收斂的區域採樣新的高斯模型。對幾個已建立的真實世界數據集進行的實驗表明，我們的方法實現了最先進的渲染質量和速度，同時保持緊湊的存儲。在8K分辨率下，我們的精簡版模型可以在Nvidia RTX 4090 GPU上以60 FPS進行渲染。

English

Novel view synthesis of dynamic scenes has been an intriguing yet challenging problem. Despite recent advancements, simultaneously achieving high-resolution photorealistic results, real-time rendering, and compact storage remains a formidable task. To address these challenges, we propose Spacetime Gaussian Feature Splatting as a novel dynamic scene representation, composed of three pivotal components. First, we formulate expressive Spacetime Gaussians by enhancing 3D Gaussians with temporal opacity and parametric motion/rotation. This enables Spacetime Gaussians to capture static, dynamic, as well as transient content within a scene. Second, we introduce splatted feature rendering, which replaces spherical harmonics with neural features. These features facilitate the modeling of view- and time-dependent appearance while maintaining small size. Third, we leverage the guidance of training error and coarse depth to sample new Gaussians in areas that are challenging to converge with existing pipelines. Experiments on several established real-world datasets demonstrate that our method achieves state-of-the-art rendering quality and speed, while retaining compact storage. At 8K resolution, our lite-version model can render at 60 FPS on an Nvidia RTX 4090 GPU.

時空高斯特徵點塗抹以進行即時動態視圖合成

Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis

摘要

Support