用于实时动态视图合成的时空高斯特征喷洒

摘要

动态场景的新视角合成一直是一个引人注目但具有挑战性的问题。尽管最近取得了进展，但同时实现高分辨率逼真结果、实时渲染和紧凑存储仍然是一项艰巨任务。为了解决这些挑战，我们提出了时空高斯特征喷洒作为一种新颖的动态场景表示，由三个关键组件组成。首先，我们通过增强3D高斯函数的时间不透明度和参数化运动/旋转来形成富有表现力的时空高斯函数。这使得时空高斯函数能够捕捉场景中的静态、动态以及瞬时内容。其次，我们引入了特征喷洒渲染，用神经特征取代球谐函数。这些特征有助于建模视角和时间相关的外观，同时保持较小的尺寸。第三，我们利用训练误差和粗深度的指导，在现有管线难以收敛的区域对新的高斯函数进行采样。在几个已建立的真实世界数据集上的实验表明，我们的方法实现了最先进的渲染质量和速度，同时保持了紧凑的存储。在8K分辨率下，我们的精简版本模型可以在Nvidia RTX 4090 GPU上以60 FPS进行渲染。

English

Novel view synthesis of dynamic scenes has been an intriguing yet challenging problem. Despite recent advancements, simultaneously achieving high-resolution photorealistic results, real-time rendering, and compact storage remains a formidable task. To address these challenges, we propose Spacetime Gaussian Feature Splatting as a novel dynamic scene representation, composed of three pivotal components. First, we formulate expressive Spacetime Gaussians by enhancing 3D Gaussians with temporal opacity and parametric motion/rotation. This enables Spacetime Gaussians to capture static, dynamic, as well as transient content within a scene. Second, we introduce splatted feature rendering, which replaces spherical harmonics with neural features. These features facilitate the modeling of view- and time-dependent appearance while maintaining small size. Third, we leverage the guidance of training error and coarse depth to sample new Gaussians in areas that are challenging to converge with existing pipelines. Experiments on several established real-world datasets demonstrate that our method achieves state-of-the-art rendering quality and speed, while retaining compact storage. At 8K resolution, our lite-version model can render at 60 FPS on an Nvidia RTX 4090 GPU.

用于实时动态视图合成的时空高斯特征喷洒

Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis

摘要

Support