光谱运动：具有镜面场景的动态三维重建

SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes

October 22, 2024

作者: Cheng-De Fan, Chen-Wei Chang, Yi-Ruei Liu, Jie-Ying Lee, Jiun-Long Huang, Yu-Chee Tseng, Yu-Lun Liu

cs.AI

摘要

我们提出了SpectroMotion，这是一种新颖的方法，将三维高斯飘粒（3DGS）与基于物理的渲染（PBR）和变形场相结合，用于重建动态镜面场景。先前将3DGS扩展到建模动态场景的方法在准确表示镜面表面方面存在困难。我们的方法通过引入残差校正技术来解决这一限制，以在变形过程中准确计算表面法线，同时配合一个可变形的环境贴图，以适应时变光照条件。我们实现了一种由粗到细的训练策略，显著增强了场景几何和镜面颜色预测。我们展示了我们的模型在包含动态镜面物体的场景视图合成方面优于先前方法，并且它是唯一能够合成逼真的真实世界动态镜面场景的现有3DGS方法，在渲染复杂、动态和镜面场景方面优于最先进的方法。

English

We present SpectroMotion, a novel approach that combines 3D Gaussian Splatting (3DGS) with physically-based rendering (PBR) and deformation fields to reconstruct dynamic specular scenes. Previous methods extending 3DGS to model dynamic scenes have struggled to accurately represent specular surfaces. Our method addresses this limitation by introducing a residual correction technique for accurate surface normal computation during deformation, complemented by a deformable environment map that adapts to time-varying lighting conditions. We implement a coarse-to-fine training strategy that significantly enhances both scene geometry and specular color prediction. We demonstrate that our model outperforms prior methods for view synthesis of scenes containing dynamic specular objects and that it is the only existing 3DGS method capable of synthesizing photorealistic real-world dynamic specular scenes, outperforming state-of-the-art methods in rendering complex, dynamic, and specular scenes.