光譜運動：具有鏡面場景的動態三維重建

SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes

October 22, 2024

作者: Cheng-De Fan, Chen-Wei Chang, Yi-Ruei Liu, Jie-Ying Lee, Jiun-Long Huang, Yu-Chee Tseng, Yu-Lun Liu

cs.AI

摘要

我們提出了SpectroMotion，一種結合了三維高斯點陣（3DGS）、基於物理的渲染（PBR）和變形場的新方法，用於重建動態鏡面場景。先前將3DGS擴展到建模動態場景的方法一直難以準確表示鏡面表面。我們的方法通過引入一種殘差校正技術來在變形過程中準確計算表面法線，並輔以一個可適應時間變化照明條件的可變環境貼圖，來解決這一限制。我們實現了一種由粗到細的訓練策略，顯著增強了場景幾何和鏡面顏色預測。我們展示了我們的模型在視角合成包含動態鏡面物體的場景方面優於先前方法，並且是唯一能夠合成逼真真實世界動態鏡面場景的現有3DGS方法，勝過了在渲染複雜、動態和鏡面場景方面的最先進方法。

English

We present SpectroMotion, a novel approach that combines 3D Gaussian Splatting (3DGS) with physically-based rendering (PBR) and deformation fields to reconstruct dynamic specular scenes. Previous methods extending 3DGS to model dynamic scenes have struggled to accurately represent specular surfaces. Our method addresses this limitation by introducing a residual correction technique for accurate surface normal computation during deformation, complemented by a deformable environment map that adapts to time-varying lighting conditions. We implement a coarse-to-fine training strategy that significantly enhances both scene geometry and specular color prediction. We demonstrate that our model outperforms prior methods for view synthesis of scenes containing dynamic specular objects and that it is the only existing 3DGS method capable of synthesizing photorealistic real-world dynamic specular scenes, outperforming state-of-the-art methods in rendering complex, dynamic, and specular scenes.