光譜點雲：基於光譜矩監督的魯棒可微分追蹤方法

SpectralSplats: Robust Differentiable Tracking via Spectral Moment Supervision

March 25, 2026

作者: Avigail Cohen Rimon, Amir Mann, Mirela Ben Chen, Or Litany

cs.AI

摘要

三維高斯潑濺（3DGS）能夠實現即時、逼真的新視角合成，使其成為基於模型的視頻追蹤中極具吸引力的表示方法。然而，在實際應用中利用3DGS渲染器的可微分特性仍存在顯著的脆弱性問題。根本瓶頸在於高斯圖元的緊湊局部支撐特性：標準光度目標函數隱式依賴於空間重疊，若嚴重的相機失準使渲染物件偏離目標局部覆蓋範圍，梯度將完全消失，導致優化器陷入停滯。我們提出SpectralSplats這一魯棒追蹤框架，通過將優化目標從空間域轉移至頻域，解決了「梯度消失」問題。藉由一組全局複數正弦特徵（頻譜矩）對渲染圖像進行監督，我們構建了全局吸引域，確保即使像素重疊完全不存在時，整個圖像域內仍存在指向目標的有效方向梯度。為利用此全局吸引域同時避免高頻率引起的週期性局部極小值，我們從第一性原理推導出理論嚴謹的頻率退火策略，使優化器能從全局凸性平滑過渡至精確空間對齊。實驗表明，SpectralSplats可作為空間損失函數的無縫替代方案，適用於多種變形參數化方法（從MLP到稀疏控制點），即使在嚴重失準的初始狀態下（傳統基於外觀的追蹤會完全失效時），仍能成功恢復複雜變形。

English

3D Gaussian Splatting (3DGS) enables real-time, photorealistic novel view synthesis, making it a highly attractive representation for model-based video tracking. However, leveraging the differentiability of the 3DGS renderer "in the wild" remains notoriously fragile. A fundamental bottleneck lies in the compact, local support of the Gaussian primitives. Standard photometric objectives implicitly rely on spatial overlap; if severe camera misalignment places the rendered object outside the target's local footprint, gradients strictly vanish, leaving the optimizer stranded. We introduce SpectralSplats, a robust tracking framework that resolves this "vanishing gradient" problem by shifting the optimization objective from the spatial to the frequency domain. By supervising the rendered image via a set of global complex sinusoidal features (Spectral Moments), we construct a global basin of attraction, ensuring that a valid, directional gradient toward the target exists across the entire image domain, even when pixel overlap is completely nonexistent. To harness this global basin without introducing periodic local minima associated with high frequencies, we derive a principled Frequency Annealing schedule from first principles, gracefully transitioning the optimizer from global convexity to precise spatial alignment. We demonstrate that SpectralSplats acts as a seamless, drop-in replacement for spatial losses across diverse deformation parameterizations (from MLPs to sparse control points), successfully recovering complex deformations even from severely misaligned initializations where standard appearance-based tracking catastrophically fails.