光谱点云：基于光谱矩监督的鲁棒可微分跟踪方法

SpectralSplats: Robust Differentiable Tracking via Spectral Moment Supervision

March 25, 2026

作者: Avigail Cohen Rimon, Amir Mann, Mirela Ben Chen, Or Litany

cs.AI

摘要

三维高斯泼溅（3DGS）能够实现实时的照片级真实感新视角合成，使其成为基于模型的视频追踪中极具吸引力的表征方法。然而，在实际场景中利用3DGS渲染器的可微性仍存在众所周知的脆弱性问题。其根本瓶颈在于高斯图元的紧凑局部支撑特性：标准光度目标函数隐式依赖于空间重叠度，若严重的相机错位导致渲染对象超出目标局部覆盖范围，梯度将完全消失，使优化器陷入停滞。我们提出SpectralSplats这一鲁棒追踪框架，通过将优化目标从空间域转换到频域，成功解决了“梯度消失”问题。通过一组全局复正弦特征（频谱矩）对渲染图像进行监督，我们构建了全局吸引域，确保即使像素重叠完全不存在时，整个图像域内仍存在指向目标的有效方向梯度。为在利用该全局吸引域的同时避免高频带来的周期性局部极小值，我们从第一性原理出发推导出理论完备的频率退火策略，使优化器能够从全局凸性平滑过渡到精确空间对齐。实验表明，SpectralSplats可作为空间损失函数的无缝替代方案，适配多种形变参数化方法（从MLP到稀疏控制点），即使在严重错位的初始化情况下——传统基于表观的追踪方法会彻底失败——也能成功恢复复杂形变。

English

3D Gaussian Splatting (3DGS) enables real-time, photorealistic novel view synthesis, making it a highly attractive representation for model-based video tracking. However, leveraging the differentiability of the 3DGS renderer "in the wild" remains notoriously fragile. A fundamental bottleneck lies in the compact, local support of the Gaussian primitives. Standard photometric objectives implicitly rely on spatial overlap; if severe camera misalignment places the rendered object outside the target's local footprint, gradients strictly vanish, leaving the optimizer stranded. We introduce SpectralSplats, a robust tracking framework that resolves this "vanishing gradient" problem by shifting the optimization objective from the spatial to the frequency domain. By supervising the rendered image via a set of global complex sinusoidal features (Spectral Moments), we construct a global basin of attraction, ensuring that a valid, directional gradient toward the target exists across the entire image domain, even when pixel overlap is completely nonexistent. To harness this global basin without introducing periodic local minima associated with high frequencies, we derive a principled Frequency Annealing schedule from first principles, gracefully transitioning the optimizer from global convexity to precise spatial alignment. We demonstrate that SpectralSplats acts as a seamless, drop-in replacement for spatial losses across diverse deformation parameterizations (from MLPs to sparse control points), successfully recovering complex deformations even from severely misaligned initializations where standard appearance-based tracking catastrophically fails.