多视角三维点云跟踪

摘要

我们推出了首个数据驱动的多视角3D点追踪器，旨在利用多摄像头视角追踪动态场景中的任意点。与现有单目追踪器在深度模糊和遮挡问题上表现不佳，或先前需要超过20个摄像头及繁琐逐序列优化的多摄像头方法不同，我们的前馈模型直接利用实用数量的摄像头（如四个）预测3D对应关系，实现了稳健且精准的在线追踪。在已知相机姿态及基于传感器或估计的多视角深度信息下，我们的追踪器将多视角特征融合为统一点云，并应用k近邻相关性结合基于Transformer的更新机制，即使在遮挡情况下也能可靠估计长距离3D对应关系。我们在5千个合成的多视角Kubric序列上进行训练，并在两个真实世界基准测试——Panoptic Studio和DexYCB上评估，分别取得了3.1厘米和2.0厘米的中位轨迹误差。我们的方法在1至8个视角、不同观察点及24至150帧视频长度的多样化摄像头配置中表现出良好的泛化能力。通过发布我们的追踪器及训练与评估数据集，我们旨在为多视角3D追踪研究设立新标准，并为实际应用提供实用工具。项目页面请访问https://ethz-vlg.github.io/mvtracker。

English

We introduce the first data-driven multi-view 3D point tracker, designed to track arbitrary points in dynamic scenes using multiple camera views. Unlike existing monocular trackers, which struggle with depth ambiguities and occlusion, or prior multi-camera methods that require over 20 cameras and tedious per-sequence optimization, our feed-forward model directly predicts 3D correspondences using a practical number of cameras (e.g., four), enabling robust and accurate online tracking. Given known camera poses and either sensor-based or estimated multi-view depth, our tracker fuses multi-view features into a unified point cloud and applies k-nearest-neighbors correlation alongside a transformer-based update to reliably estimate long-range 3D correspondences, even under occlusion. We train on 5K synthetic multi-view Kubric sequences and evaluate on two real-world benchmarks: Panoptic Studio and DexYCB, achieving median trajectory errors of 3.1 cm and 2.0 cm, respectively. Our method generalizes well to diverse camera setups of 1-8 views with varying vantage points and video lengths of 24-150 frames. By releasing our tracker alongside training and evaluation datasets, we aim to set a new standard for multi-view 3D tracking research and provide a practical tool for real-world applications. Project page available at https://ethz-vlg.github.io/mvtracker.

多视角三维点云跟踪

Multi-View 3D Point Tracking

摘要

Support