多視角三維點追蹤

摘要

我們推出了首個數據驅動的多視角三維點追蹤器，旨在利用多個相機視角來追蹤動態場景中的任意點。與現有的單目追蹤器（在深度模糊和遮擋方面存在困難）或先前需要超過20個相機並進行繁瑣的每序列優化的多相機方法不同，我們的前饋模型直接使用實際數量的相機（例如四個）預測三維對應關係，從而實現了穩健且準確的在線追蹤。在已知相機姿態和基於傳感器或估計的多視角深度的情況下，我們的追蹤器將多視角特徵融合成統一的點雲，並應用k近鄰相關性以及基於變壓器的更新，即使在遮擋情況下也能可靠地估計長距離三維對應關係。我們在5K個合成的多視角Kubric序列上進行訓練，並在兩個真實世界基準測試（Panoptic Studio和DexYCB）上進行評估，分別實現了3.1厘米和2.0厘米的中位軌跡誤差。我們的方法能夠很好地泛化到1-8個視角的不同相機設置，具有不同的視點和24-150幀的視頻長度。通過發布我們的追蹤器以及訓練和評估數據集，我們旨在為多視角三維追蹤研究樹立新標準，並為實際應用提供實用工具。項目頁面請訪問https://ethz-vlg.github.io/mvtracker。

English

We introduce the first data-driven multi-view 3D point tracker, designed to track arbitrary points in dynamic scenes using multiple camera views. Unlike existing monocular trackers, which struggle with depth ambiguities and occlusion, or prior multi-camera methods that require over 20 cameras and tedious per-sequence optimization, our feed-forward model directly predicts 3D correspondences using a practical number of cameras (e.g., four), enabling robust and accurate online tracking. Given known camera poses and either sensor-based or estimated multi-view depth, our tracker fuses multi-view features into a unified point cloud and applies k-nearest-neighbors correlation alongside a transformer-based update to reliably estimate long-range 3D correspondences, even under occlusion. We train on 5K synthetic multi-view Kubric sequences and evaluate on two real-world benchmarks: Panoptic Studio and DexYCB, achieving median trajectory errors of 3.1 cm and 2.0 cm, respectively. Our method generalizes well to diverse camera setups of 1-8 views with varying vantage points and video lengths of 24-150 frames. By releasing our tracker alongside training and evaluation datasets, we aim to set a new standard for multi-view 3D tracking research and provide a practical tool for real-world applications. Project page available at https://ethz-vlg.github.io/mvtracker.

多視角三維點追蹤

Multi-View 3D Point Tracking

摘要

Support