MVTrack4Gen：多視角點追蹤作為4D影片生成的幾何監督

摘要

從單目參考影片沿著目標相機軌跡合成新視角影片，需要同時兼顧參考影片的幾何一致性與動態真實性。現有基於顯式三維表徵的方法，受限於現成重建模組的精確度，往往難以準確處理單目影片中動態物體的幾何結構。相較之下，僅依賴相機條件的方法雖能達成高視覺品質，卻常難以維持幾何與動態的一致性。本研究提出MVTrack4Gen（基於多視角點追蹤的新視角生成），這是一個動態感知訓練框架，透過引入多視角點追蹤作為額外的幾何與動態監督訊號，強化了僅依賴相機條件的擴散模型。我們的核心發現是：特定注意力層編碼了強烈的對應關係——查詢特徵會關注不同視角及時序上幾何對應位置的鍵特徵，而這些對應關係的錯位會導致動態不一致。基於此現象，我們將這些特徵導入輔助的多視角追蹤頭，並以點追蹤目標聯合訓練擴散模型。透過明確強化這些動態感知對應關係，MVTrack4Gen能提升現有模型對參考視角動態的追蹤能力，同時維持跨視角的幾何一致性。在多元基準測試中，本方法達成了最先進的幾何一致性與具競爭力的相機軌跡精確度。

English

Synthesizing a novel-view video from a monocular reference video along a target camera trajectory requires both geometric consistency and motion fidelity with respect to the reference video. Existing methods based on explicit 3D representations are limited by the accuracy of off-the-shelf reconstruction modules, which often produce inaccurate geometry for dynamic objects in monocular videos. In contrast, camera-conditioning-only methods can achieve high visual quality but often struggle to preserve geometric and motion consistency. In this work, we introduce MVTrack4Gen (Multi-View point Tracking for Novel-View Generation), a motion-aware training framework that leverages multi-view point tracking as an additional geometric and motion supervision signal for camera-conditioning-only novel-view video diffusion models. Our key finding is that specific attention layers encode strong correspondence cues, where query features attend to key features at geometrically corresponding locations across views and over time, and the misalignment of these correspondences causes motion inconsistency. Based on this observation, we route these features into an auxiliary multi-view tracking head and jointly train the diffusion model with a point-tracking objective. By explicitly strengthening these motion-aware correspondences, MVTrack4Gen improves existing models to better follow the motion in the reference view and maintain cross-view geometric consistency. Across diverse benchmarks, our method achieves state-of-the-art geometric consistency and competitive camera accuracy.