MonST3R：一种在运动存在的情况下估计几何形状的简单方法

摘要

从动态场景中估计几何形状，其中物体随时间移动和变形，仍然是计算机视觉中的一个核心挑战。当前的方法通常依赖于多阶段流程或全局优化，将问题分解为深度和光流等子任务，导致复杂系统容易出现错误。在本文中，我们提出了Motion DUSt3R（MonST3R），这是一种新颖的以几何为先的方法，直接从动态场景中估计每个时间步的几何形状。我们的关键洞察是，通过简单地为每个时间步估计一个点地图，我们可以有效地将DUST3R的表示适应到动态场景中，而该表示先前仅用于静态场景。然而，这种方法面临着一个重大挑战：适用的训练数据稀缺，即带深度标签的动态姿势视频。尽管如此，我们展示了通过将问题定位为微调任务，识别几个合适的数据集，并在这些有限数据上策略性地训练模型，我们可以令模型出人意料地处理动态，即使没有显式的运动表示。基于此，我们为几个下游视频特定任务引入了新的优化，并在视频深度和相机姿态估计方面展示了强大的性能，优于先前的工作在鲁棒性和效率方面。此外，MonST3R在主要的前馈4D重建方面显示出有希望的结果。

English

Estimating geometry from dynamic scenes, where objects move and deform over time, remains a core challenge in computer vision. Current approaches often rely on multi-stage pipelines or global optimizations that decompose the problem into subtasks, like depth and flow, leading to complex systems prone to errors. In this paper, we present Motion DUSt3R (MonST3R), a novel geometry-first approach that directly estimates per-timestep geometry from dynamic scenes. Our key insight is that by simply estimating a pointmap for each timestep, we can effectively adapt DUST3R's representation, previously only used for static scenes, to dynamic scenes. However, this approach presents a significant challenge: the scarcity of suitable training data, namely dynamic, posed videos with depth labels. Despite this, we show that by posing the problem as a fine-tuning task, identifying several suitable datasets, and strategically training the model on this limited data, we can surprisingly enable the model to handle dynamics, even without an explicit motion representation. Based on this, we introduce new optimizations for several downstream video-specific tasks and demonstrate strong performance on video depth and camera pose estimation, outperforming prior work in terms of robustness and efficiency. Moreover, MonST3R shows promising results for primarily feed-forward 4D reconstruction.

MonST3R：一种在运动存在的情况下估计几何形状的简单方法

MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion

摘要

Summary

Support

Support