DepthCrafter：为开放世界视频生成连贯的长深度序列

DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos

September 3, 2024

作者: Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, Ying Shan

cs.AI

摘要

尽管在静态图像的单目深度估计方面取得了显著进展，但在开放世界中估计视频深度仍然具有挑战性，因为开放世界视频在内容、运动、摄像机移动和长度上具有极大的多样性。我们提出了DepthCrafter，这是一种创新方法，可以为开放世界视频生成具有复杂细节的时间一致的长深度序列，而无需任何额外信息，如摄像机姿势或光流。DepthCrafter通过从预训练的图像到视频扩散模型训练视频到深度模型，通过我们精心设计的三阶段训练策略和编制的配对视频深度数据集，实现了对开放世界视频的泛化能力。我们的训练方法使模型能够一次生成长度可变的深度序列，最多达到110帧，并从真实和合成数据集中获取精确的深度细节和丰富的内容多样性。我们还提出了一种推断策略，通过分段估计和无缝拼接处理极长视频。在多个数据集上进行的全面评估显示，DepthCrafter在零样本设置下实现了开放世界视频深度估计的最先进性能。此外，DepthCrafter促进了各种下游应用，包括基于深度的视觉效果和有条件的视频生成。

English

Despite significant advancements in monocular depth estimation for static images, estimating video depth in the open world remains challenging, since open-world videos are extremely diverse in content, motion, camera movement, and length. We present DepthCrafter, an innovative method for generating temporally consistent long depth sequences with intricate details for open-world videos, without requiring any supplementary information such as camera poses or optical flow. DepthCrafter achieves generalization ability to open-world videos by training a video-to-depth model from a pre-trained image-to-video diffusion model, through our meticulously designed three-stage training strategy with the compiled paired video-depth datasets. Our training approach enables the model to generate depth sequences with variable lengths at one time, up to 110 frames, and harvest both precise depth details and rich content diversity from realistic and synthetic datasets. We also propose an inference strategy that processes extremely long videos through segment-wise estimation and seamless stitching. Comprehensive evaluations on multiple datasets reveal that DepthCrafter achieves state-of-the-art performance in open-world video depth estimation under zero-shot settings. Furthermore, DepthCrafter facilitates various downstream applications, including depth-based visual effects and conditional video generation.