VidPanos：从随意摄制的摄像视频生成全景视频

VidPanos: Generative Panoramic Videos from Casual Panning Videos

October 17, 2024

作者: Jingwei Ma, Erika Lu, Roni Paiss, Shiran Zada, Aleksander Holynski, Tali Dekel, Brian Curless, Michael Rubinstein, Forrester Cole

cs.AI

摘要

全景图像拼接提供了一个统一的、广角的场景视图，超出了摄像机的视野范围。将全景视频的帧拼接成全景照片对于静止场景是一个众所周知的问题，但是当物体在移动时，静态全景图无法捕捉到整个场景。我们提出了一种方法，可以从随意拍摄的全景视频中合成全景视频，就好像原始视频是用广角摄像头拍摄的一样。我们将全景合成视为一个时空外描问题，旨在创建一个与输入视频长度相同的完整全景视频。一致完成时空体积需要对视频内容和运动进行强大、真实的先验建模，为此我们采用生成式视频模型进行调整。然而，现有的生成模型并不能立即扩展到全景完成，正如我们所展示的。相反，我们将视频生成应用作为全景合成系统的一个组成部分，并展示如何利用模型的优势同时最小化它们的局限性。我们的系统可以为各种野外场景创建视频全景，包括人物、车辆、流动的水以及静止的背景特征。

English

Panoramic image stitching provides a unified, wide-angle view of a scene that extends beyond the camera's field of view. Stitching frames of a panning video into a panoramic photograph is a well-understood problem for stationary scenes, but when objects are moving, a still panorama cannot capture the scene. We present a method for synthesizing a panoramic video from a casually-captured panning video, as if the original video were captured with a wide-angle camera. We pose panorama synthesis as a space-time outpainting problem, where we aim to create a full panoramic video of the same length as the input video. Consistent completion of the space-time volume requires a powerful, realistic prior over video content and motion, for which we adapt generative video models. Existing generative models do not, however, immediately extend to panorama completion, as we show. We instead apply video generation as a component of our panorama synthesis system, and demonstrate how to exploit the strengths of the models while minimizing their limitations. Our system can create video panoramas for a range of in-the-wild scenes including people, vehicles, and flowing water, as well as stationary background features.