生成式视点拼接

摘要

自回归视频扩散模型能够生成稳定且与历史帧保持连贯的长序列，但其无法通过未来帧的引导信息来调控当前生成过程。在基于预设相机轨迹的摄像引导视频生成任务中，这一局限性会导致生成场景与轨迹发生碰撞，进而引发自回归过程的快速崩溃。为解决该问题，我们提出生成式视角缝合技术（GVS），通过并行采样整个序列确保生成场景精准契合预设相机轨迹的每个区段。我们的核心贡献是一种采样算法，该算法将机器人规划领域的扩散缝合技术拓展至视频生成领域。此类缝合方法通常需依赖专门训练的模型，而GVS兼容任何采用"扩散强制"框架训练的现成视频模型——我们证明这一主流序列扩散框架已具备缝合所需的支持能力。此外，我们提出全向引导技术，通过联合过去与未来帧的条件约束增强缝合时域一致性，并藉此实现闭环机制以保障长程连贯性。总体而言，GVS实现的摄像引导视频生成具有稳定性、无碰撞性、帧间连贯性及闭环特性，可适配多种预设相机路径（包括奥斯卡·路特斯瓦德的彭罗斯阶梯）。建议通过https://andrewsonga.github.io/gvs观看视频结果以获得最佳体验。

English

Autoregressive video diffusion models are capable of long rollouts that are stable and consistent with history, but they are unable to guide the current generation with conditioning from the future. In camera-guided video generation with a predefined camera trajectory, this limitation leads to collisions with the generated scene, after which autoregression quickly collapses. To address this, we propose Generative View Stitching (GVS), which samples the entire sequence in parallel such that the generated scene is faithful to every part of the predefined camera trajectory. Our main contribution is a sampling algorithm that extends prior work on diffusion stitching for robot planning to video generation. While such stitching methods usually require a specially trained model, GVS is compatible with any off-the-shelf video model trained with Diffusion Forcing, a prevalent sequence diffusion framework that we show already provides the affordances necessary for stitching. We then introduce Omni Guidance, a technique that enhances the temporal consistency in stitching by conditioning on both the past and future, and that enables our proposed loop-closing mechanism for delivering long-range coherence. Overall, GVS achieves camera-guided video generation that is stable, collision-free, frame-to-frame consistent, and closes loops for a variety of predefined camera paths, including Oscar Reutersv\"ard's Impossible Staircase. Results are best viewed as videos at https://andrewsonga.github.io/gvs.