遵循您的画布：具有广泛内容生成的高分辨率视频修复

摘要

本文探讨了利用广泛内容生成进行更高分辨率视频外推的方法。我们指出了现有方法在尝试大规模外推视频时面临的常见问题：生成低质量内容和GPU内存所施加的限制。为了解决这些挑战，我们提出了一种基于扩散的方法，称为Follow-Your-Canvas。它建立在两个核心设计之上。首先，我们不采用“单次拍摄”外推的常见做法，而是将任务分布在空间窗口中，并无缝地合并它们。这使我们能够在不受GPU内存限制的情况下外推任何大小和分辨率的视频。其次，源视频及其相对位置关系被注入到每个窗口的生成过程中。这使得每个窗口内生成的空间布局与源视频协调一致。结合这两个设计，我们能够生成具有丰富内容的高分辨率外推视频，同时保持空间和时间的一致性。Follow-Your-Canvas在大规模视频外推方面表现出色，例如，从512X512到1152X2048（9倍），同时产生高质量且美观的结果。它在各种分辨率和比例设置下均取得了最佳的定量结果。代码已发布在https://github.com/mayuelala/FollowYourCanvas。

English

This paper explores higher-resolution video outpainting with extensive content generation. We point out common issues faced by existing methods when attempting to largely outpaint videos: the generation of low-quality content and limitations imposed by GPU memory. To address these challenges, we propose a diffusion-based method called Follow-Your-Canvas. It builds upon two core designs. First, instead of employing the common practice of "single-shot" outpainting, we distribute the task across spatial windows and seamlessly merge them. It allows us to outpaint videos of any size and resolution without being constrained by GPU memory. Second, the source video and its relative positional relation are injected into the generation process of each window. It makes the generated spatial layout within each window harmonize with the source video. Coupling with these two designs enables us to generate higher-resolution outpainting videos with rich content while keeping spatial and temporal consistency. Follow-Your-Canvas excels in large-scale video outpainting, e.g., from 512X512 to 1152X2048 (9X), while producing high-quality and aesthetically pleasing results. It achieves the best quantitative results across various resolution and scale setups. The code is released on https://github.com/mayuelala/FollowYourCanvas

遵循您的画布：具有广泛内容生成的高分辨率视频修复

Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generation

摘要

Summary

Support

Support