跟隨畫布：具有廣泛內容生成的高解析度視頻修補

摘要

本文探討具有廣泛內容生成的高解析度視頻外描。我們指出現有方法在嘗試大範圍外描視頻時所面臨的常見問題：生成低質量內容和 GPU 記憶體所施加的限制。為應對這些挑戰，我們提出了一種基於擴散的方法，稱為Follow-Your-Canvas。它建立在兩個核心設計之上。首先，我們不採用“單次拍攝”外描的常見做法，而是將任務分佈在空間窗口上並無縫地合併它們。這使我們能夠在不受 GPU 記憶體限制的情況下外描任何大小和解析度的視頻。其次，源視頻及其相對位置關係被注入到每個窗口的生成過程中。這使得每個窗口內生成的空間佈局與源視頻協調一致。結合這兩個設計，我們能夠生成具有豐富內容的高解析度外描視頻，同時保持空間和時間一致性。Follow-Your-Canvas在大規模視頻外描方面表現出色，例如，從512X512到1152X2048（9倍），同時產生高質量和美觀的結果。它在各種解析度和比例設置下均取得最佳的定量結果。代碼已在https://github.com/mayuelala/FollowYourCanvas 上發布。

English

This paper explores higher-resolution video outpainting with extensive content generation. We point out common issues faced by existing methods when attempting to largely outpaint videos: the generation of low-quality content and limitations imposed by GPU memory. To address these challenges, we propose a diffusion-based method called Follow-Your-Canvas. It builds upon two core designs. First, instead of employing the common practice of "single-shot" outpainting, we distribute the task across spatial windows and seamlessly merge them. It allows us to outpaint videos of any size and resolution without being constrained by GPU memory. Second, the source video and its relative positional relation are injected into the generation process of each window. It makes the generated spatial layout within each window harmonize with the source video. Coupling with these two designs enables us to generate higher-resolution outpainting videos with rich content while keeping spatial and temporal consistency. Follow-Your-Canvas excels in large-scale video outpainting, e.g., from 512X512 to 1152X2048 (9X), while producing high-quality and aesthetically pleasing results. It achieves the best quantitative results across various resolution and scale setups. The code is released on https://github.com/mayuelala/FollowYourCanvas

跟隨畫布：具有廣泛內容生成的高解析度視頻修補

Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generation

摘要

Summary

Support

Support