Be-Your-Outpainter：通過輸入特定適應性掌握視頻外描技術

Be-Your-Outpainter: Mastering Video Outpainting through Input-Specific Adaptation

March 20, 2024

作者: Fu-Yun Wang, Xiaoshi Wu, Zhaoyang Huang, Xiaoyu Shi, Dazhong Shen, Guanglu Song, Yu Liu, Hongsheng Li

cs.AI

摘要

視訊外補是一項具有挑戰性的任務，旨在在保持幀間和幀內一致性的同時，在輸入視訊的視口之外生成視訊內容。現有方法在生成質量或靈活性方面存在不足。我們介紹了MOTIA（MOTIA Mastering Video Outpainting Through Input-Specific Adaptation），這是一種基於擴散的流程，利用源視訊的內在數據特定模式和圖像/視訊生成先驗進行有效的外補。MOTIA包括兩個主要階段：特定於輸入的適應和模式感知外補。特定於輸入的適應階段涉及對單次拍攝源視訊進行高效且有效的偽外補學習。此過程鼓勵模型識別並學習源視訊中的模式，並彌合標準生成過程和外補之間的差距。隨後的模式感知外補階段致力於將這些學習到的模式泛化以生成外補結果。提出了包括空間感知插入和噪聲傳播在內的額外策略，以更好地利用擴散模型的生成先驗和從源視訊獲得的視訊模式。廣泛的評估突顯了MOTIA的優越性，在廣泛認可的基準測試中優於現有的最先進方法。值得注意的是，這些進步是在不需要大量特定任務調整的情況下實現的。

English

Video outpainting is a challenging task, aiming at generating video content outside the viewport of the input video while maintaining inter-frame and intra-frame consistency. Existing methods fall short in either generation quality or flexibility. We introduce MOTIA Mastering Video Outpainting Through Input-Specific Adaptation, a diffusion-based pipeline that leverages both the intrinsic data-specific patterns of the source video and the image/video generative prior for effective outpainting. MOTIA comprises two main phases: input-specific adaptation and pattern-aware outpainting. The input-specific adaptation phase involves conducting efficient and effective pseudo outpainting learning on the single-shot source video. This process encourages the model to identify and learn patterns within the source video, as well as bridging the gap between standard generative processes and outpainting. The subsequent phase, pattern-aware outpainting, is dedicated to the generalization of these learned patterns to generate outpainting outcomes. Additional strategies including spatial-aware insertion and noise travel are proposed to better leverage the diffusion model's generative prior and the acquired video patterns from source videos. Extensive evaluations underscore MOTIA's superiority, outperforming existing state-of-the-art methods in widely recognized benchmarks. Notably, these advancements are achieved without necessitating extensive, task-specific tuning.