I2VEdit：透過圖像到視頻擴散模型引導的首幀視頻編輯

I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models

May 26, 2024

作者: Wenqi Ouyang, Yi Dong, Lei Yang, Jianlou Si, Xingang Pan

cs.AI

摘要

擴散模型卓越的生成能力激發了在影像和影片編輯領域的廣泛研究。相較於面臨時間維度上額外挑戰的影片編輯，影像編輯已經見證了更多元、高品質方法的發展，以及諸如Photoshop等更具能力的軟體。鑑於這種差距，我們提出了一種新穎且通用的解決方案，通過使用預先訓練的影像轉影片模型，將編輯從單幀擴展到整個影片，從而將影像編輯工具的應用範圍擴展到影片。我們的方法名為I2VEdit，根據編輯的程度，能夠適應性地保留源影片的視覺和運動完整性，有效處理全局編輯、局部編輯和中等形狀變化，這是現有方法無法完全實現的。我們方法的核心包括兩個主要過程：粗略運動提取，用於將基本運動模式與原始影片對齊，以及外觀細化，用於使用細粒度注意力匹配進行精確調整。我們還採用了跳過間隔策略，以減輕由於跨多個影片片段的自回歸生成而導致的質量降低。實驗結果證明了我們框架在細粒度影片編輯方面的卓越表現，證明了其能夠生成高品質、時間上一致的輸出。

English

The remarkable generative capabilities of diffusion models have motivated extensive research in both image and video editing. Compared to video editing which faces additional challenges in the time dimension, image editing has witnessed the development of more diverse, high-quality approaches and more capable software like Photoshop. In light of this gap, we introduce a novel and generic solution that extends the applicability of image editing tools to videos by propagating edits from a single frame to the entire video using a pre-trained image-to-video model. Our method, dubbed I2VEdit, adaptively preserves the visual and motion integrity of the source video depending on the extent of the edits, effectively handling global edits, local edits, and moderate shape changes, which existing methods cannot fully achieve. At the core of our method are two main processes: Coarse Motion Extraction to align basic motion patterns with the original video, and Appearance Refinement for precise adjustments using fine-grained attention matching. We also incorporate a skip-interval strategy to mitigate quality degradation from auto-regressive generation across multiple video clips. Experimental results demonstrate our framework's superior performance in fine-grained video editing, proving its capability to produce high-quality, temporally consistent outputs.