通過：一個用於全球和本地視頻編輯的時空視頻適應框架

VIA: A Spatiotemporal Video Adaptation Framework for Global and Local Video Editing

June 18, 2024

作者: Jing Gu, Yuwei Fang, Ivan Skorokhodov, Peter Wonka, Xinya Du, Sergey Tulyakov, Xin Eric Wang

cs.AI

摘要

影片編輯是數位媒體的基石，從娛樂、教育到專業溝通無所不包。然而，先前的方法常常忽略全球和本地情境的全面理解的必要性，導致時空維度中不準確和不一致的編輯，尤其是對於長影片。本文介紹了VIA，一個統一的時空影片適應框架，用於全球和本地影片編輯，推動著對於一分鐘長影片的一致性編輯極限。首先，為了確保單個幀內的本地一致性，VIA的基礎是一種新穎的測試時間編輯適應方法，該方法適應了預先訓練的圖像編輯模型，以改善潛在編輯方向和文本指示之間的一致性，並適應了遮罩潛在變量以實現精確的本地控制。此外，為了在整個影片序列中保持全局一致性，我們引入了時空適應，該方法適應了關鍵幀中的一致性注意變量，並在整個序列中策略性地應用它們以實現編輯效果。大量實驗表明，與基準方法相比，我們的VIA方法產生的編輯更忠實於原始影片，更具時空上的連貫性，並在本地控制方面更為精確。更重要的是，我們展示了VIA可以在幾分鐘內實現一致的長影片編輯，從而釋放了對於長影片序列進行高級影片編輯任務的潛力。

English

Video editing stands as a cornerstone of digital media, from entertainment and education to professional communication. However, previous methods often overlook the necessity of comprehensively understanding both global and local contexts, leading to inaccurate and inconsistency edits in the spatiotemporal dimension, especially for long videos. In this paper, we introduce VIA, a unified spatiotemporal VIdeo Adaptation framework for global and local video editing, pushing the limits of consistently editing minute-long videos. First, to ensure local consistency within individual frames, the foundation of VIA is a novel test-time editing adaptation method, which adapts a pre-trained image editing model for improving consistency between potential editing directions and the text instruction, and adapts masked latent variables for precise local control. Furthermore, to maintain global consistency over the video sequence, we introduce spatiotemporal adaptation that adapts consistent attention variables in key frames and strategically applies them across the whole sequence to realize the editing effects. Extensive experiments demonstrate that, compared to baseline methods, our VIA approach produces edits that are more faithful to the source videos, more coherent in the spatiotemporal context, and more precise in local control. More importantly, we show that VIA can achieve consistent long video editing in minutes, unlocking the potentials for advanced video editing tasks over long video sequences.