跟隨點擊：通過簡短提示實現開放領域區域圖像動畫

Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts

March 13, 2024

作者: Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung-Yeung Shum, Wei Liu, Qifeng Chen

cs.AI

摘要

儘管近年來在影像轉視頻生成方面取得了進展，但更好的可控性和局部動畫卻鮮少被探索。大多數現有的影像轉視頻方法並不具備局部感知能力，往往會移動整個場景。然而，人類藝術家可能需要控制不同物體或區域的運動。此外，目前的影像轉視頻方法不僅要求用戶描述目標運動，還需要提供冗贅的對幀內容的詳細描述。這兩個問題阻礙了目前影像轉視頻工具的實際應用。本文提出了一個名為「Follow-Your-Click」的實用框架，通過簡單的用戶點擊（用於指定移動對象）和簡短的運動提示（用於指定如何移動）來實現影像動畫。技術上，我們提出了首幀遮罩策略，顯著提高了視頻生成質量，並配備了一個短運動提示數據集的運動增強模塊，以提高我們模型對短提示的跟隨能力。為了進一步控制運動速度，我們提出了基於流的運動幅度控制，以更精確地控制目標運動的速度。我們的框架具有更簡單但精確的用戶控制，並且比以前的方法具有更好的生成性能。與7個基準方法（包括商業工具和研究方法）在8個指標上進行的大量實驗表明了我們方法的優越性。項目頁面：https://follow-your-click.github.io/

English

Despite recent advances in image-to-video generation, better controllability and local animation are less explored. Most existing image-to-video methods are not locally aware and tend to move the entire scene. However, human artists may need to control the movement of different objects or regions. Additionally, current I2V methods require users not only to describe the target motion but also to provide redundant detailed descriptions of frame contents. These two issues hinder the practical utilization of current I2V tools. In this paper, we propose a practical framework, named Follow-Your-Click, to achieve image animation with a simple user click (for specifying what to move) and a short motion prompt (for specifying how to move). Technically, we propose the first-frame masking strategy, which significantly improves the video generation quality, and a motion-augmented module equipped with a short motion prompt dataset to improve the short prompt following abilities of our model. To further control the motion speed, we propose flow-based motion magnitude control to control the speed of target movement more precisely. Our framework has simpler yet precise user control and better generation performance than previous methods. Extensive experiments compared with 7 baselines, including both commercial tools and research methods on 8 metrics, suggest the superiority of our approach. Project Page: https://follow-your-click.github.io/