跟随点击：通过简短提示实现开放领域区域图像动画

Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts

March 13, 2024

作者: Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung-Yeung Shum, Wei Liu, Qifeng Chen

cs.AI

摘要

尽管图像到视频生成取得了一些进展，但更好的可控性和局部动画却鲜有探索。大多数现有的图像到视频方法并不具备局部意识，往往会移动整个场景。然而，人类艺术家可能需要控制不同对象或区域的运动。此外，当前的图像到视频方法不仅要求用户描述目标运动，还要提供冗余的详细帧内容描述。这两个问题阻碍了当前图像到视频工具的实际应用。在本文中，我们提出了一个实用框架，名为Follow-Your-Click，通过简单的用户点击（用于指定移动对象）和简短的运动提示（用于指定如何移动）来实现图像动画。在技术上，我们提出了首帧遮罩策略，显著提高了视频生成质量，并配备了一个短运动提示数据集的运动增强模块，以提高我们模型的短提示跟随能力。为了进一步控制运动速度，我们提出了基于流的运动幅度控制，以更精确地控制目标运动的速度。我们的框架具有更简单但精确的用户控制，以及比先前方法更好的生成性能。与包括商业工具和研究方法在内的7个基准进行了广泛实验，涉及8个指标，结果表明我们方法的优越性。项目页面：https://follow-your-click.github.io/

English

Despite recent advances in image-to-video generation, better controllability and local animation are less explored. Most existing image-to-video methods are not locally aware and tend to move the entire scene. However, human artists may need to control the movement of different objects or regions. Additionally, current I2V methods require users not only to describe the target motion but also to provide redundant detailed descriptions of frame contents. These two issues hinder the practical utilization of current I2V tools. In this paper, we propose a practical framework, named Follow-Your-Click, to achieve image animation with a simple user click (for specifying what to move) and a short motion prompt (for specifying how to move). Technically, we propose the first-frame masking strategy, which significantly improves the video generation quality, and a motion-augmented module equipped with a short motion prompt dataset to improve the short prompt following abilities of our model. To further control the motion speed, we propose flow-based motion magnitude control to control the speed of target movement more precisely. Our framework has simpler yet precise user control and better generation performance than previous methods. Extensive experiments compared with 7 baselines, including both commercial tools and research methods on 8 metrics, suggest the superiority of our approach. Project Page: https://follow-your-click.github.io/