MotionCtrl：一种统一灵活的视频生成运动控制器

摘要

视频中的运动主要包括由摄像机移动引起的摄像机运动和由物体移动导致的物体运动。准确控制摄像机和物体运动对于视频生成至关重要。然而，现有研究要么主要关注一种类型的运动，要么未明确区分这两种类型，从而限制了它们的控制能力和多样性。因此，本文提出了MotionCtrl，这是一个统一灵活的视频生成运动控制器，旨在有效独立地控制摄像机和物体运动。MotionCtrl的架构和训练策略经过精心设计，考虑了摄像机运动、物体运动和不完美训练数据的固有属性。与先前的方法相比，MotionCtrl具有三个主要优势：1）它有效独立地控制摄像机运动和物体运动，实现更精细的运动控制，促进两种类型运动的灵活多样组合。2）其运动条件由摄像机姿势和轨迹确定，这些条件与外观无关，最小程度地影响生成视频中物体的外观或形状。3）它是一个相对通用的模型，一旦训练完成，就可以适应各种摄像机姿势和轨迹。进行了大量的定性和定量实验，以展示MotionCtrl相对于现有方法的优越性。

English

Motions in a video primarily consist of camera motion, induced by camera movement, and object motion, resulting from object movement. Accurate control of both camera and object motion is essential for video generation. However, existing works either mainly focus on one type of motion or do not clearly distinguish between the two, limiting their control capabilities and diversity. Therefore, this paper presents MotionCtrl, a unified and flexible motion controller for video generation designed to effectively and independently control camera and object motion. The architecture and training strategy of MotionCtrl are carefully devised, taking into account the inherent properties of camera motion, object motion, and imperfect training data. Compared to previous methods, MotionCtrl offers three main advantages: 1) It effectively and independently controls camera motion and object motion, enabling more fine-grained motion control and facilitating flexible and diverse combinations of both types of motion. 2) Its motion conditions are determined by camera poses and trajectories, which are appearance-free and minimally impact the appearance or shape of objects in generated videos. 3) It is a relatively generalizable model that can adapt to a wide array of camera poses and trajectories once trained. Extensive qualitative and quantitative experiments have been conducted to demonstrate the superiority of MotionCtrl over existing methods.

MotionCtrl：一种统一灵活的视频生成运动控制器

MotionCtrl: A Unified and Flexible Motion Controller for Video Generation

摘要

Support