MotionCtrl：一個統一且靈活的影片生成運動控制器

摘要

影片中的運動主要包括由攝影機移動引起的攝影機運動和由物體移動導致的物體運動。精確控制攝影機和物體運動對於影片生成至關重要。然而，現有研究要麼主要專注於一種類型的運動，要麼沒有清楚區分這兩種類型，限制了它們的控制能力和多樣性。因此，本文提出了MotionCtrl，這是一個統一且靈活的運動控制器，旨在有效且獨立地控制攝影機和物體運動以進行影片生成。MotionCtrl的架構和訓練策略經過精心設計，考慮了攝影機運動、物體運動和不完美的訓練數據的固有特性。與先前的方法相比，MotionCtrl具有三個主要優勢：1）它有效且獨立地控制攝影機運動和物體運動，實現更精細的運動控制，促進兩種類型運動的靈活和多樣組合。2）其運動條件由攝影機姿勢和軌跡確定，這些條件與外觀無關，對生成的影片中的物體外觀或形狀影響最小。3）它是一個相對通用的模型，一旦訓練完畢，可以適應各種攝影機姿勢和軌跡。已進行了廣泛的定性和定量實驗，以證明MotionCtrl相對於現有方法的優越性。

English

Motions in a video primarily consist of camera motion, induced by camera movement, and object motion, resulting from object movement. Accurate control of both camera and object motion is essential for video generation. However, existing works either mainly focus on one type of motion or do not clearly distinguish between the two, limiting their control capabilities and diversity. Therefore, this paper presents MotionCtrl, a unified and flexible motion controller for video generation designed to effectively and independently control camera and object motion. The architecture and training strategy of MotionCtrl are carefully devised, taking into account the inherent properties of camera motion, object motion, and imperfect training data. Compared to previous methods, MotionCtrl offers three main advantages: 1) It effectively and independently controls camera motion and object motion, enabling more fine-grained motion control and facilitating flexible and diverse combinations of both types of motion. 2) Its motion conditions are determined by camera poses and trajectories, which are appearance-free and minimally impact the appearance or shape of objects in generated videos. 3) It is a relatively generalizable model that can adapt to a wide array of camera poses and trajectories once trained. Extensive qualitative and quantitative experiments have been conducted to demonstrate the superiority of MotionCtrl over existing methods.

MotionCtrl：一個統一且靈活的影片生成運動控制器

MotionCtrl: A Unified and Flexible Motion Controller for Video Generation

摘要

Support