MotionCanvas：具有可控影像轉影片生成功能的電影鏡頭設計

MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation

February 6, 2025

作者: Jinbo Xing, Long Mai, Cusuh Ham, Jiahui Huang, Aniruddha Mahapatra, Chi-Wing Fu, Tien-Tsin Wong, Feng Liu

cs.AI

摘要

本文提出了一種方法，允許用戶在圖像到視頻生成的背景下設計電影般的視頻鏡頭。鏡頭設計是電影製作中的一個關鍵方面，涉及精心計劃場景中的攝像機運動和物體運動。然而，在現代圖像到視頻生成系統中實現直觀的鏡頭設計面臨兩個主要挑戰：首先，有效捕捉用戶對運動設計的意圖，在這裡攝像機運動和場景中物體運動必須共同指定；其次，表示運動信息，以便視頻擴散模型能夠有效地合成圖像動畫。為應對這些挑戰，我們引入了MotionCanvas，一種將用戶驅動控制整合到圖像到視頻（I2V）生成模型中的方法，使用戶能夠以場景感知方式控制場景中的物體和攝像機運動。通過結合古典計算機圖形學和當代視頻生成技術的見解，我們展示了在I2V合成中實現具有3D感知運動控制的能力，而無需昂貴的3D相關訓練數據。MotionCanvas使用戶能夠直觀地描述場景空間運動意圖，並將其轉換為視頻擴散模型的時空運動條件信號。我們展示了我們的方法在各種真實世界圖像內容和鏡頭設計場景上的有效性，突出了它在數字內容創作中增強創意工作流程並適應各種圖像和視頻編輯應用的潛力。

English

This paper presents a method that allows users to design cinematic video shots in the context of image-to-video generation. Shot design, a critical aspect of filmmaking, involves meticulously planning both camera movements and object motions in a scene. However, enabling intuitive shot design in modern image-to-video generation systems presents two main challenges: first, effectively capturing user intentions on the motion design, where both camera movements and scene-space object motions must be specified jointly; and second, representing motion information that can be effectively utilized by a video diffusion model to synthesize the image animations. To address these challenges, we introduce MotionCanvas, a method that integrates user-driven controls into image-to-video (I2V) generation models, allowing users to control both object and camera motions in a scene-aware manner. By connecting insights from classical computer graphics and contemporary video generation techniques, we demonstrate the ability to achieve 3D-aware motion control in I2V synthesis without requiring costly 3D-related training data. MotionCanvas enables users to intuitively depict scene-space motion intentions, and translates them into spatiotemporal motion-conditioning signals for video diffusion models. We demonstrate the effectiveness of our method on a wide range of real-world image content and shot-design scenarios, highlighting its potential to enhance the creative workflows in digital content creation and adapt to various image and video editing applications.