DreamCinema：具有自由相機和3D角色的電影轉換

DreamCinema: Cinematic Transfer with Free Camera and 3D Character

August 22, 2024

作者: Weiliang Chen, Fangfu Liu, Diankun Wu, Haowen Sun, Haixu Song, Yueqi Duan

cs.AI

摘要

我們正處於數位媒體蓬勃發展的時代，每個人都有潛力成為個人電影製作人。目前關於電影風格轉移的研究賦予製片人重現和操控視覺元素（例如攝影和角色行為）從經典鏡頭中。然而，在重新想像的電影中，角色仍然依賴手動製作，這涉及重大的技術複雜性和高成本，使普通用戶難以實現。此外，他們估計的攝影由於對幀間運動的不足捕捉和物理軌跡建模而缺乏流暢性。幸運的是，2D和3D AIGC的卓越成功開啟了有效生成符合用戶需求的角色、多樣化攝影的可能性。在本文中，我們提出了DreamCinema，一個開創性將生成式人工智慧引入電影製作範式的新型電影風格轉移框架，旨在促進用戶友好的電影創作。具體而言，我們首先提取電影元素（即人類和攝影機姿勢）並優化攝影機軌跡。然後，我們應用角色生成器有效地創建具有人類結構先驗的高質量3D角色。最後，我們開發了一種結構引導的運動轉移策略，將生成的角色納入電影創作並通過3D圖形引擎平滑轉移。大量實驗證明了我們的方法對於創建具有自由攝影機和3D角色的高質量電影的有效性。

English

We are living in a flourishing era of digital media, where everyone has the potential to become a personal filmmaker. Current research on cinematic transfer empowers filmmakers to reproduce and manipulate the visual elements (e.g., cinematography and character behaviors) from classic shots. However, characters in the reimagined films still rely on manual crafting, which involves significant technical complexity and high costs, making it unattainable for ordinary users. Furthermore, their estimated cinematography lacks smoothness due to inadequate capturing of inter-frame motion and modeling of physical trajectories. Fortunately, the remarkable success of 2D and 3D AIGC has opened up the possibility of efficiently generating characters tailored to users' needs, diversifying cinematography. In this paper, we propose DreamCinema, a novel cinematic transfer framework that pioneers generative AI into the film production paradigm, aiming at facilitating user-friendly film creation. Specifically, we first extract cinematic elements (i.e., human and camera pose) and optimize the camera trajectory. Then, we apply a character generator to efficiently create 3D high-quality characters with a human structure prior. Finally, we develop a structure-guided motion transfer strategy to incorporate generated characters into film creation and transfer it via 3D graphics engines smoothly. Extensive experiments demonstrate the effectiveness of our method for creating high-quality films with free camera and 3D characters.