DreamCinema：自由摄像机和3D角色的电影化转换

DreamCinema: Cinematic Transfer with Free Camera and 3D Character

August 22, 2024

作者: Weiliang Chen, Fangfu Liu, Diankun Wu, Haowen Sun, Haixu Song, Yueqi Duan

cs.AI

摘要

我们正处于数字媒体蓬勃发展的时代，在这个时代，每个人都有可能成为个人电影制作人。关于电影风格转移的当前研究赋予电影制作人重新制作和操纵经典镜头中的视觉元素（如摄影和角色行为）的能力。然而，重新想象电影中的角色仍然依赖于手工制作，这涉及到重要的技术复杂性和高成本，使普通用户难以实现。此外，由于对帧间运动的捕捉不足和物理轨迹建模不足，他们估计的摄影术缺乏流畅性。幸运的是，2D和3D人工智能生成角色（AIGC）取得了显著成功，为高效生成符合用户需求的角色、丰富摄影术打开了可能性。在本文中，我们提出了DreamCinema，这是一个开创性地将生成式人工智能引入电影制作范式的新型电影风格转移框架，旨在促进用户友好的电影创作。具体而言，我们首先提取电影元素（即人类和摄像机姿势）并优化摄像机轨迹。然后，我们应用一个角色生成器高效地创建具有人类结构先验的3D高质量角色。最后，我们开发了一个结构引导的运动转移策略，将生成的角色整合到电影创作中，并通过3D图形引擎平稳地转移。大量实验证明了我们的方法在创建具有自由摄像机和3D角色的高质量电影方面的有效性。

English

We are living in a flourishing era of digital media, where everyone has the potential to become a personal filmmaker. Current research on cinematic transfer empowers filmmakers to reproduce and manipulate the visual elements (e.g., cinematography and character behaviors) from classic shots. However, characters in the reimagined films still rely on manual crafting, which involves significant technical complexity and high costs, making it unattainable for ordinary users. Furthermore, their estimated cinematography lacks smoothness due to inadequate capturing of inter-frame motion and modeling of physical trajectories. Fortunately, the remarkable success of 2D and 3D AIGC has opened up the possibility of efficiently generating characters tailored to users' needs, diversifying cinematography. In this paper, we propose DreamCinema, a novel cinematic transfer framework that pioneers generative AI into the film production paradigm, aiming at facilitating user-friendly film creation. Specifically, we first extract cinematic elements (i.e., human and camera pose) and optimize the camera trajectory. Then, we apply a character generator to efficiently create 3D high-quality characters with a human structure prior. Finally, we develop a structure-guided motion transfer strategy to incorporate generated characters into film creation and transfer it via 3D graphics engines smoothly. Extensive experiments demonstrate the effectiveness of our method for creating high-quality films with free camera and 3D characters.