ReCamMaster：基于单视频的相机控制生成式渲染
ReCamMaster: Camera-Controlled Generative Rendering from A Single Video
March 14, 2025
作者: Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, Di Zhang
cs.AI
摘要
在文本或图像条件约束的视频生成任务中，相机控制已得到深入研究。然而，尽管在视频创作领域具有重要意义，对给定视频的相机轨迹进行修改仍属探索不足。这一挑战源于需同时维护多帧外观与动态同步的额外约束。为此，我们提出了ReCamMaster，一个相机控制的生成式视频重渲染框架，它能在新颖的相机轨迹下重现输入视频的动态场景。其核心创新在于，通过一种简单而强大的视频条件机制，充分利用了预训练文本到视频模型的生成能力——这一能力在当前研究中常被忽视。为应对高质量训练数据的稀缺，我们利用Unreal Engine 5构建了一个全面的多相机同步视频数据集，该数据集精心设计以遵循现实世界的拍摄特征，涵盖多样化的场景与相机运动，有助于模型泛化至真实场景视频。最后，通过精心设计的训练策略，我们进一步提升了模型对多样化输入的鲁棒性。大量实验表明，我们的方法显著超越了现有的最先进方法与强基线。此外，我们的方法在视频稳定、超分辨率及外延绘制等方面展现出广阔的应用前景。项目页面：https://jianhongbai.github.io/ReCamMaster/
English
Camera control has been actively studied in text or image conditioned video
generation tasks. However, altering camera trajectories of a given video
remains under-explored, despite its importance in the field of video creation.
It is non-trivial due to the extra constraints of maintaining multiple-frame
appearance and dynamic synchronization. To address this, we present
ReCamMaster, a camera-controlled generative video re-rendering framework that
reproduces the dynamic scene of an input video at novel camera trajectories.
The core innovation lies in harnessing the generative capabilities of
pre-trained text-to-video models through a simple yet powerful video
conditioning mechanism -- its capability often overlooked in current research.
To overcome the scarcity of qualified training data, we construct a
comprehensive multi-camera synchronized video dataset using Unreal Engine 5,
which is carefully curated to follow real-world filming characteristics,
covering diverse scenes and camera movements. It helps the model generalize to
in-the-wild videos. Lastly, we further improve the robustness to diverse inputs
through a meticulously designed training strategy. Extensive experiments tell
that our method substantially outperforms existing state-of-the-art approaches
and strong baselines. Our method also finds promising applications in video
stabilization, super-resolution, and outpainting. Project page:
https://jianhongbai.github.io/ReCamMaster/