ReCamMaster：基於單一視頻的相機控制生成式渲染

ReCamMaster: Camera-Controlled Generative Rendering from A Single Video

March 14, 2025

作者: Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, Di Zhang

cs.AI

摘要

在文本或圖像條件下的視頻生成任務中，相機控制已被積極研究。然而，儘管在視頻創作領域中具有重要性，改變給定視頻的相機軌跡仍然未被充分探索。這是由於需要維持多幀外觀和動態同步的額外約束，使得這一任務非比尋常。為此，我們提出了ReCamMaster，這是一個相機控制的生成式視頻重渲染框架，能夠在新穎的相機軌跡上重現輸入視頻的動態場景。其核心創新在於利用預訓練的文本到視頻模型的生成能力，通過一個簡單而強大的視頻條件機制——這一能力在當前研究中常被忽視。為克服合格訓練數據的稀缺性，我們使用Unreal Engine 5構建了一個全面的多相機同步視頻數據集，該數據集精心策劃以遵循現實世界的拍攝特性，涵蓋了多樣化的場景和相機運動。這有助於模型泛化到野外視頻。最後，我們通過精心設計的訓練策略進一步提高了對多樣化輸入的魯棒性。大量實驗表明，我們的方法顯著優於現有的最先進方法和強基線。我們的方法還在視頻穩定、超分辨率和外繪等領域找到了有前景的應用。項目頁面：https://jianhongbai.github.io/ReCamMaster/

English

Camera control has been actively studied in text or image conditioned video generation tasks. However, altering camera trajectories of a given video remains under-explored, despite its importance in the field of video creation. It is non-trivial due to the extra constraints of maintaining multiple-frame appearance and dynamic synchronization. To address this, we present ReCamMaster, a camera-controlled generative video re-rendering framework that reproduces the dynamic scene of an input video at novel camera trajectories. The core innovation lies in harnessing the generative capabilities of pre-trained text-to-video models through a simple yet powerful video conditioning mechanism -- its capability often overlooked in current research. To overcome the scarcity of qualified training data, we construct a comprehensive multi-camera synchronized video dataset using Unreal Engine 5, which is carefully curated to follow real-world filming characteristics, covering diverse scenes and camera movements. It helps the model generalize to in-the-wild videos. Lastly, we further improve the robustness to diverse inputs through a meticulously designed training strategy. Extensive experiments tell that our method substantially outperforms existing state-of-the-art approaches and strong baselines. Our method also finds promising applications in video stabilization, super-resolution, and outpainting. Project page: https://jianhongbai.github.io/ReCamMaster/