光场视频生成

摘要

相機控制的生成式視頻重渲染方法（如ReCamMaster）已取得顯著進展。然而，儘管在單視角設定中表現出色，這類方法在多視角場景下往往難以保持一致性。由於生成模型固有的隨機性，確保虛構區域的時空連貫性仍是挑戰。為此，我們提出PlenopticDreamer框架，通過同步生成式幻覺來維持時空記憶。其核心思想是採用自回歸方式訓練多輸入單輸出的視頻條件模型，並輔以相機引導的視頻檢索策略——該策略能自適應地選取過往生成中的顯著視頻作為條件輸入。此外，我們在訓練中融入三項關鍵技術：通過漸進式上下文擴展提升收斂效率，採用自條件機制抵禦誤差累積導致的長程視覺退化，以及引入長視頻條件機制支持擴展視頻生成。在Basic與Agibot基準上的大量實驗表明，PlenopticDreamer實現了業界領先的視頻重渲染效果，在視角同步性、視覺保真度、相機控制精度及多樣化視角轉換（如第三人稱視角互轉、機械臂操作中頭部視角到夾爪視角的轉換）方面均表現優異。項目頁面：https://research.nvidia.com/labs/dir/plenopticdreamer/

English

Camera-controlled generative video re-rendering methods, such as ReCamMaster, have achieved remarkable progress. However, despite their success in single-view setting, these works often struggle to maintain consistency across multi-view scenarios. Ensuring spatio-temporal coherence in hallucinated regions remains challenging due to the inherent stochasticity of generative models. To address it, we introduce PlenopticDreamer, a framework that synchronizes generative hallucinations to maintain spatio-temporal memory. The core idea is to train a multi-in-single-out video-conditioned model in an autoregressive manner, aided by a camera-guided video retrieval strategy that adaptively selects salient videos from previous generations as conditional inputs. In addition, Our training incorporates progressive context-scaling to improve convergence, self-conditioning to enhance robustness against long-range visual degradation caused by error accumulation, and a long-video conditioning mechanism to support extended video generation. Extensive experiments on the Basic and Agibot benchmarks demonstrate that PlenopticDreamer achieves state-of-the-art video re-rendering, delivering superior view synchronization, high-fidelity visuals, accurate camera control, and diverse view transformations (e.g., third-person to third-person, and head-view to gripper-view in robotic manipulation). Project page: https://research.nvidia.com/labs/dir/plenopticdreamer/

光场视频生成

Plenoptic Video Generation

摘要

Support