光场视频生成

摘要

相机控制的生成式视频重渲染方法（如ReCamMaster）已取得显著进展。然而尽管在单视角设置中表现成功，这些方法在多视角场景下往往难以保持一致性。由于生成模型固有的随机性，确保幻觉区域的时空连贯性仍是挑战。为此，我们提出PlenopticDreamer框架，通过同步生成式幻觉来维持时空记忆。其核心思想是以自回归方式训练多输入单输出的视频条件模型，辅以相机引导的视频检索策略——该策略能自适应地从先前生成结果中选择显著视频作为条件输入。此外，我们的训练方案包含渐进式上下文扩展以提升收敛性，自条件机制以增强对误差累积导致的长程视觉退化的鲁棒性，以及长视频条件机制以支持扩展视频生成。在Basic和Agibot基准测试上的大量实验表明，PlenopticDreamer实现了最先进的视频重渲染效果，在视角同步性、视觉保真度、相机控制精度和多样化视角转换（如第三人称到第三人称、机器人操作中头部视角到夹爪视角）方面均表现优异。项目页面：https://research.nvidia.com/labs/dir/plenopticdreamer/

English

Camera-controlled generative video re-rendering methods, such as ReCamMaster, have achieved remarkable progress. However, despite their success in single-view setting, these works often struggle to maintain consistency across multi-view scenarios. Ensuring spatio-temporal coherence in hallucinated regions remains challenging due to the inherent stochasticity of generative models. To address it, we introduce PlenopticDreamer, a framework that synchronizes generative hallucinations to maintain spatio-temporal memory. The core idea is to train a multi-in-single-out video-conditioned model in an autoregressive manner, aided by a camera-guided video retrieval strategy that adaptively selects salient videos from previous generations as conditional inputs. In addition, Our training incorporates progressive context-scaling to improve convergence, self-conditioning to enhance robustness against long-range visual degradation caused by error accumulation, and a long-video conditioning mechanism to support extended video generation. Extensive experiments on the Basic and Agibot benchmarks demonstrate that PlenopticDreamer achieves state-of-the-art video re-rendering, delivering superior view synchronization, high-fidelity visuals, accurate camera control, and diverse view transformations (e.g., third-person to third-person, and head-view to gripper-view in robotic manipulation). Project page: https://research.nvidia.com/labs/dir/plenopticdreamer/

光场视频生成

Plenoptic Video Generation

摘要

Support