Voyager：面向可探索3D場景生成的長距離與世界一致性視頻擴散模型

Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation

June 4, 2025

作者: Tianyu Huang, Wangguandong Zheng, Tengfei Wang, Yuhao Liu, Zhenwei Wang, Junta Wu, Jie Jiang, Hui Li, Rynson W. H. Lau, Wangmeng Zuo, Chunchao Guo

cs.AI

摘要

在現實世界的應用中，如視頻遊戲和虛擬現實，通常需要能夠建模3D場景，讓用戶能夠沿著自定義的相機軌跡進行探索。儘管在從文本或圖像生成3D物體方面取得了顯著進展，但創建長距離、3D一致且可探索的3D場景仍然是一個複雜且具有挑戰性的問題。在本研究中，我們提出了Voyager，這是一種新穎的視頻擴散框架，能夠從單一圖像生成世界一致的3D點雲序列，並根據用戶定義的相機路徑進行生成。與現有方法不同，Voyager實現了端到端的場景生成與重建，具有跨幀的內在一致性，消除了對3D重建流程（如結構從運動或多視圖立體）的需求。我們的方法整合了三個關鍵組件：1）世界一致的視頻擴散：一個統一的架構，聯合生成對齊的RGB和深度視頻序列，基於現有的世界觀測來確保全局一致性；2）長距離世界探索：一個高效的世界緩存，帶有點雲剔除和自動回歸推理，通過平滑的視頻採樣進行迭代場景擴展，保持上下文感知的一致性；3）可擴展的數據引擎：一個視頻重建流程，自動化相機姿態估計和度量深度預測，適用於任意視頻，實現大規模、多樣化的訓練數據收集，無需手動3D註釋。這些設計共同帶來了視覺質量和幾何精確度上的明顯提升，具有廣泛的應用前景。

English

Real-world applications like video gaming and virtual reality often demand the ability to model 3D scenes that users can explore along custom camera trajectories. While significant progress has been made in generating 3D objects from text or images, creating long-range, 3D-consistent, explorable 3D scenes remains a complex and challenging problem. In this work, we present Voyager, a novel video diffusion framework that generates world-consistent 3D point-cloud sequences from a single image with user-defined camera path. Unlike existing approaches, Voyager achieves end-to-end scene generation and reconstruction with inherent consistency across frames, eliminating the need for 3D reconstruction pipelines (e.g., structure-from-motion or multi-view stereo). Our method integrates three key components: 1) World-Consistent Video Diffusion: A unified architecture that jointly generates aligned RGB and depth video sequences, conditioned on existing world observation to ensure global coherence 2) Long-Range World Exploration: An efficient world cache with point culling and an auto-regressive inference with smooth video sampling for iterative scene extension with context-aware consistency, and 3) Scalable Data Engine: A video reconstruction pipeline that automates camera pose estimation and metric depth prediction for arbitrary videos, enabling large-scale, diverse training data curation without manual 3D annotations. Collectively, these designs result in a clear improvement over existing methods in visual quality and geometric accuracy, with versatile applications.