HoloScene：從單一視頻生成即時可交互的3D模擬場景

HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video

October 7, 2025

作者: Hongchi Xia, Chih-Hao Lin, Hao-Yu Hsu, Quentin Leboutet, Katelyn Gao, Michael Paulitsch, Benjamin Ummenhofer, Shenlong Wang

cs.AI

摘要

將物理世界數位化為精確且適合模擬的虛擬環境，在增強現實、虛擬現實、遊戲和機器人等多個領域提供了重大機遇。然而，當前的三維重建與場景理解方法通常在一或多個關鍵方面存在不足，如幾何完整性、物體交互性、物理合理性、照片級真實感渲染，或缺乏可靠的動態模擬所需的真實物理屬性。為解決這些限制，我們引入了HoloScene，這是一種新穎的交互式三維重建框架，能夠同時滿足上述所有要求。HoloScene利用全面的交互式場景圖表示，編碼物體的幾何形狀、外觀和物理屬性，以及層次結構和物體間的關係。重建被表述為一個基於能量的優化問題，將觀測數據、物理約束和生成先驗整合到一個統一且連貫的目標函數中。通過結合基於採樣的探索與基於梯度的細化，實現了高效的優化過程。由此產生的數字孿生體展現出完整精確的幾何形狀、物理穩定性，以及從新視角觀看的真實感渲染。在多個基準數據集上的評估顯示了其卓越的性能，而在交互式遊戲和實時數字孿生操作中的實際應用案例，則展示了HoloScene廣泛的適用性和有效性。項目頁面：https://xiahongchi.github.io/HoloScene。

English

Digitizing the physical world into accurate simulation-ready virtual environments offers significant opportunities in a variety of fields such as augmented and virtual reality, gaming, and robotics. However, current 3D reconstruction and scene-understanding methods commonly fall short in one or more critical aspects, such as geometry completeness, object interactivity, physical plausibility, photorealistic rendering, or realistic physical properties for reliable dynamic simulation. To address these limitations, we introduce HoloScene, a novel interactive 3D reconstruction framework that simultaneously achieves these requirements. HoloScene leverages a comprehensive interactive scene-graph representation, encoding object geometry, appearance, and physical properties alongside hierarchical and inter-object relationships. Reconstruction is formulated as an energy-based optimization problem, integrating observational data, physical constraints, and generative priors into a unified, coherent objective. Optimization is efficiently performed via a hybrid approach combining sampling-based exploration with gradient-based refinement. The resulting digital twins exhibit complete and precise geometry, physical stability, and realistic rendering from novel viewpoints. Evaluations conducted on multiple benchmark datasets demonstrate superior performance, while practical use-cases in interactive gaming and real-time digital-twin manipulation illustrate HoloScene's broad applicability and effectiveness. Project page: https://xiahongchi.github.io/HoloScene.