HoloScene：从单一视频生成即用型交互式3D世界

HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video

October 7, 2025

作者: Hongchi Xia, Chih-Hao Lin, Hao-Yu Hsu, Quentin Leboutet, Katelyn Gao, Michael Paulitsch, Benjamin Ummenhofer, Shenlong Wang

cs.AI

摘要

将物理世界精确地数字化为仿真就绪的虚拟环境，在增强现实、虚拟现实、游戏及机器人等多个领域展现出巨大潜力。然而，现有的三维重建与场景理解方法往往在几何完整性、物体交互性、物理合理性、照片级真实感渲染或可靠动态模拟所需的真实物理属性等关键方面存在不足。为应对这些挑战，我们提出了HoloScene，一种创新的交互式三维重建框架，能够同时满足上述所有要求。HoloScene采用了一种全面的交互式场景图表示法，不仅编码了物体的几何形状、外观及物理属性，还囊括了层级结构与物体间的关系。重建过程被构建为一个基于能量的优化问题，将观测数据、物理约束与生成先验统一整合进一个连贯的目标函数中。通过结合采样探索与梯度优化的混合策略，优化过程得以高效执行。由此生成的数字孪生体展现出完整精确的几何结构、物理稳定性以及从新视角观察时的逼真渲染效果。在多个基准数据集上的评估验证了其卓越性能，而在互动游戏与实时数字孪生操作中的实际应用案例，则进一步彰显了HoloScene广泛的适用性与高效性。项目页面：https://xiahongchi.github.io/HoloScene。

English

Digitizing the physical world into accurate simulation-ready virtual environments offers significant opportunities in a variety of fields such as augmented and virtual reality, gaming, and robotics. However, current 3D reconstruction and scene-understanding methods commonly fall short in one or more critical aspects, such as geometry completeness, object interactivity, physical plausibility, photorealistic rendering, or realistic physical properties for reliable dynamic simulation. To address these limitations, we introduce HoloScene, a novel interactive 3D reconstruction framework that simultaneously achieves these requirements. HoloScene leverages a comprehensive interactive scene-graph representation, encoding object geometry, appearance, and physical properties alongside hierarchical and inter-object relationships. Reconstruction is formulated as an energy-based optimization problem, integrating observational data, physical constraints, and generative priors into a unified, coherent objective. Optimization is efficiently performed via a hybrid approach combining sampling-based exploration with gradient-based refinement. The resulting digital twins exhibit complete and precise geometry, physical stability, and realistic rendering from novel viewpoints. Evaluations conducted on multiple benchmark datasets demonstrate superior performance, while practical use-cases in interactive gaming and real-time digital-twin manipulation illustrate HoloScene's broad applicability and effectiveness. Project page: https://xiahongchi.github.io/HoloScene.