SimRecon：基于真实视频的SimReady组合式场景重建

SimRecon: SimReady Compositional Scene Reconstruction from Real Videos

March 2, 2026

作者: Chong Xia, Kai Zhu, Zizhuo Wang, Fangfu Liu, Zhizheng Zhang, Yueqi Duan

cs.AI

摘要

组合式场景重建旨在从现实世界视频中创建以物体为中心的表示，而非整体场景，这种范式天然适用于仿真与交互任务。传统组合重建方法主要关注视觉外观，对现实场景的泛化能力有限。本文提出SimRecon框架，通过"感知-生成-仿真"三级流水线实现复杂场景重建：首先从视频输入完成场景级语义重建，随后进行单物体生成，最终在仿真器中组装这些资产。然而简单串联这三个阶段会导致生成资产的视觉失真与最终场景的物理失真，该问题在复杂场景中尤为突出。为此，我们特别设计两个桥接模块来衔接三级流程：针对影响视觉真实性的感知到生成阶段过渡，提出主动视角优化方法，通过在三维空间中主动搜索获取最优投影图像作为单物体补全条件；针对决定物理合理性的生成到仿真阶段过渡，提出场景图合成器，指导三维仿真器从零开始构建场景，模拟现实世界固有的构造性原理。在ScanNet数据集上的大量实验表明，本方法显著超越了现有最优方法的性能。

English

Compositional scene reconstruction seeks to create object-centric representations rather than holistic scenes from real-world videos, which is natively applicable for simulation and interaction. Conventional compositional reconstruction approaches primarily emphasize on visual appearance and show limited generalization ability to real-world scenarios. In this paper, we propose SimRecon, a framework that realizes a "Perception-Generation-Simulation" pipeline towards cluttered scene reconstruction, which first conducts scene-level semantic reconstruction from video input, then performs single-object generation, and finally assembles these assets in the simulator. However, naively combining these three stages leads to visual infidelity of generated assets and physical implausibility of the final scene, a problem particularly severe for complex scenes. Thus, we further propose two bridging modules between the three stages to address this problem. To be specific, for the transition from Perception to Generation, critical for visual fidelity, we introduce Active Viewpoint Optimization, which actively searches in 3D space to acquire optimal projected images as conditions for single-object completion. Moreover, for the transition from Generation to Simulation, essential for physical plausibility, we propose a Scene Graph Synthesizer, which guides the construction from scratch in 3D simulators, mirroring the native, constructive principle of the real world. Extensive experiments on the ScanNet dataset validate our method's superior performance over previous state-of-the-art approaches.