基于高斯溅射的高保真模拟数据生成，助力现实世界零样本机器人操作学习

High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic Manipulation Learning with Gaussian Splatting

October 12, 2025

作者: Haoyu Zhao, Cheng Zeng, Linghao Zhuang, Yaxi Zhao, Shengke Xue, Hao Wang, Xingyue Zhao, Zhongyu Li, Kehan Li, Siteng Huang, Mingxiu Chen, Xin Li, Deli Zhao, Hua Zou

cs.AI

摘要

机器人学习的可扩展性从根本上受到现实世界数据收集的高成本和大量人力投入的限制。尽管模拟数据提供了一种可扩展的替代方案，但由于视觉外观、物理属性及物体交互方面的显著差异，其往往难以泛化到现实世界。为解决这一问题，我们提出了RoboSimGS，一种创新的Real2Sim2Real框架，该框架将多视角现实世界图像转化为可扩展、高保真且支持物理交互的模拟环境，专为机器人操作设计。我们的方法采用混合表示重建场景：3D高斯溅射（3DGS）捕捉环境的逼真外观，而交互对象的网格基元则确保了精确的物理模拟。尤为关键的是，我们率先利用多模态大语言模型（MLLM）来自动生成物理上合理、可关节化的资产。MLLM通过分析视觉数据，不仅推断出物体的物理属性（如密度、刚度），还能识别复杂的运动学结构（如铰链、滑轨）。我们证明，完全基于RoboSimGS生成数据训练的策略，在多种现实世界操作任务中实现了成功的零样本模拟到现实迁移。此外，RoboSimGS的数据显著提升了当前最先进方法的性能与泛化能力。我们的结果验证了RoboSimGS作为弥合模拟与现实差距的强大且可扩展解决方案的有效性。

English

The scalability of robotic learning is fundamentally bottlenecked by the significant cost and labor of real-world data collection. While simulated data offers a scalable alternative, it often fails to generalize to the real world due to significant gaps in visual appearance, physical properties, and object interactions. To address this, we propose RoboSimGS, a novel Real2Sim2Real framework that converts multi-view real-world images into scalable, high-fidelity, and physically interactive simulation environments for robotic manipulation. Our approach reconstructs scenes using a hybrid representation: 3D Gaussian Splatting (3DGS) captures the photorealistic appearance of the environment, while mesh primitives for interactive objects ensure accurate physics simulation. Crucially, we pioneer the use of a Multi-modal Large Language Model (MLLM) to automate the creation of physically plausible, articulated assets. The MLLM analyzes visual data to infer not only physical properties (e.g., density, stiffness) but also complex kinematic structures (e.g., hinges, sliding rails) of objects. We demonstrate that policies trained entirely on data generated by RoboSimGS achieve successful zero-shot sim-to-real transfer across a diverse set of real-world manipulation tasks. Furthermore, data from RoboSimGS significantly enhances the performance and generalization capabilities of SOTA methods. Our results validate RoboSimGS as a powerful and scalable solution for bridging the sim-to-real gap.