基於高斯潑濺技術的高保真模擬數據生成，用於現實世界中的零樣本機器人操作學習

High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic Manipulation Learning with Gaussian Splatting

October 12, 2025

作者: Haoyu Zhao, Cheng Zeng, Linghao Zhuang, Yaxi Zhao, Shengke Xue, Hao Wang, Xingyue Zhao, Zhongyu Li, Kehan Li, Siteng Huang, Mingxiu Chen, Xin Li, Deli Zhao, Hua Zou

cs.AI

摘要

機器人學習的可擴展性根本上受到現實世界數據收集的高成本和勞動力投入的限制。雖然模擬數據提供了一種可擴展的替代方案，但由於視覺外觀、物理屬性和物體交互之間存在顯著差距，它往往難以泛化到現實世界。為了解決這一問題，我們提出了RoboSimGS，這是一種新穎的Real2Sim2Real框架，能夠將多視角現實世界圖像轉換為可擴展、高保真且具有物理交互性的模擬環境，用於機器人操作。我們的方法採用混合表示來重建場景：3D高斯濺射（3DGS）捕捉環境的逼真外觀，而交互物體的網格基元則確保了精確的物理模擬。關鍵的是，我們率先使用多模態大語言模型（MLLM）來自動創建物理上合理的關節化資產。MLLM分析視覺數據，不僅推斷物體的物理屬性（如密度、剛度），還推斷其複雜的運動學結構（如鉸鏈、滑動軌道）。我們證明，完全基於RoboSimGS生成的數據訓練的策略在多樣化的現實世界操作任務中實現了成功的零樣本模擬到現實的遷移。此外，來自RoboSimGS的數據顯著提升了最先進方法的性能和泛化能力。我們的結果驗證了RoboSimGS作為一種強大且可擴展的解決方案，能夠有效彌合模擬與現實之間的差距。

English

The scalability of robotic learning is fundamentally bottlenecked by the significant cost and labor of real-world data collection. While simulated data offers a scalable alternative, it often fails to generalize to the real world due to significant gaps in visual appearance, physical properties, and object interactions. To address this, we propose RoboSimGS, a novel Real2Sim2Real framework that converts multi-view real-world images into scalable, high-fidelity, and physically interactive simulation environments for robotic manipulation. Our approach reconstructs scenes using a hybrid representation: 3D Gaussian Splatting (3DGS) captures the photorealistic appearance of the environment, while mesh primitives for interactive objects ensure accurate physics simulation. Crucially, we pioneer the use of a Multi-modal Large Language Model (MLLM) to automate the creation of physically plausible, articulated assets. The MLLM analyzes visual data to infer not only physical properties (e.g., density, stiffness) but also complex kinematic structures (e.g., hinges, sliding rails) of objects. We demonstrate that policies trained entirely on data generated by RoboSimGS achieve successful zero-shot sim-to-real transfer across a diverse set of real-world manipulation tasks. Furthermore, data from RoboSimGS significantly enhances the performance and generalization capabilities of SOTA methods. Our results validate RoboSimGS as a powerful and scalable solution for bridging the sim-to-real gap.