Real2Render2Real：无需动力学仿真或机器人硬件即可扩展机器人数据

Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware

May 14, 2025

作者: Justin Yu, Letian Fu, Huang Huang, Karim El-Refai, Rares Andrei Ambrus, Richard Cheng, Muhammad Zubair Irshad, Ken Goldberg

cs.AI

摘要

扩展机器人学习需要大量且多样化的数据集。然而，当前主流的数据收集方式——人类远程操控——仍然成本高昂，并受限于人工操作和物理机器人的可及性。我们提出了Real2Render2Real（R2R2R），一种无需依赖物体动力学模拟或机器人硬件远程操控即可生成机器人训练数据的新方法。该方法输入为智能手机扫描的一个或多个物体以及一段人类示范视频。R2R2R通过重建精细的3D物体几何与外观，并追踪6自由度物体运动，渲染出数千个高视觉保真度、与机器人无关的示范。R2R2R利用3D高斯溅射（3DGS）技术，为刚性和铰接物体实现灵活的资产生成与轨迹合成，并将这些表示转换为网格，以保持与可扩展渲染引擎如IsaacLab的兼容性，但关闭了碰撞建模。R2R2R生成的机器人示范数据可直接集成到基于机器人本体感知状态和图像观测的模型中，如视觉-语言-动作模型（VLA）和模仿学习策略。物理实验表明，仅用一次人类示范生成的R2R2R数据训练模型，其性能可媲美基于150次人类远程操控示范训练的模型。项目页面：https://real2render2real.com

English

Scaling robot learning requires vast and diverse datasets. Yet the prevailing data collection paradigm-human teleoperation-remains costly and constrained by manual effort and physical robot access. We introduce Real2Render2Real (R2R2R), a novel approach for generating robot training data without relying on object dynamics simulation or teleoperation of robot hardware. The input is a smartphone-captured scan of one or more objects and a single video of a human demonstration. R2R2R renders thousands of high visual fidelity robot-agnostic demonstrations by reconstructing detailed 3D object geometry and appearance, and tracking 6-DoF object motion. R2R2R uses 3D Gaussian Splatting (3DGS) to enable flexible asset generation and trajectory synthesis for both rigid and articulated objects, converting these representations to meshes to maintain compatibility with scalable rendering engines like IsaacLab but with collision modeling off. Robot demonstration data generated by R2R2R integrates directly with models that operate on robot proprioceptive states and image observations, such as vision-language-action models (VLA) and imitation learning policies. Physical experiments suggest that models trained on R2R2R data from a single human demonstration can match the performance of models trained on 150 human teleoperation demonstrations. Project page: https://real2render2real.com