Real2Render2Real：無需動態模擬或機器人硬體即可擴展機器人數據

Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware

May 14, 2025

作者: Justin Yu, Letian Fu, Huang Huang, Karim El-Refai, Rares Andrei Ambrus, Richard Cheng, Muhammad Zubair Irshad, Ken Goldberg

cs.AI

摘要

擴展機器人學習需要龐大且多樣化的數據集。然而，當前主流的數據收集範式——人類遙控操作——仍然成本高昂，並受到人工努力和物理機器人訪問的限制。我們引入了Real2Render2Real（R2R2R），這是一種新穎的方法，用於生成機器人訓練數據，而無需依賴物體動力學模擬或機器人硬件的遙控操作。其輸入為智能手機捕捉的一個或多個物體的掃描以及一段人類示範的單一視頻。R2R2R通過重建詳細的3D物體幾何形狀和外觀，並追蹤6自由度物體運動，渲染出數千個高視覺保真度的機器人無關示範。R2R2R利用3D高斯潑濺（3DGS）來實現靈活的資產生成和軌跡合成，適用於剛性和關節物體，並將這些表示轉換為網格，以保持與可擴展渲染引擎（如IsaacLab）的兼容性，但關閉碰撞建模。由R2R2R生成的機器人示範數據可直接與基於機器人本體感知狀態和圖像觀測的模型集成，例如視覺-語言-動作模型（VLA）和模仿學習策略。物理實驗表明，基於單一人類示範的R2R2R數據訓練的模型，其性能可與基於150次人類遙控操作示範訓練的模型相媲美。項目頁面：https://real2render2real.com

English

Scaling robot learning requires vast and diverse datasets. Yet the prevailing data collection paradigm-human teleoperation-remains costly and constrained by manual effort and physical robot access. We introduce Real2Render2Real (R2R2R), a novel approach for generating robot training data without relying on object dynamics simulation or teleoperation of robot hardware. The input is a smartphone-captured scan of one or more objects and a single video of a human demonstration. R2R2R renders thousands of high visual fidelity robot-agnostic demonstrations by reconstructing detailed 3D object geometry and appearance, and tracking 6-DoF object motion. R2R2R uses 3D Gaussian Splatting (3DGS) to enable flexible asset generation and trajectory synthesis for both rigid and articulated objects, converting these representations to meshes to maintain compatibility with scalable rendering engines like IsaacLab but with collision modeling off. Robot demonstration data generated by R2R2R integrates directly with models that operate on robot proprioceptive states and image observations, such as vision-language-action models (VLA) and imitation learning policies. Physical experiments suggest that models trained on R2R2R data from a single human demonstration can match the performance of models trained on 150 human teleoperation demonstrations. Project page: https://real2render2real.com