Ego2Robot：從自我中心人類數據進行可擴展的機器人數據合成

摘要

學習可泛化的機器人操作策略需要大規模且多樣化的示範數據。第一人稱視角的人類操作影片提供了豐富的場景與任務多樣性，先前研究已表明，將此類影片重新定向並渲染成機器人格式數據，可以在小規模下產出有效的單任務策略。然而，此方法能否為視覺-語言-動作模型在大規模下提供預訓練益處，仍有待探索。我們提出 Ego2Robot，這是一個可擴展的流程，透過動作重定向、機器人手臂視覺合成以及多層級品質篩選，將第一人稱視角的人類操作影片轉換為機器人訓練數據。Ego2Robot 同時支援精選數據集與野外影片，產出橫跨 15 種機器人型態、共 18,561 小時的機器人訓練數據，使其成為迄今規模最大的自我中心至機器人數據集。為評估泛化能力，我們擴展 RoboTwin2.0，納入涵蓋視覺外觀、場景佈局、具身形態與任務語義的解耦擾動軸。實驗顯示，在 Ego2Robot 合成數據與機器人數據上進行聯合預訓練，能一致性地提升多種擾動類型下的分佈外泛化能力，其效益並已在真實機器人部署中獲得驗證。項目頁面：https://www-ye.github.io/ego2robot_blog/

English

Learning generalizable robot manipulation policies requires large-scale and diverse demonstration data. Egocentric human manipulation videos offer rich scene and task diversity, and prior work has shown that retargeting and rendering such videos into robot-format data can yield effective per-task policies at small scale. However, whether this approach can provide pretraining benefits for vision-language-action models at scale remains unexplored. We present Ego2Robot, a scalable pipeline that converts egocentric human manipulation videos into robot training data through action retargeting, robot-arm visual synthesis, and multi-level quality curation. Ego2Robot supports both curated datasets and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies, making it the largest ego-to-robot dataset to date. To evaluate generalization, we extend RoboTwin2.0 with disentangled perturbation axes covering visual appearance, scene layout, embodiment morphology, and task semantics. Experiments show that joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple perturbation types, with benefits validated on real-robot deployment. Project page: https://www-ye.github.io/ego2robot_blog/