Ego2Robot：从自我中心人类数据实现可扩展的机器人数据合成

摘要

学习具有泛化能力的机器人操作策略需要大规模、多样化的示范数据。第一人称视角的人类操作视频蕴含丰富的场景与任务多样性，已有研究表明，将此类视频通过重定向与渲染转换为机器人格式的数据，能够在较小规模上生成有效的单任务策略。然而，该方法能否在规模化层面为视觉-语言-动作模型提供预训练收益，仍尚待探索。我们提出Ego2Robot，一种可扩展的流水线，通过动作重定向、机械臂视觉合成以及多层级质量筛选，将第一人称视角的人类操作视频转化为机器人训练数据。Ego2Robot同时支持精选数据集与野外自然视频，生成了涵盖15种机器人形态、总计18,561小时的机器人训练数据，是迄今最大的从第一视角到机器人的数据集。为评估泛化能力，我们扩展了RoboTwin2.0，引入涵盖视觉外观、场景布局、具身形态及任务语义的解耦扰动维度。实验表明，在Ego2Robot合成数据与真实机器人数据上进行联合预训练，能够持续提升多种扰动类型下的分布外泛化性能，并在真实机器人部署中验证了其有效性。项目主页：https://www-ye.github.io/ego2robot_blog/

English

Learning generalizable robot manipulation policies requires large-scale and diverse demonstration data. Egocentric human manipulation videos offer rich scene and task diversity, and prior work has shown that retargeting and rendering such videos into robot-format data can yield effective per-task policies at small scale. However, whether this approach can provide pretraining benefits for vision-language-action models at scale remains unexplored. We present Ego2Robot, a scalable pipeline that converts egocentric human manipulation videos into robot training data through action retargeting, robot-arm visual synthesis, and multi-level quality curation. Ego2Robot supports both curated datasets and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies, making it the largest ego-to-robot dataset to date. To evaluate generalization, we extend RoboTwin2.0 with disentangled perturbation axes covering visual appearance, scene layout, embodiment morphology, and task semantics. Experiments show that joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple perturbation types, with benefits validated on real-robot deployment. Project page: https://www-ye.github.io/ego2robot_blog/