EgoSteer：面向第一人称视频的可控灵巧操作全栈系统

摘要

可操控性是通用机器人策略的核心能力，但由于缺乏大规模、语言对齐且动作精准的演示数据，这一能力在灵巧手系统中仍然基本缺失。为突破这一瓶颈，我们提出了一套全栈系统，该系统从第一人称人类视频中扩展灵巧VLA预训练，并实现了数据高效的实体机器人后训练。它集成了三个核心组件：EgoSmith数据管道——将野外第一人称视频整理成9600小时的高质量预训练数据，吞吐量较此前最优方法提升9倍且精度更高；一套用于遥操作和人在环路校正的统一机器人框架；以及EgoSteer——基于优化基础设施训练的、由世界模型增强的VLA模型。人类数据预训练赋予了EgoSteer语言引导的操作先验知识，这些先验知识通过机器人后训练落地，并借助DAgger改进得到优化。实验结果表明，EgoSteer能够稳健地执行40余种不同任务的自由形式指令，展现出故障恢复、灵巧操作和泛化能力。该预训练模型还能在两种实体上通过少样本学习适应复杂的长时程任务（包括折叠箱子），成功率超过75%。我们已在https://egosteer.github.io/开源了该系统、数据和模型。

English

Steerability is a defining capability of generalist robot policies, yet remains largely absent in dexterous-hand systems for lack of large-scale, language-aligned, and action-accurate demonstration data. To address this bottleneck, we present a full-stack system that scales dexterous VLA pre-training from egocentric human videos and enables data-efficient real-robot post-training. It integrates EgoSmith, a data pipeline that curates in-the-wild egocentric videos into 9.6K hours of high-quality pre-training data with 9x higher throughput and better accuracy than prior SOTA; a unified robot stack for teleoperation and human-in-the-loop correction; and EgoSteer, a world-model-enhanced VLA trained on optimized infrastructure. Human-data pre-training equips EgoSteer with language-guided manipulation priors, which are grounded through robot post-training and improved by DAgger refinement. Empirically, EgoSteer robustly executes free-form instructions across 40+ diverse tasks, demonstrating failure recovery, dexterity, and generalization. The pre-trained model also few-shot adapts to complex long-horizon tasks, including box folding, on two embodiments with 75+% success. We open-source the system, data, and model at https://egosteer.github.io/.