EXIMO：视觉语言模型引导的VLA策略探索

摘要

如何高效微调机器人策略以即时学习新任务？最先进的机器人操作策略基于行为克隆，在大型遥操作数据集上对具有数十亿参数的视觉-语言-动作（VLA）模型进行训练。尽管这种简单方法已推动机器人操作取得显著进展，但微调VLA策略以学习新任务仍是一个开放性问题。特别是，收集遥操作数据集需耗费数百小时昂贵的人力，而替代方案——强化学习（RL）——的样本效率众所周知地低下，尤其是在长时程任务中。此外，由于模型规模和架构设计的限制，将RL与VLA结合使用还面临诸多挑战。在本工作中，我们提出EXIMO，一种用于VLA策略微调的高效算法。EXIMO分三个阶段运行：探索、模仿和优化。在探索阶段，EXIMO为VLA配备一个充当规划器的视觉语言模型（VLM）。VLM进行思考并将具有挑战性的长时程问题分解为较短的子问题供VLA处理。VLM与VLA协同工作，用于在新任务上收集编排数据集。在模仿阶段，利用编排数据对VLA进行微调。最后，在优化阶段，我们使用残差离策略强化学习进一步微调策略。在我们的实验中，我们对EXIMO的三个阶段进行了消融研究，结果表明其在样本效率和最终性能方面显著优于现有方法。

English

How to efficiently finetune robot policies to learn new tasks on the fly? State of the art robotic manipulation policies are based on behaviour cloning of large vision-language-action (VLA) models with billions of parameters on huge teleoperation datasets. While this simple approach has enabled significant advances for robotic manipulation, finetuning of VLA policies for learning new tasks still remains an open problem. In particular, collecting teleoperation datasets requires hundreds of hours of expensive human labour and the alternative, reinforcement learning (RL), can be notoriously sample-inefficient especially for long-horizon tasks. In addition, RL with VLAs imposes several challenges due to the model's size and architectural design. In this work, we propose EXIMO, an efficient algorithm for finetuning of VLA policies. EXIMO operates in three stages: explore, imitate, and optimize. During the explore phase, EXIMO equips the VLA with a vision language model (VLM) that acts as a planner. The VLM thinks and breaks down challenging long-horizon problems into shorter ones for the VLA. The VLM, together with the VLA, is used to collect an orchestrated dataset on new tasks. During the imitate phase, the VLA is finetuned with the orchestrated data. Finally, during the optimize stage, we use residual off-policy RL to further finetune the policy. In our experiments, we ablate all three stages of EXIMO and show that it outperforms existing approaches significantly in terms of sample-efficiency and final performance.