EXIMO：VLM引導的VLA策略探索

摘要

如何高效微調機器人策略以即時學習新任務？目前最先進的機器人操作策略是基於行為克隆，即在龐大的遙操作資料集上訓練具有數十億參數的大型視覺-語言-動作（VLA）模型。雖然這種簡單方法已推動機器人操作取得顯著進展，但微調VLA策略以學習新任務仍然是個待解決的問題。具體而言，收集遙操作資料集需要數百小時昂貴的人力，而另一種替代方案——強化學習（RL）——則以樣本效率低下而聞名，尤其在長時程任務中。此外，將RL用於VLA會因模型規模與架構設計而帶來諸多挑戰。在本研究中，我們提出EXIMO，一種用於微調VLA策略的高效演算法。EXIMO分三個階段運作：探索、模仿與優化。在探索階段，EXIMO為VLA配備一個作為規劃器的視覺語言模型（VLM）。VLM會思考，並將具挑戰性的長時程問題分解為較短的子問題，以供VLA處理。VLM與VLA共同用於收集新任務的編排資料集。在模仿階段，使用編排資料對VLA進行微調。最後，在優化階段，我們使用殘差離策略強化學習進一步微調策略。在實驗中，我們對EXIMO的三個階段進行消融實驗，結果顯示其在樣本效率與最終性能上皆顯著優於現有方法。

English

How to efficiently finetune robot policies to learn new tasks on the fly? State of the art robotic manipulation policies are based on behaviour cloning of large vision-language-action (VLA) models with billions of parameters on huge teleoperation datasets. While this simple approach has enabled significant advances for robotic manipulation, finetuning of VLA policies for learning new tasks still remains an open problem. In particular, collecting teleoperation datasets requires hundreds of hours of expensive human labour and the alternative, reinforcement learning (RL), can be notoriously sample-inefficient especially for long-horizon tasks. In addition, RL with VLAs imposes several challenges due to the model's size and architectural design. In this work, we propose EXIMO, an efficient algorithm for finetuning of VLA policies. EXIMO operates in three stages: explore, imitate, and optimize. During the explore phase, EXIMO equips the VLA with a vision language model (VLM) that acts as a planner. The VLM thinks and breaks down challenging long-horizon problems into shorter ones for the VLA. The VLM, together with the VLA, is used to collect an orchestrated dataset on new tasks. During the imitate phase, the VLA is finetuned with the orchestrated data. Finally, during the optimize stage, we use residual off-policy RL to further finetune the policy. In our experiments, we ablate all three stages of EXIMO and show that it outperforms existing approaches significantly in terms of sample-efficiency and final performance.