OpenVLA：一個開源的視覺-語言-行動模型

摘要

大型政策預訓練模型，結合互聯網視覺語言數據和多樣化機器人示範，具有改變我們教導機器人新技能的潛力：不再從頭開始訓練新行為，我們可以微調這種視覺-語言-動作（VLA）模型，以獲得強大、可泛化的視覺運動控制政策。然而，廣泛應用VLA於機器人領域一直存在挑戰，原因是1）現有的VLA主要是封閉的，對公眾不可及，以及2）先前的研究未能探索有效微調VLA以應用於新任務的方法，這是應用的關鍵組成部分。為應對這些挑戰，我們介紹了OpenVLA，一個擁有70億參數的開源VLA，訓練於包含970k真實世界機器人示範的多樣化數據集。OpenVLA基於Llama 2語言模型，結合視覺編碼器，融合了來自DINOv2和SigLIP的預訓練特徵。由於增加的數據多樣性和新模型組件，OpenVLA在通用操作方面取得了出色的結果，對於29個任務和多個機器人實體，其絕對任務成功率比RT-2-X（550億）等封閉模型高出16.5％，並且參數數量少7倍。我們進一步展示，我們可以有效地對OpenVLA進行微調以適應新設置，在涉及多個對象和強語言基礎能力的多任務環境中表現出特別強大的泛化結果，並且在結果上表現優於從頭開始的模仿學習方法，如Diffusion Policy，提高了20.4％。我們還探索了計算效率；作為另一項貢獻，我們展示OpenVLA可以通過現代低秩適應方法在消費者GPU上進行微調，並通過量化高效地提供，而不會影響下游成功率。最後，我們釋放了模型檢查點、微調筆記本和我們的PyTorch代碼庫，內置支持在Open X-Embodiment數據集上規模訓練VLA。

English

Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As a product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters. We further show that we can effectively fine-tune OpenVLA for new settings, with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, and outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4%. We also explore compute efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyTorch codebase with built-in support for training VLAs at scale on Open X-Embodiment datasets.

OpenVLA：一個開源的視覺-語言-行動模型

OpenVLA: An Open-Source Vision-Language-Action Model

摘要

Support