RoboAlign：面向视觉-语言-动作模型的测试时推理学习以实现语言-动作对齐

摘要

提升多模态大语言模型（MLLMs）的具身推理能力，对于在其基础上构建视觉-语言-动作模型（VLAs）以实现多模态理解向低层级动作的即时转化至关重要。近期研究尝试通过视觉问答类监督来增强MLLMs的具身推理能力，但据报告这些方法会导致VLA性能不稳定，往往仅产生微弱增益甚至出现性能倒退。本文提出一种更系统化的MLLM训练框架RoboAlign，可稳定提升VLA性能。我们的核心思路是通过零样本自然语言推理采样动作令牌，并利用强化学习（RL）优化推理过程以提高动作准确性。RoboAlign由此弥合了MLLMs中语言与低层级动作的模态鸿沟，促进了从MLLM到VLA的知识迁移。为验证RoboAlign的有效性，我们在MLLM骨干网络上添加基于扩散模型的动作头来训练VLAs，并在主流机器人基准上进行评估。值得注意的是，仅使用不足1%的数据进行监督微调（SFT）后实施基于RL的对齐，RoboAlign在LIBERO、CALVIN和真实环境中的性能较SFT基线分别提升17.5%、18.9%和106.6%。

English

Improving embodied reasoning in multimodal-large-language models (MLLMs) is essential for building vision-language-action models (VLAs) on top of them to readily translate multimodal understanding into low-level actions. Accordingly, recent work has explored enhancing embodied reasoning in MLLMs through supervision of vision-question-answering type. However, these approaches have been reported to result in unstable VLA performance, often yielding only marginal or even negative gains. In this paper, we propose a more systematic MLLM training framework RoboAlign that reliably improves VLA performance. Our key idea is to sample action tokens via zero-shot natural language reasoning and refines this reasoning using reinforcement learning (RL) to improve action accuracy. As a result, RoboAlign bridges the modality gap between language and low-level actions in MLLMs, and facilitate knowledge transfer from MLLM to VLA. To validate the effectiveness of RoboAlign, we train VLAs by adding a diffusion-based action head on top of an MLLM backbone and evaluate them on major robotics benchmarks. Remarkably, by performing RL-based alignment after SFT using less than 1\% of the data, RoboAlign achieves performance improvements of 17.5\%, 18.9\%, and 106.6\% over SFT baselines on LIBERO, CALVIN, and real-world environments, respectively.

RoboAlign：面向视觉-语言-动作模型的测试时推理学习以实现语言-动作对齐

RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models

摘要

Support