WCM：面向视觉-语言-动作强化学习的世界评论模型

摘要

强化学习（RL）后训练视觉-语言-动作（VLA）模型在机器人操作领域展现出强大潜力。在各类RL方法中，基于评论家的方法依赖于价值估计器，而该估计器主要处理单帧观测或单帧VLM骨干网络的潜在表示，这与机器人控制的部分可观测特性存在根本性不匹配。将观测历史纳入评论家的朴素方法在高维视觉空间中会产生指数级复杂度，且仍然无法奏效，因为纯标量回报回归无法为学习跨时间动态提供充分监督。我们指出其根本原因在于状态近似问题：缺乏显式的世界建模目标时，评论家的表示无法捕捉准确价值估计所需的时间结构。为此，我们提出世界评论家模型（WCM），其构建于轻量级LeJEPA架构之上；WCM联合预测未来潜在状态并估计价值，从而使评论家的表示被显式训练以捕捉时间动态，而非仅仅回归标量回报。WCM可无缝集成到在线策略和离线策略训练流程中，并与包括Pi0、Pi0.5和OpenVLA-OFT在内的最先进VLA骨干网络兼容。在四个基准的149项任务上进行的大量实验表明，WCM在分布内和分布外场景中均持续取得最先进性能，尤其在泛化能力方面提升显著。我们进一步使用OpenVLA-OFT和Pi0.5结合离线策略RL，在七项真实世界操作任务上验证了WCM，证实了其在多样化场景中的稳定部署能力。

English

Reinforcement learning (RL) post-training of Vision-Language-Action (VLA) models has shown strong promise for robotic manipulation. Among RL methods, critic-based approaches rely on a value estimator that predominantly operates on single-frame observations or single-frame VLM backbone latents, which is a fundamental mismatch with the partially observable nature of robot control. A naive approach to incorporate observation history into the critic incurs exponential complexity with high-dimensional visual space, and still fails because pure scalar-return regression provides insufficient supervision for learning cross-temporal dynamics. We identify the root cause as a state approximation problem: without an explicit world modeling objective, the critic's representation cannot capture the temporal structure needed for accurate value estimation. To address this, we propose the World Critic Model (WCM), built on a lightweight LeJEPA architecture; WCM jointly predicts future latent state and estimates values, such that the critic's representation is explicitly trained to capture temporal dynamics rather than merely regress scalar returns. WCM integrates seamlessly into both on-policy and off-policy training pipelines and is compatible with state-of-the-art VLA backbones including Pi0, Pi0.5, and OpenVLA-OFT. Extensive experiments on 149 tasks across four benchmarks demonstrate that WCM consistently achieves state-of-the-art performance in both in-distribution and out-of-distribution settings, with particularly strong generalization gains. We further validate WCM on seven real-world manipulation tasks using OpenVLA-OFT and Pi0.5 with off-policy RL, confirming stable deployment across diverse settings.