WCM：用於視覺-語言-動作強化學習的世界評論家模型

摘要

強化學習（RL）對視覺-語言-動作（VLA）模型進行後訓練，已在機器人操作領域展現出強勁的潛力。在各種強化學習方法中，基於評論家（critic）的方法依賴於價值估計器，而該估計器主要處理單幀觀測或單幀VLM骨幹網絡潛在表徵，這與機器人控制本質上的部分可觀測特性存在根本性的不匹配。若以樸素方式將觀測歷史納入評論家網絡，會在高維視覺空間中造成指數級的複雜度，且由於純標量回報回歸所提供的監督訊號不足以學習跨時間動態，此方法仍會失效。我們將根本原因歸結為狀態逼近問題：在缺乏明確世界建模目標的情況下，評論家網絡的表徵無法捕捉準確價值估計所需的時間結構。為解決此問題，我們提出世界評論家模型（World Critic Model, WCM），其建立在輕量級LeJEPA架構之上；WCM聯合預測未來潛在狀態並估計價值，使得評論家網絡的表徵被明確地訓練以捕捉時間動態，而非僅是回歸標量回報。WCM可無縫整合至在策略（on-policy）與離策略（off-policy）訓練流程中，並與最先進的VLA骨幹模型（包括Pi0、Pi0.5及OpenVLA-OFT）相容。涵蓋四個基準、共149項任務的大量實驗表明，WCM在分佈內與分佈外設定下均 consistently 達到最先進效能，尤其在泛化能力上表現出顯著的提升。我們進一步利用OpenVLA-OFT與Pi0.5搭配離策略強化學習，在七項真實世界操作任務上驗證了WCM，確認其能在多樣化場景中穩定部署。

English

Reinforcement learning (RL) post-training of Vision-Language-Action (VLA) models has shown strong promise for robotic manipulation. Among RL methods, critic-based approaches rely on a value estimator that predominantly operates on single-frame observations or single-frame VLM backbone latents, which is a fundamental mismatch with the partially observable nature of robot control. A naive approach to incorporate observation history into the critic incurs exponential complexity with high-dimensional visual space, and still fails because pure scalar-return regression provides insufficient supervision for learning cross-temporal dynamics. We identify the root cause as a state approximation problem: without an explicit world modeling objective, the critic's representation cannot capture the temporal structure needed for accurate value estimation. To address this, we propose the World Critic Model (WCM), built on a lightweight LeJEPA architecture; WCM jointly predicts future latent state and estimates values, such that the critic's representation is explicitly trained to capture temporal dynamics rather than merely regress scalar returns. WCM integrates seamlessly into both on-policy and off-policy training pipelines and is compatible with state-of-the-art VLA backbones including Pi0, Pi0.5, and OpenVLA-OFT. Extensive experiments on 149 tasks across four benchmarks demonstrate that WCM consistently achieves state-of-the-art performance in both in-distribution and out-of-distribution settings, with particularly strong generalization gains. We further validate WCM on seven real-world manipulation tasks using OpenVLA-OFT and Pi0.5 with off-policy RL, confirming stable deployment across diverse settings.