機器人操作的價值世界模型

摘要

通用價值模型在從大規模、混合品質數據中擴展機器人策略學習方面扮演著關鍵角色。從數學角度來看，精確的價值估計需要深度的時間理解，要求模型既能利用歷史脈絡來確立當前信念，又能規劃未來結果。然而，現有的機器人價值模型大多基於視覺語言模型（VLM）骨幹，這些模型主要針對靜態或時間稀疏的視覺觀測進行預訓練，缺乏價值估計所需的時序建模能力。與VLM不同，世界模型天生擅長時序建模與未來規劃，因此成為學習可泛化價值函數的理想基礎。基於此洞見，我們將世界模型與價值估計相結合，構建了一種新型的通用機器人價值模型——世界價值模型（WVM），該模型能提供準確的任務進展評估，以判斷數據品質。在標準基準測試中，WVM在價值序相關（VOC）指標上達到了當前最優（SOTA）結果。為補充僅包含專家數據的標準評估套件，我們進一步引入了次優價值基準（Suboptimal-Value-Bench），這是一個包含800條次優軌跡的多具身基準，並具備高保真度的人工標註幀級標記。評估結果顯示，WVM在次優價值基準上仍保持SOTA性能，證明了其在處理專家數據與次優數據時的穩健性。在應用於策略學習時，WVM在模擬環境與真實世界的部署中，均能提升多種策略提取方法在操作任務上的表現，為從混合品質數據中學習提供了穩健的指導。

English

Generalist value models play a pivotal role in scaling robotic policy learning from large-scale, mixed-quality data. Mathematically, accurate value estimation demands deep temporal understanding, requiring models to both ground the current belief using historical context and plan over future outcomes. However, most existing robotic value models are built on Vision-Language Model (VLM) backbones that are pretrained primarily on static or temporally sparse visual observations, lacking the requisite temporal modeling capabilities for value estimation. Unlike VLMs, world models naturally excel at temporal modeling and future planning, making them ideal foundations for learning generalizable value functions. Driven by this insight, we marry world models with value estimation to construct a new generalist robotic value model, World Value Model (WVM), that offers accurate task progressions to assess data quality. On standard benchmarks, WVM delivers state-of-the-art (SOTA) Value-Order Correlation (VOC) results. Complementing standard evaluation suites that contains only expert data, we further introduce Suboptimal-Value-Bench, a multi-embodiment benchmark consisting of 800 suboptimal trajectories with high-fidelity, human-labeled frame annotations. Our evaluations show that WVM maintains its SOTA performance on Suboptimal-Value-Bench, establishing its robustness in handling both expert and suboptimal data. When deployed for policy learning, WVM improves manipulation performance across various policy extraction approaches in both simulated and real-world deployment, providing robust guidance for learning from mixed-quality data.