PRM作為裁判1.5：機器人流程評估工具包

摘要

細粒度的機器人評估對於理解具身模型至關重要，它超越了二元成功率與基於規則的過程評分。我們提出了 PRM-as-a-Judge 1.5，這是一個用於機器人過程評估的工具包，可將執行影片轉化為密集的進度曲線，並推導出多個細粒度指標。PRM-as-a-Judge 1.5 在 1.0 版本的基礎上引入了三個指標，分別表徵失敗側的進展、回撤後的恢復以及成功側的執行品質，幫助用戶理解具身模型的能力。基於基準中的執行影片，我們對具身模型進行了全面評估，提供了一些細粒度指標結果與關鍵發現。我們進一步引入了 RoboPulse++ 來評估過程獎勵模型（PRM）的可靠性，為評估者提供更準確的測試平台。此外，我們發布了一個用戶友好的評估套件，包括基準、指標實現與可視化工具，以支持可重現的操作過程評估。我們呼籲社群重新思考機器人的評估方式，並將透明、程序化且可重現的評估確立為下一代具身智能的基礎。

English

Fine-grained robotic evaluation matters for understanding embodied models, going beyond binary success rates and rule-based process scores. We present PRM-as-a-Judge 1.5, a toolkit for robot process assessment that turns rollout videos into dense progress curves and derives multiple fine metrics. PRM-as-a-Judge 1.5 introduces three metrics, building on version 1.0, that characterize failure-side progress, post-drawdown recovery, and success-side execution quality, helping users understand embodied model capability. Based on the rollout videos from benchmarks, we perform a comprehensive assessment of the embodied models, providing some fine-grained metric results and key findings. We further introduce RoboPulse++ to evaluate the reliability of process reward models (PRM), providing evaluators with a more accurate testing platform. Moreover, we release a user-friendly assessment suite, including the benchmark, metric implementation, and visualization tools, to support reproducible manipulation process evaluation. We call on the community to rethink how robots are evaluated and establish transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence.