PRM-as-a-Judge 1.5：机器人过程评估工具包

摘要

细粒度的机器人评估对于理解具身模型至关重要，它超越了二元成功率和基于规则的过程评分。我们提出了PRM-as-a-Judge 1.5，一个用于机器人过程评估的工具包，它将rollout视频转化为密集的进度曲线，并导出多种精细指标。PRM-as-a-Judge 1.5在1.0版本的基础上引入了三个指标，分别刻画失败侧进度、回撤后恢复和成功侧执行质量，帮助用户理解具身模型的能力。基于基准测试中的rollout视频，我们对具身模型进行了全面评估，提供了一些细粒度指标结果和关键发现。我们还引入了RoboPulse++来评估过程奖励模型（PRM）的可靠性，为评估者提供更准确的测试平台。此外，我们发布了一个用户友好的评估套件，包括基准、指标实现和可视化工具，以支持可复现的操作过程评估。我们呼吁社区重新思考机器人的评估方式，并将透明、程序化且可复现的评估确立为下一代具身智能的基础。

English

Fine-grained robotic evaluation matters for understanding embodied models, going beyond binary success rates and rule-based process scores. We present PRM-as-a-Judge 1.5, a toolkit for robot process assessment that turns rollout videos into dense progress curves and derives multiple fine metrics. PRM-as-a-Judge 1.5 introduces three metrics, building on version 1.0, that characterize failure-side progress, post-drawdown recovery, and success-side execution quality, helping users understand embodied model capability. Based on the rollout videos from benchmarks, we perform a comprehensive assessment of the embodied models, providing some fine-grained metric results and key findings. We further introduce RoboPulse++ to evaluate the reliability of process reward models (PRM), providing evaluators with a more accurate testing platform. Moreover, we release a user-friendly assessment suite, including the benchmark, metric implementation, and visualization tools, to support reproducible manipulation process evaluation. We call on the community to rethink how robots are evaluated and establish transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence.