OSReward：建立跨平台计算机使用奖励模型的标准化评估

摘要

计算机使用智能体（CUAs）正在数字世界中快速发展。CUA轨迹记录了智能体的动作、状态和推理过程。验证其是否完成了任务指令，对于CUA的评估、数据整理和强化学习至关重要。人工编写的验证器和人工标注者都无法在规模上提供此类验证，因此该领域日益转向使用视觉语言模型（VLMs）作为CUA轨迹的裁判。但一个根本性问题长期以来未被审视：这些VLM裁判是否足够可靠？为了系统地研究这一问题，我们提出了OSReward，一个真实、高质量的基准，用于评估VLM裁判对CUA轨迹的判断。这些轨迹来自多种智能体骨干，在不同平台上执行经过人工验证的指令，随后通过多阶段人工标注严格标定真实结论。在此基础上，我们构建了OSReward-Hard，一个专注于真正困难案例的挑战集，以及OSReward-Multi，用于细粒度的效率和一致性评分。迄今为止最全面的VLM裁判评估发现，即使是最先进的模型也达不到理想裁判的标准，它们普遍存在系统性宽松偏差，将失败的运行误标为成功。少数足够可靠、值得信赖的模型在大规模运行上过于昂贵，而价格实惠的开源模型则远远落后。为了缩小这一差距，我们构建并发布了OS-Shepherd-100K，这是一个面向CUA社区的、带有推理标注的轨迹判断开放语料库。在此基础上，我们训练了OS-Shepherd（9B和35B），这些开源奖励模型提供低成本、稳定且可靠的奖励信号，以比前沿模型低30%-60%的成本达到与商业裁判相当的性能。广泛的分析进一步为大规模可靠CUA奖励的设计提供了启示。我们的代码、基准、数据集和模型检查点可在 https://os-copilot.github.io/OSReward-Home/ 获取。

English

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.