OSReward：為跨平台電腦使用獎勵模型建立標準化評估

摘要

使用電腦的代理程式（CUA）正於數位世界中快速發展。CUA 軌跡記錄了代理程式的動作、狀態與推理過程。驗證其是否完成任務指令，是 CUA 評估、資料整理與強化學習的核心環節。無論是人類撰寫的驗證器或人類標注者，皆無法在此規模下提供此類驗證，因此該領域日益轉向以視覺語言模型（VLM）作為 CUA 軌跡的評判者。然而，一個根本性問題長久以來未獲審視：這些 VLM 評判者是否足夠可靠？為系統性探討此問題，我們提出 OSReward，一個真實且高品質的基準測試，用以評量 VLM 評判者在 CUA 軌跡上的表現。這些軌跡來自多元代理骨架，在跨平台環境中執行經人類驗證的指令，並透過多階段人工標注嚴謹地賦予真實標籤判定。在此基礎上，我們衍生出 OSReward-Hard，一個聚焦於真實困難案例的挑戰集，以及 OSReward-Multi，用於細粒度效率與對齊評分。迄今最全面的 VLM 評判者評估顯示，即使是最先進的模型仍未能達到理想評判者之標準，且普遍存在系統性的寬容偏差，將失敗的執行誤判為成功。少數足夠可靠的模型其執行成本過高，難以大規模運行，而價格可負擔的開源模型則遠遠落後。為縮短此差距，我們建構並釋出 OS-Shepherd-100K，一個為 CUA 社群提供具推理註解的軌跡判定開放語料庫。在此基礎上，我們訓練出 OS-Shepherd（9B 與 35B），這些開放獎勵模型能以低成本、穩定且可靠的方式提供獎勵訊號，以較前緣模型低 30–60% 的成本達到與商業評判者相當的表現。廣泛的分析進一步為大規模可靠 CUA 獎勵的設計提供啟發。我們的程式碼、基準測試、資料集與模型檢查點皆可於 https://os-copilot.github.io/OSReward-Home/ 取得。

English

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.