OSWorld2.0：在長期真實世界任務中對電腦使用代理進行基準測試

摘要

現有的電腦使用基準測試未能捕捉真實世界電腦使用的現實性、複雜性及長期需求，從而限制了其揭示前沿代理模型局限性的能力。我們推出 OSWorld 2.0，這是一個包含 108 項長期電腦使用工作流程的基準測試，涵蓋日常與專業任務，旨在捕捉真實世界中複雜且具挑戰性的現象。每項任務皆代表一個實際的端到端工作流程，人類使用者完成其中位時間約需 1.6 小時，而 Claude Opus 4.7 在最大思考模式下平均需執行 318 次工具調用——相較之下，OSWorld 1.0 僅需約 30 次。OSWorld 2.0 聚焦於真實工作流程中常見、但在先前基準測試中未充分呈現的挑戰性現象，涵蓋互動設計層面的挑戰（如串流互動與動態環境），以及代理模式層面的挑戰（如跨來源推理、隱含狀態推斷與視覺空間精準度）。任務植基於真實的輸入工件，並與具狀態的真實使用者設定檔資料交叉參照，且包含獨立的安全報告以審核安全敏感性執行。在我們主要採用 500 步二元完成指標下，Claude Opus 4.8 搭配最大思考與批次工具調用表現最佳，但仍僅完成 20.6% 的任務，部分得分為 54.8%；GPT-5.5 的 token 使用效率遠高，但表現停滯於約 13%。這些結果顯示，當前代理模型距離專業級電腦使用仍有很大差距：它們並非在基本 GUI 操作或程式碼編寫上失誤，而是會忽略約束條件、錯過任務執行中出現的資訊、寧可猜測也不詢問使用者，以及跳過驗證步驟；當任務高度依賴必須自行恢復的隱藏狀態時，它們的表現尤其艱難。

English

Existing computer-use benchmarks fail to capture the realism, complexity, and long-horizon demands of real-world computer use, limiting their ability to reveal the limitations of frontier agents. We introduce OSWorld 2.0, a benchmark of 108 long-horizon computer-use workflows across everyday and professional tasks, designed to capture complex and challenging real-world phenomena. Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete and requires an average of 318 tool calls with Claude Opus 4.7 using maximum thinking, compared with about 30 in OSWorld 1.0. OSWorld 2.0 targets challenge phenomena that are common in real workflows yet underrepresented in prior benchmarks, spanning interaction-design challenges such as streaming interaction and dynamic environments, as well as agent-pattern challenges such as cross-source reasoning, implicit-state inference, and visual-spatial precision. Tasks are grounded in authentic input artifacts and cross-referenced against realistic stateful user profile data, and include separate safety reports auditing safety-sensitive execution. Under our primary binary-completion metric at 500 steps, Claude Opus 4.8 with maximum thinking and batched tool calls scores best but still completes only 20.6% of tasks at a 54.8% partial score; GPT-5.5 is far more token-efficient yet plateaus near 13%. These results show that current agents are still far from professional-level computer use: rather than stumbling on basic GUI control or coding, they lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification, struggling most when a task hinges on hidden state they must recover.