GUI vs. CLI：純螢幕與技能中介電腦使用代理的執行瓶頸

摘要

電腦使用代理可透過圖形介面或程式化指令介面執行軟體任務，但現有的評估方法常將互動模式與任務差異、初始狀態、驗證器及允許的操作混為一談。我們提出一個經匹配的執行層基準測試，涵蓋18個應用程式與12類工作流程中的440項桌面任務。在此基準中，僅限螢幕操作的圖形使用者介面（GUI）代理與以技能為基礎的命令列介面（CLI）代理，在接收相同目標、狀態及最終狀態驗證器的條件下，僅能使用其原生互動模式允許的操作。在此控制環境下，最強的GUI代理達成59.1%的完整通過率，優於最強原始技能CLI代理的48.2%；然而，透過驗證器引導的技能強化，CLI的成功率提升至69.3%，顯示CLI的劣勢大多來自技能覆蓋範圍不足，而非單純的模型能力問題。這些結果表明，GUI與CLI暴露了不同的執行瓶頸：GUI代理受限於長期工作流程中可靠的具身互動，而CLI代理則受限於其技能介面的覆蓋率與可擴展性。

English

Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goals, states, and final-state verifiers while being restricted to modality-native actions. In this controlled setting, the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%; however, verifier-guided skill augmentation raises CLI success to 69.3%, showing that much of the CLI deficit comes from incomplete skill coverage rather than model capability alone. These results suggest that GUI and CLI expose different execution bottlenecks: GUI agents are limited by reliable grounded interaction over long-horizon workflows, whereas CLI agents are limited by the coverage and scalability of their skill interfaces.