TUA-Bench：通用終端使用代理基準

摘要

隨著大型語言模型與整合框架持續進步，操作於終端機的代理（agent）不僅能執行程式碼相關任務，還能勝任更廣泛的一般電腦操作工作。然而，現有基準測試未能充分評估通用終端機使用代理（terminal-use agent, TUA）的能力：一般電腦操作基準測試主要針對圖形使用者介面（GUI），而終端機為主的基準測試則側重於傳統上原生於命令列的技術與程式設計導向工作流程。我們提出 TUA-Bench，一個適用於終端機代理的通用基準測試。TUA-Bench 包含涵蓋五大任務類別的 120 項實際任務，涵蓋例行數位活動——包括文件編輯、電子郵件管理及即時網頁資訊搜尋——以及與具博士級領域專家共同設計、需使用專業軟體的科學與工程工作流程。此廣度使 TUA-Bench 有別於先前以命令列或特定領域為主的基準測試。每項任務均為手動設計，在實際終端機中以確定性設定腳本執行，並透過基於執行結果的評分協議進行評估。我們發現，最強大的前沿代理——搭配 Claude Opus 4.8 最大推理努力的 Claude Code——整體表現達 65.8%，且在兩個軌道間均存在顯著差距。藉由提供廣泛且貼近實際的終端機使用能力評估，TUA-Bench 旨在加速從狹隘、特定任務的輔助工具，邁向能在多樣數位環境中穩定運作的通用代理之轉變。

English

As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities-including document editing, email management, and live-web information seeking-as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.