AgentHijack：電腦使用智能體對常見環境干擾之穩健性基準測試

摘要

基於多模態大型語言模型（MLLMs）驅動的自主電腦操作代理，正逐步成為完成複雜數位工作流程的有效輔助工具。然而，真實的執行環境遠非理想：彈出視窗、解析度變更以及競爭應用程式經常干擾代理的感知與控制能力。我們提出 AgentHijack——一個專為評估電腦操作代理在常見干擾下魯棒性而設計的基準測試，此類干擾源自動態環境中的不確定性，雖非直接惡意意圖，卻會中斷執行流程。具體而言，AgentHijack 引入了 9 種可配置的常見干擾，以模擬現實中的不完美場景。我們評估了多項運用基於 MLLM 代理的桌面任務，發現即便是微小的干擾也可能導致顯著的性能衰退，這凸顯了代理的脆弱性以及魯棒性評估的必要性。隨後，我們提出了 AgentHijack-Agent 框架，該框架整合了一個具備增強調校能力的動作生成器，以及一個負責行為摘要與環境檢查的旁觀者。大量實驗驗證了其有效性。我們的程式碼、環境、基準模型與數據已公開於：https://AgentHijack.github.io。

English

Autonomous computer use agents that powered by multimodal large language models (MLLMs) are emerging as capable assistants for completing complex digital workflows. However, real-world execution environments are far from ideal: pop-ups, resolution changes, and competing applications frequently interfere with agent perception and control. We introduce AgentHijack, a benchmark designed to evaluate the robustness of computer-use agents under common corruptions, where the uncertainties in dynamic environment disrupt the execution flow without direct adversarial intent. Specifically, AgentHijack introduces 9 configurable common corruptions to replicate realistic imperfect scenarios. We evaluate a variety of desktop tasks that utilize MLLM-based agents and discover that even minor instances of corruption can result in substantial performance degradation, which emphasizes the fragility of agents and underscores the necessity of robustness evaluation. Afterward, we propose AgentHijack-Agent, a framework that integrates an action generator with enhanced grounding capabilities and an onlooker responsible for behavior summarization and environment checking. Extensive experiments validate its effectiveness. Our code, environment, baseline models and data are publicly available at: https://AgentHijack.github.io.