AgentHijack：计算机使用代理在常见环境干扰下的鲁棒性基准测试

摘要

由多模态大语言模型驱动的自主计算机操作代理正逐渐成为完成复杂数字工作流的高效助手。然而，真实世界的执行环境远非理想：弹窗、分辨率变化以及竞争应用程序频繁干扰代理的感知与控制。我们提出AgentHijack——一个专为评估计算机操作代理在常见干扰下鲁棒性而设计的基准测试，其中动态环境的不确定性会在无直接对抗意图的情况下破坏执行流程。具体而言，AgentHijack引入了9种可配置的常见干扰，以复现现实中的不完美场景。我们评估了多种基于多模态大语言模型的桌面任务代理，发现即便是微小的干扰也可能导致显著的性能下降，这凸显了代理的脆弱性以及进行鲁棒性评估的必要性。随后，我们提出AgentHijack-Agent框架，该框架集成一个具备增强环境定位能力的动作生成器，以及一个负责行为总结与环境检查的观察者。大量实验验证了其有效性。我们的代码、环境、基线模型及数据均可在https://AgentHijack.github.io公开获取。

English

Autonomous computer use agents that powered by multimodal large language models (MLLMs) are emerging as capable assistants for completing complex digital workflows. However, real-world execution environments are far from ideal: pop-ups, resolution changes, and competing applications frequently interfere with agent perception and control. We introduce AgentHijack, a benchmark designed to evaluate the robustness of computer-use agents under common corruptions, where the uncertainties in dynamic environment disrupt the execution flow without direct adversarial intent. Specifically, AgentHijack introduces 9 configurable common corruptions to replicate realistic imperfect scenarios. We evaluate a variety of desktop tasks that utilize MLLM-based agents and discover that even minor instances of corruption can result in substantial performance degradation, which emphasizes the fragility of agents and underscores the necessity of robustness evaluation. Afterward, we propose AgentHijack-Agent, a framework that integrates an action generator with enhanced grounding capabilities and an onlooker responsible for behavior summarization and environment checking. Extensive experiments validate its effectiveness. Our code, environment, baseline models and data are publicly available at: https://AgentHijack.github.io.