HarnessRisk: エージェントハーネス安全性のためのライフサイクル指向ベンチマーク

要旨

大语言模型正日益通过管理工具、扩展、持久状态、权限和外部操作的代理框架进行部署。现有的安全基准主要针对个别攻击机制或有限的操作环境子集，这使得难以比较不同框架职责下安全故障的出现方式。我们提出了HarnessRisk，一个面向生命周期的基准，将代理框架安全组织为六个操作阶段，包括框架配置、能力扩展、运行时操作、状态持久化、动作控制和事件恢复。HarnessRisk包含128个沙箱案例，每个案例将一个良性用户目标与嵌入在不可信工作流工件中的对抗性指令配对。我们使用效用、攻击成功率、持久性和检测来评估每个轨迹。在三个框架、六个语言模型和14种模型与框架配置中，攻击成功率从12.6%到80.9%不等，而效用保持在75.0%到97.6%之间。框架配置是所有三个框架中最脆弱的阶段，表明攻击可以通过在原本被授权的工作流中改变安全敏感参数来成功。我们还发现，明确的风险识别并不能可靠地导致安全行动，因为某些配置在超过90%的运行中检测到风险，同时仍保持相当高的攻击成功率。这些结果突显了需要在多个框架职责层面以及已部署模型和框架配置的层面上评估代理安全性的必要性。

English

Large language models are increasingly deployed through agent harnesses that manage tools, extensions, persistent state, permissions, and external actions. Existing safety benchmarks mainly target individual attack mechanisms or a limited subset of operational settings, making it difficult to compare how safety failures emerge across different harness responsibilities. We present HarnessRisk, a lifecycle oriented benchmark that organizes agent harness safety into six operational phases including Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery. HarnessRisk contains 128 sandboxed cases, each pairing a benign user objective with an adversarial instruction embedded in an untrusted workflow artifact. We evaluate each trajectory using Utility, Attack Success Rate, Persistence, and Detection. Across three harnesses, six language models, and 14 model and harness configurations, attack success ranges from 12.6% to 80.9%, while Utility remains between 75.0% and 97.6%. Harness Configuration is the most vulnerable phase across all three harnesses, showing that attacks can succeed by altering security sensitive parameters within otherwise authorized workflows. We also find that explicit risk recognition does not reliably lead to safe action, as some configurations detect risks in more than 90% of runs while retaining substantial attack success. These results highlight the need to evaluate agent safety across multiple harness responsibilities and at the level of the deployed model and harness configuration.