OpenART：通过开放式环境演化扩展智能体红队测试

摘要

AI智能体运行于持续性环境中，早期状态变化可能对远期决策产生影响。与传统语言模型交互不同，智能体行为通过共享状态进行中介，该状态在长周期工作流中被反复修改和复用。当前的安全基准测试往往无法捕捉这些累积性风险，因为它们聚焦于短时、静态的任务。为弥补这些局限，我们提出OpenART——一个通过环境演化实现可扩展智能体红队测试的开放式竞技场。OpenART提供覆盖50个领域、超过10,000个经过验证的有状态场景，这些场景源自500,000余个工具和技能的池子。这些任务的中位数工具调用次数为97次，并支持在75种不同的智能体模型配置上进行统一评估。为系统性地探索这些不断演化的攻击面，我们提出演化马尔可夫超图攻击（EMHA）。EMHA是一种黑盒策略，通过协调授权状态转换来执行反馈驱动的环境演化，无需参数更新。在整个评估过程中，任务目标保持不变，仅环境状态发生变化。在所有配置中，EMHA实现了85.0%的汇总攻击成功率（ASR）。相较于仅依赖指令的演化，其优势从简单环境中的约2%扩展到最复杂环境中的超过17%，这表明随着任务复杂度的增长，环境演化日益有效地暴露安全漏洞。此外，我们的分析表明，智能体的具体运行时实现在安全差异中占显著比重，其影响超越了底层模型本身的能力差异。这些结果确立了OpenART作为在复杂演化环境中研究智能体安全的可扩展基础平台。

English

AI agents operate in persistent environments where early state changes can influence decisions far into the future. Unlike conventional language-model interactions, agent behavior is mediated through a shared state that is repeatedly modified and reused across long-horizon workflows. Current safety benchmarks often fail to capture these cumulative risks because they focus on short, static tasks. To address these limitations, we introduce OpenART, an open-ended arena for scalable agent red teaming through environment evolution. OpenART provides over 10,000 validated stateful scenarios across 50 domains, drawing from a pool of more than 500,000 tools and skills. These tasks require a median of 97 tool calls and enable unified evaluation across 75 different agent-model configurations. To systematically explore these evolving attack surfaces, we propose the Evolutionary Markov Hypergraph Attack (EMHA). EMHA is a black-box policy that performs feedback-driven environment evolution by coordinating authorized state transitions without requiring parameter updates. Throughout the evaluation, task objectives remain fixed while only the environment state changes. Across all configurations, EMHA achieves a pooled Attack Success Rate (ASR) of 85.0%. Its advantage over instruction-only evolution increases from approximately 2% on simple environments to over 17% on the most complex ones, demonstrating that environment evolution increasingly exposes safety failures as task complexity grows. Furthermore, our analysis shows that the specific runtime implementation of an agent explains a significant portion of safety variation beyond the underlying model's capabilities. These results establish OpenART as a scalable foundation for studying agent safety in complex, evolving environments.