从受控到真实环境：面向现实世界的渗透测试代理评估

摘要

AI渗透测试代理作为进攻性安全系统正日益可信，但现有基准测试在指导哪些代理能在真实目标上表现最佳方面仍存在局限。当前评估协议在简化或狭窄场景中，针对预设目标（如夺旗赛、远程代码执行、漏洞复现或轨迹相似性）进行评估与优化。这些工具虽有助于衡量受限能力，但未能充分捕捉真实渗透测试所需的复杂性、开放式探索和战略决策。本文提出一种实用评估协议，将评估重点从任务完成转向已验证的漏洞发现，从而允许在覆盖多攻击面和漏洞类型的足够复杂目标中进行评估。该协议结合结构化真实数据与基于大语言模型的语义匹配来识别漏洞，采用二分图解析在模糊性下对发现结果进行评分，持续维护真实数据，对随机代理进行重复与累积评估，引入效率指标，并通过精简套件选择实现可持续实验。该协议通过提供对AI渗透测试代理更真实、更具操作信息性的比较，扩展了现有技术水平。为确保可重复性，我们还发布了专家标注的真实数据及所提评估协议的代码：https://github.com/ethiack/ethibench。

English

AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perform best in real-world targets. Existing evaluation protocols assess and optimize for predefined goals such as capture-the-flag, remote code execution, exploit reproduction, or trajectory similarity, in simplified or narrow settings. These tools are valuable for measuring bounded capabilities, yet they do not adequately capture the complexity, open-ended exploration, and strategic decision-making required in realistic pentesting. In this paper, we present a practical evaluation protocol that shifts assessment from task completion to validated vulnerability discovery, allowing evaluation in sufficiently complex targets spanning multiple attack surfaces and vulnerability classes. The protocol combines structured ground-truth with LLM-based semantic matching to identify vulnerabilities, bipartite resolution to score findings under realistic ambiguity, continuous ground-truth maintenance, repeated and cumulative evaluation of stochastic agents, efficiency metrics, and reduced-suite selection for sustainable experimentation. This protocol extends the state of the art by enabling a more realistic, operationally informative comparison of AI pentesting agents. To enable reproducibility, we also release expert-annotated ground truth and code for the proposed evaluation protocol: https://github.com/ethiack/ethibench.