從受控到野外：針對真實世界的滲透測試代理評估

摘要

AI滲透測試代理作為進攻性安全系統的可信度日益提高，但現有基準測試在哪些代理能在真實目標中表現最佳方面，仍提供有限的指引。現行評估協議在簡化或狹隘的環境中，針對預先定義的目標（如奪旗賽、遠端程式碼執行、漏洞重現或軌跡相似性）進行評估與優化。這些工具對衡量有限範疇的能力極具價值，卻未能充分捕捉真實滲透測試所需的複雜性、開放式探索與策略性決策。本文提出一套實務評估協議，將評估重點從任務完成轉向經驗證的漏洞發現，使評估能在涵蓋多個攻擊面與漏洞類別的足夠複雜目標中進行。該協議結合結構化真實數據與基於大型語言模型的語義匹配來識別漏洞，採用二分圖解析在現實模糊性下對發現結果進行評分，持續維護真實數據，對隨機代理進行重複與累積評估，納入效率指標，並以精簡測試套件選擇達成可持續實驗。此協議拓展了現有技術，實現對AI滲透測試代理更具現實性與操作資訊性的比較。為確保可重複性，我們亦發布專家註釋的真實數據與所提評估協議的程式碼：https://github.com/ethiack/ethibench。

English

AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perform best in real-world targets. Existing evaluation protocols assess and optimize for predefined goals such as capture-the-flag, remote code execution, exploit reproduction, or trajectory similarity, in simplified or narrow settings. These tools are valuable for measuring bounded capabilities, yet they do not adequately capture the complexity, open-ended exploration, and strategic decision-making required in realistic pentesting. In this paper, we present a practical evaluation protocol that shifts assessment from task completion to validated vulnerability discovery, allowing evaluation in sufficiently complex targets spanning multiple attack surfaces and vulnerability classes. The protocol combines structured ground-truth with LLM-based semantic matching to identify vulnerabilities, bipartite resolution to score findings under realistic ambiguity, continuous ground-truth maintenance, repeated and cumulative evaluation of stochastic agents, efficiency metrics, and reduced-suite selection for sustainable experimentation. This protocol extends the state of the art by enabling a more realistic, operationally informative comparison of AI pentesting agents. To enable reproducibility, we also release expert-annotated ground truth and code for the proposed evaluation protocol: https://github.com/ethiack/ethibench.