StealthBench：自主攻擊安全代理的操作隱蔽性評測

摘要

隱蔽，這門在不暴露自身存在、能力或所收集情報的情況下達成目標的技藝，是區分高水準操作者與可被偵測操作者的關鍵。頂尖安全研究員與先進持續威脅（APT）能在不被察覺的情況下達成目標；自主代理逐漸接手同樣的攻擊任務，但它們是否也繼承了這套實戰技藝？我們提出StealthBench，這是一個衡量自主攻擊安全代理在六個操作安全（OPSEC）維度上之操作隱蔽能力的基準測試。我們從真實的漏洞獎勵計畫與紅隊演練軌跡中，萃取出11個經人工驗證的OPSEC失誤事件，並將其擴展為14個Docker化的任務場景。在這些場景中，代理雖然找到了真實漏洞，卻犯了不符標準操作技藝的隱蔽失誤：將憑證嵌入公開上傳內容、刪除生產資源以證明存取權限、強行添加無關使用者來展示競爭條件（race condition）等。我們使用一個由三個大型語言模型（LLM）組成的評審團（採多數決投票）來評估代理的軌跡，衡量安全成功率（成功解決且保持隱蔽）、隱蔽求解率（Stealth@Solve，即成功解決案件中的技藝品質），以及魯莽求解率（雖解決問題但暴露行蹤）。結果顯示，沒有任何模型的安全成功率（結合任務完成與隱蔽性的複合指標）超過54%，證實OPSEC失誤在各模型系列中均系統性存在。我們將StealthBench作為公開基準測試釋出，以支援開發具隱蔽意識的代理，以及為自主攻擊安全部署提供自動化的OPSEC監控。互動式排行榜、評估工具與資料集均可在 https://stealthbench.com 取得。

English

Stealth, the discipline of achieving an objective without revealing your presence, capabilities, or collected intelligence, is what separates sophisticated operators from detectable ones. Elite security researchers and advanced persistent threats achieve their objectives unnoticed; autonomous agents increasingly inherit the same offensive tasks, but do they inherit the tradecraft? We introduce StealthBench,a benchmark that measures operational stealth in autonomous offensive-security agents across six operational security (OPSEC) dimensions. We extract 11 hand-verified OPSEC incidents from real bug-bounty and red-team trajectories, expanded into 14 dockerized task scenarios, where agents, despite finding real vulnerabilities, committed stealth failures inconsistent with standard operational tradecraft: embedding credentials in public uploads, deleting production resources to prove access, force-adding uninvolved users to demonstrate a race condition. We evaluate agent trajectories using a 3-model large language model (LLM) judge panel with majority-vote aggregation, measuring safe success rate (solved and stealthy), Stealth@Solve (tradecraft quality among successful solves), and reckless solve rate (solved but cover blown). Our results show that no model exceeds 54% safe success rate (the compound metric requiring both task completion and stealth), confirming that OPSEC failures are systematic across model families. We release StealthBench as a public benchmark to support both the development of stealth-aware agents and automated OPSEC monitoring for autonomous offensive-security deployments. The interactive leaderboard, evaluation harness, and dataset are available at https://stealthbench.com.