SecRespond：針對真實世界入侵後事件應變的人工智慧代理基準測試

摘要

大语言模型（LLM）代理正越来越多地被应用于真实安全运维场景中，能够访问主机工件和命令行界面（CLI），因此对其安全能力进行全面评估至关重要。然而，现有的网络安全基准测试主要聚焦于攻击发生前的"预入侵"设定——代理在攻击开始前被置于一个干净、理想化的环境中。这导致"被入侵后"的设定未被充分探索。为填补这一空白，我们提出了SecRespond，这是首个用于评估LLM代理在"被入侵后"事件响应工作流中表现的基准测试。代理需要基于受损主机的取证磁盘快照，以及主机安全产品报告的告警、漏洞扫描和基线检查结果，生成入侵取证报告、基线风险评估报告、漏洞风险评估报告，并制定修复方案。我们基于10个网络靶场实例化该任务，每个靶场由一台不同的受损云主机构建，涵盖4种入侵入口类型、21种ATT&CK技术及5种操作系统。我们在OpenCode代理框架上评估了23个前沿LLM。实验结果表明，尽管当前代理能够可靠地揭示告警暴露的问题，但在主动调查磁盘中静默入侵行为、生成全面且经过验证的修复方案方面仍存在困难——没有任何模型能在单个靶场上实现完全的检测与修复。这揭示了构建用于真实世界事件响应代理的根本瓶颈。该基准测试已在 https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond 公开提供。

English

Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SecRespond, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available at https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond.