SecRespond：针对真实世界妥协后事件响应的AI代理基准测试

摘要

大语言模型（LLM）智能体正越来越多地被应用于实际安全运维场景中，能够访问主机工件和命令行接口（CLI），因此全面评估其安全能力至关重要。然而，现有的网络安全基准测试主要聚焦于攻击发生前的预入侵场景，即智能体处于一个干净且理想化的环境中。这导致后入侵场景的研究尚属空白。为填补这一缺失，我们提出SecRespond——首个用于评估LLM智能体在后入侵事件响应工作流中表现的基准测试。给定一个被入侵主机的取证磁盘快照，以及主机安全产品报告的安全告警、漏洞扫描和基线检查结果，智能体需要生成针对入侵行为、基线风险和漏洞风险的取证报告，并输出修复方案。我们基于10个网络靶场实例化该任务，每个靶场均来自于一个独立的被入侵云主机，涵盖4种入侵入口类型、21种ATT&CK技术和5种操作系统。我们在OpenCode智能体框架上评估了23个前沿LLM。实验结果表明，尽管当前智能体能够可靠地揭示告警暴露的问题，但在主动调查磁盘中的静默入侵行为、生成全面且经过验证的修复方案方面仍存在困难——没有哪个模型能在任何一个网络靶场上实现完全检测与修复。这揭示了构建面向实际事件响应的智能体所面临的根本性瓶颈。该基准测试公开于https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond。

English

Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SecRespond, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available at https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond.