智能体安全应当是运行时契约

摘要

主流范式将AI安全视为在模型训练阶段通过RLHF、DPO或宪法AI注入模型的一种属性。我们认为，对于能够执行代码、修改文件、发送消息和变更数据库的自主智能体而言，这种范式在结构上是不充分的。智能体安全应当是由执行框架（harness）强制实施的运行时契约，该契约包含两个互补的面向。预防性面向通过沙箱、权限门控、输出过滤器和轨迹监控器，在危险行为发生之前予以阻断；证据性面向则要求可验证的证据证明良好行为确实发生，并以测试运行、日志捕获、文件差异和引用溯源等硬性证据作为任务提交的门控条件。我们以四类公开证据支撑这一立场，行级协议和数据见补充JSON文件：对52起有记录的AI智能体与LLM安全事故的调查、一项包含31个无争议核心案例及1个有争议示例性案例的虚假完成审计、对12个公开智能体系统和执行框架的轨迹模式审计，以及对NeurIPS、ICML和ICLR 2023-2025年录用的全部28,560篇论文的标题级审计（显示训练时与部署时发表数量之间存在8至12倍的合并失衡）。两个先前的社区——计算机安全和实验科学——在需要强制安全时，都收敛到了兼具预防性和证据性要素的运行时契约；智能体AI如今正面临同样的压力。我们形式化了智能体轨迹模式和证据链，基于标准监控器组合提出了一个组合式门控命题，并概述了一项研究议程。智能体AI中正确的安全单元是“带可核查证据的轨迹”，而非模型本身。

English

The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages, and modify databases. Agent safety should be a runtime contract enforced by the harness, and the contract has two complementary faces. The preventive face blocks dangerous actions before they happen via sandboxes, permission gates, output filters, and trajectory monitors. The evidential face requires verifiable proof that good actions actually happened, gating task submission on hard evidence such as test runs, log captures, file diffs, and citation grounding. We ground the position in four lines of public evidence, with row-level protocols and data released in the supplementary JSON files: a survey of 52 documented AI-agent and LLM safety incidents, a false-completion audit with 31 non-contested core cases plus one disputed illustrative case, a trajectory-schema audit of 12 public agent systems and harnesses, and a title-level audit of all 28,560 papers accepted at NeurIPS, ICML, and ICLR 2023-2025 showing a pooled 8-12x imbalance between training-time and deployment-time publication. Two prior communities that needed to enforce safety, computer security and the experimental sciences, converged on runtime contracts with both preventive and evidential elements; agentic AI is now under the same pressure. We formalize an Agent Trajectory Schema and Evidence Chain, state a compositional gating proposition based on standard monitor composition, and outline a research agenda. The right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model.