SecOPD：透過在策略蒸餾緩解適應性提示注入

摘要

提示注入被列為 AI 代理的第一大威脅。當代理從網站、檔案或電子郵件存取外部資料時，攻擊者可能將提示注入資料中，例如說：「忽略所有先前的指令，並執行<攻擊者的任務>。」為了防止代理被任意操控，防禦者試圖訓練安全的 LLM；然而，面對自適應提示注入，這些模型仍然承受接近 100% 的攻擊成功率（ASR）。我們注意到，這是因為現有的防禦性微調方法依賴於序列層級的回饋信號（如 DPO 或 GRPO）。將整個輸出同等對待，使模型無法精確學習哪些輸出 token 是不安全的。在本文中，我們提出安全在策略蒸餾（Secure On-Policy Distillation, SecOPD），提供 token 層級的回饋以指導防禦性微調。該 LLM 接收注入樣本並產生 rollout，其 token 由初始化模型在對應的乾淨輸入條件下進行評分。憑藉更細粒度的訓練信號，我們防禦後的 Qwen3.6-27B 在面對 SoTA 的 PISmith 自適應提示注入時達到 9.0% 的 ASR，相較之下，先前 SoTA 的 Meta-SecAlign 為 94.0%。所獲得的安全性可泛化到訓練中完全未見的領域：在代理工具呼叫中，SecOPD 達到 4.7% 的 ASR，而 Meta-SecAlign 則為 5.5%。程式碼與模型可在 https://github.com/pppyb/SecOPD 和 https://huggingface.co/pybbb/Qwen3.6-27B-SecOPD 取得。

English

Prompt injection is listed as the \#1 threat to AI agents. When an agent accesses external data from websites, files, or emails, an attacker may inject a prompt into the data, saying, "Ignore all prior instructions and perform <an attacker's task>." To prevent arbitrary manipulation of agents, defenders try to train secure LLMs, which, however, still suffer from near 100% attack success rates (ASRs) against adaptive prompt injections. We note that this is because existing defensive finetuning recipes rely on sequence-level feedback signals (in DPO or GRPO). Treating an entire output equally prevents the model from learning precisely which output tokens are insecure. In this paper, we propose Secure On-Policy Distillation (SecOPD) that provides token-level feedback to guide defensive fine-tuning. The LLM receives an injected sample and produces a rollout, whose tokens are scored by the initialization model given the corresponding clean input. With more fine-grained training signals, our defended Qwen3.6-27B achieves a 9.0% ASR against the SoTA PISmith adaptive prompt injections, compared to 94.0% for the prior SoTA, Meta-SecAlign. The obtained security generalizes to domains completely unseen in training: in agentic tool calling, SecOPD achieves a 4.7% ASR compared to 5.5% for Meta-SecAlign. Code and the model are available at https://github.com/pppyb/SecOPD and https://huggingface.co/pybbb/Qwen3.6-27B-SecOPD.