當上下文反噬：透過文檔層級注意力崩潰偵測RAG投毒

摘要

檢索增強生成（Retrieval-augmented generation, RAG）在增強大型語言模型方面不可或缺。然而，RAG 系統日益容易受到投毒攻擊，此類攻擊透過注入對抗性文件來操控生成器的輸出。先前的方法依賴輸出端訊號（如困惑度與一致性檢查）來偵測此類攻擊。然而，我們的分析顯示，蓄意攻擊常會誘發虛假信心，使受污染的輸出表現出比良性輸出更低的困惑度，導致基於不確定性的偵測失效。為應對此挑戰，我們探討生成器的內部動態，並識別出一種稱為「注意力崩潰」（Attention Collapse）的獨特特徵。與良性生成中注意力分散不同，受攻擊生成在注意力集中於受污染文件時會表現出熵的下降。基於這些發現，我們提出 D-SCAN（文件層級訊號崩潰分析，Document-level Signal Collapse Analysis），這是一個輕量級的偵測框架，透過監控注意力動態來識別受攻擊的生成。在多個攻擊基準上的大量實驗證明了我們方法的有效性。此外，即使攻擊未能改變最終答案，D-SCAN 也能偵測到此類攻擊。程式碼可於 https://github.com/yingtaoren/D-Scan.git 取得。

English

Retrieval-augmented generation (RAG) is indispensable for enhancing large language models. However, RAGs are increasingly susceptible to poisoning attacks, in which adversarial documents are injected to manipulate generator outputs. Previous methods rely on output-side signals such as perplexity and consistency checks to detect such attacks. Nevertheless, our analysis reveals that deliberate attacks often induce false confidence, where poisoned outputs exhibit even lower perplexity than benign ones, rendering uncertainty-based detection ineffective. To address this challenge, we explore the internal dynamics of the generator and identify a distinctive signature termed Attention Collapse. Unlike the dispersed attention in benign generations, attacked generations exhibit a decrease in entropy as attention concentrates on poisoned documents. Building on these findings, we propose D-SCAN (Document-level Signal Collapse Analysis), a lightweight detection framework that monitors attention dynamics to identify attacked generations. Extensive experiments on multiple attack benchmarks demonstrate the effectiveness of our method. Moreover, D-SCAN can detect attacks even when they fail to alter the final answer. Code is available at https://github.com/yingtaoren/D-Scan.git.