基于模型内部的答案归因用于可信的检索增强生成

Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation

June 19, 2024

作者: Jirui Qi, Gabriele Sarti, Raquel Fernández, Arianna Bisazza

cs.AI

摘要

确保模型答案的可验证性是检索增强生成（RAG）在问答（QA）领域面临的基本挑战。最近，提出了自引用提示，旨在使大型语言模型（LLMs）生成对支持文档的引用以及其答案。然而，自引用的LLMs经常难以匹配所需格式，参考不存在的来源，并未能忠实地反映LLMs在生成过程中的上下文使用。在这项工作中，我们提出了MIRAGE -- 基于模型内部的RAG解释 -- 一种使用模型内部进行忠实答案归因的即插即用方法。MIRAGE通过显著性方法检测上下文敏感的答案标记，并将其与通过检索的文档配对，这些文档有助于其预测。我们在一个多语言抽取式QA数据集上评估了我们提出的方法，发现与人类答案归因高度一致。在开放式QA上，MIRAGE实现了与自引用相当的引文质量和效率，同时还允许对归因参数进行更精细的控制。我们的定性评估突出了MIRAGE归因的忠实性，并强调了将模型内部应用于RAG答案归因的前景。

English

Ensuring the verifiability of model answers is a fundamental challenge for retrieval-augmented generation (RAG) in the question answering (QA) domain. Recently, self-citation prompting was proposed to make large language models (LLMs) generate citations to supporting documents along with their answers. However, self-citing LLMs often struggle to match the required format, refer to non-existent sources, and fail to faithfully reflect LLMs' context usage throughout the generation. In this work, we present MIRAGE --Model Internals-based RAG Explanations -- a plug-and-play approach using model internals for faithful answer attribution in RAG applications. MIRAGE detects context-sensitive answer tokens and pairs them with retrieved documents contributing to their prediction via saliency methods. We evaluate our proposed approach on a multilingual extractive QA dataset, finding high agreement with human answer attribution. On open-ended QA, MIRAGE achieves citation quality and efficiency comparable to self-citation while also allowing for a finer-grained control of attribution parameters. Our qualitative evaluation highlights the faithfulness of MIRAGE's attributions and underscores the promising application of model internals for RAG answer attribution.