了解LLM的需求：雙重偏好對齊用於檢索增強生成

Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation

June 26, 2024

作者: Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Zhicheng Dou, Ji-Rong Wen

cs.AI

摘要

檢索增強生成（RAG）已證明在緩解大型語言模型（LLMs）的幻覺問題方面具有效果。然而，將檢索器與多樣的LLMs知識偏好對齊的困難不可避免地在開發可靠的RAG系統時帶來挑戰。為了應對這個問題，我們提出了DPA-RAG，這是一個旨在對齊RAG系統內多樣知識偏好的通用框架。具體而言，我們首先引入了一個偏好知識構建流程，並結合了五種新的查詢擴充策略來緩解偏好數據稀缺的問題。基於偏好數據，DPA-RAG實現了外部和內部偏好對齊：1）它將成對、點對和對比偏好對齊能力共同整合到重新排序器中，實現了RAG組件之間的外部偏好對齊。2）它進一步引入了在普通監督微調（SFT）之前的預對齊階段，使LLMs能夠隱式捕捉與其推理偏好對齊的知識，實現了LLMs的內部對齊。在四個知識密集型QA數據集上的實驗結果表明，DPA-RAG優於所有基準線，並無縫集成了黑盒和開源LLMs讀者。進一步的定性分析和討論還提供了實證指導，以實現可靠的RAG系統。我們的代碼公開在https://github.com/dongguanting/DPA-RAG。

English

Retrieval-augmented generation (RAG) has demonstrated effectiveness in mitigating the hallucination problem of large language models (LLMs). However, the difficulty of aligning the retriever with the diverse LLMs' knowledge preferences inevitably poses an inevitable challenge in developing a reliable RAG system. To address this issue, we propose DPA-RAG, a universal framework designed to align diverse knowledge preferences within RAG systems. Specifically, we initially introduce a preference knowledge construction pipline and incorporate five novel query augmentation strategies to alleviate preference data scarcity. Based on preference data, DPA-RAG accomplishes both external and internal preference alignment: 1) It jointly integrate pair-wise, point-wise, and contrastive preference alignment abilities into the reranker, achieving external preference alignment among RAG components. 2) It further introduces a pre-aligned stage before vanilla Supervised Fine-tuning (SFT), enabling LLMs to implicitly capture knowledge aligned with their reasoning preferences, achieving LLMs' internal alignment. Experimental results across four knowledge-intensive QA datasets demonstrate that DPA-RAG outperforms all baselines and seamlessly integrates both black-box and open-sourced LLM readers. Further qualitative analysis and discussions also provide empirical guidance for achieving reliable RAG systems. Our code is publicly available at https://github.com/dongguanting/DPA-RAG.