LongRAG：透過長文本語言模型增強檢索輔助生成

LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs

June 21, 2024

作者: Ziyan Jiang, Xueguang Ma, Wenhu Chen

cs.AI

摘要

在傳統的RAG框架中，基本的檢索單元通常很短。像是DPR這樣的常見檢索器通常與100字的維基百科段落一起工作。這樣的設計迫使檢索器在大型語料庫中尋找「針」單元。相較之下，讀者只需要從短檢索單元中提取答案。這種不平衡的「重型」檢索器和「輕型」讀者設計可能導致次優異的表現。為了減輕這種不平衡，我們提出了一個新框架LongRAG，包括一個「長檢索器」和一個「長讀者」。LongRAG將整個維基百科處理成4K令牌單元，比以前長30倍。通過增加單元大小，我們顯著地將總單元數從22M減少到700K。這顯著降低了檢索器的負擔，從而產生了顯著的檢索分數：NQ上的答案召回率@1=71%（之前為52%），HotpotQA（全文）上的答案召回率@2=72%（之前為47%）。然後，我們將前k個檢索單元（約30K令牌）餵入現有的長內容LLM以執行零-shot答案提取。LongRAG無需任何訓練即實現了NQ上的62.7% EM，這是已知最佳結果。LongRAG還在HotpotQA（全文）上實現了64.3%，與SoTA模型相當。我們的研究為將RAG與長內容LLM相結合的未來路線提供了見解。

English

In traditional RAG framework, the basic retrieval units are normally short. The common retrievers like DPR normally work with 100-word Wikipedia paragraphs. Such a design forces the retriever to search over a large corpus to find the `needle' unit. In contrast, the readers only need to extract answers from the short retrieved units. Such an imbalanced `heavy' retriever and `light' reader design can lead to sub-optimal performance. In order to alleviate the imbalance, we propose a new framework LongRAG, consisting of a `long retriever' and a `long reader'. LongRAG processes the entire Wikipedia into 4K-token units, which is 30x longer than before. By increasing the unit size, we significantly reduce the total units from 22M to 700K. This significantly lowers the burden of retriever, which leads to a remarkable retrieval score: answer recall@1=71% on NQ (previously 52%) and answer recall@2=72% (previously 47%) on HotpotQA (full-wiki). Then we feed the top-k retrieved units (approx 30K tokens) to an existing long-context LLM to perform zero-shot answer extraction. Without requiring any training, LongRAG achieves an EM of 62.7% on NQ, which is the best known result. LongRAG also achieves 64.3% on HotpotQA (full-wiki), which is on par of the SoTA model. Our study offers insights into the future roadmap for combining RAG with long-context LLMs.