LongRAG：利用长上下文LLMs增强检索增强生成

LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs

June 21, 2024

作者: Ziyan Jiang, Xueguang Ma, Wenhu Chen

cs.AI

摘要

在传统的RAG框架中，基本的检索单元通常很短。常见的检索器如DPR通常使用包含100个词的维基百科段落。这样的设计迫使检索器在大语料库中搜索以找到“针”单元。相比之下，读者只需要从短检索单元中提取答案。这种不平衡的“重型”检索器和“轻型”读者设计可能导致次优性能。为了缓解这种不平衡，我们提出了一个新框架LongRAG，由“长检索器”和“长读者”组成。LongRAG将整个维基百科处理成包含4K个标记的单元，比以前长30倍。通过增加单元大小，我们将总单元数量从22M减少到700K。这显著降低了检索器的负担，从而导致显著的检索分数提升：在NQ上，答案召回率@1=71%（之前为52%），在HotpotQA（全维基）上，答案召回率@2=72%（之前为47%）。然后我们将前k个检索到的单元（约30K个标记）馈送给现有的长上下文LLM来进行零样本答案提取。在不需要任何训练的情况下，LongRAG在NQ上实现了62.7%的EM，这是已知的最佳结果。LongRAG还在HotpotQA（全维基）上实现了64.3%，与SoTA模型持平。我们的研究为将RAG与长上下文LLMs相结合的未来路线提供了见解。

English

In traditional RAG framework, the basic retrieval units are normally short. The common retrievers like DPR normally work with 100-word Wikipedia paragraphs. Such a design forces the retriever to search over a large corpus to find the `needle' unit. In contrast, the readers only need to extract answers from the short retrieved units. Such an imbalanced `heavy' retriever and `light' reader design can lead to sub-optimal performance. In order to alleviate the imbalance, we propose a new framework LongRAG, consisting of a `long retriever' and a `long reader'. LongRAG processes the entire Wikipedia into 4K-token units, which is 30x longer than before. By increasing the unit size, we significantly reduce the total units from 22M to 700K. This significantly lowers the burden of retriever, which leads to a remarkable retrieval score: answer recall@1=71% on NQ (previously 52%) and answer recall@2=72% (previously 47%) on HotpotQA (full-wiki). Then we feed the top-k retrieved units (approx 30K tokens) to an existing long-context LLM to perform zero-shot answer extraction. Without requiring any training, LongRAG achieves an EM of 62.7% on NQ, which is the best known result. LongRAG also achieves 64.3% on HotpotQA (full-wiki), which is on par of the SoTA model. Our study offers insights into the future roadmap for combining RAG with long-context LLMs.