從RAG到記憶：大型語言模型的非參數持續學習

From RAG to Memory: Non-Parametric Continual Learning for Large Language Models

February 20, 2025

作者: Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, Yu Su

cs.AI

摘要

我們持續獲取、組織並運用知識的能力，是人類智能的關鍵特徵，也是人工智慧系統必須逼近以充分發揮其潛力的目標。考慮到大型語言模型（LLMs）在持續學習方面面臨的挑戰，檢索增強生成（RAG）已成為引入新信息的主導方式。然而，其對向量檢索的依賴限制了它模擬人類長期記憶動態且相互關聯特性的能力。近期的RAG方法通過知識圖譜等多種結構增強向量嵌入，以彌補在理解與關聯性方面的不足。但這些方法在基礎事實記憶任務上的表現卻顯著低於標準RAG。我們針對這一非預期的性能下降提出解決方案，並推出HippoRAG 2框架，該框架在事實記憶、理解記憶及關聯記憶任務上全面超越標準RAG。HippoRAG 2基於HippoRAG中使用的個性化PageRank算法，通過更深層次的段落整合及更高效的LLM在線應用加以強化。這一組合使RAG系統更接近人類長期記憶的效能，在關聯記憶任務上相比最先進的嵌入模型提升了7%，同時展現出更優的事實知識與理解記憶能力。此項工作為LLMs的非參數持續學習鋪平了道路。我們的代碼與數據將於https://github.com/OSU-NLP-Group/HippoRAG發布。

English

Our ability to continuously acquire, organize, and leverage knowledge is a key feature of human intelligence that AI systems must approximate to unlock their full potential. Given the challenges in continual learning with large language models (LLMs), retrieval-augmented generation (RAG) has become the dominant way to introduce new information. However, its reliance on vector retrieval hinders its ability to mimic the dynamic and interconnected nature of human long-term memory. Recent RAG approaches augment vector embeddings with various structures like knowledge graphs to address some of these gaps, namely sense-making and associativity. However, their performance on more basic factual memory tasks drops considerably below standard RAG. We address this unintended deterioration and propose HippoRAG 2, a framework that outperforms standard RAG comprehensively on factual, sense-making, and associative memory tasks. HippoRAG 2 builds upon the Personalized PageRank algorithm used in HippoRAG and enhances it with deeper passage integration and more effective online use of an LLM. This combination pushes this RAG system closer to the effectiveness of human long-term memory, achieving a 7% improvement in associative memory tasks over the state-of-the-art embedding model while also exhibiting superior factual knowledge and sense-making memory capabilities. This work paves the way for non-parametric continual learning for LLMs. Our code and data will be released at https://github.com/OSU-NLP-Group/HippoRAG.