透過多智能體協作擴展大型語言模型上下文窗口之外的外部知識輸入

Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration

May 27, 2025

作者: Zijun Liu, Zhennan Wan, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu

cs.AI

摘要

隨著推理與資訊檢索後處理技術的快速進步，大型語言模型（LLMs）能夠整合大量檢索到的知識來解決複雜任務。然而，LLMs有限的上下文窗口阻礙了外部知識輸入規模的擴展，限制了進一步的性能提升，尤其是對於需要大量外部知識的任務。現有的上下文窗口擴展方法不可避免地會導致資訊損失。基於LLM的多智能體方法作為一種新範式應運而生，以分佈式方式處理海量輸入，我們在現有的知識同步與推理過程中識別出兩個核心瓶頸。在本研究中，我們開發了一個多智能體框架——ExtAgents，以克服這些瓶頸，並在不進行長上下文訓練的情況下，實現推理時知識整合的更好可擴展性。通過我們增強的多跳問答測試集$boldsymbol{inftyBench+}$以及其他包括長篇調查生成的公開測試集進行基準測試，ExtAgents在相同外部知識輸入量的情況下，無論其是否超出上下文窗口，均顯著提升了性能，超越了現有的非訓練方法。此外，由於高度並行性，該方法保持了高效率。進一步研究在增加外部知識輸入時LLM智能體的協調，將有益於現實世界的應用。

English

With the rapid advancement of post-training techniques for reasoning and information seeking, large language models (LLMs) can incorporate a large quantity of retrieved knowledge to solve complex tasks. However, the limited context window of LLMs obstructs scaling the amount of external knowledge input, prohibiting further improvement, especially for tasks requiring significant amount of external knowledge. Existing context window extension methods inevitably cause information loss. LLM-based multi-agent methods emerge as a new paradigm to handle massive input in a distributional manner, where we identify two core bottlenecks in existing knowledge synchronization and reasoning processes. In this work, we develop a multi-agent framework, ExtAgents, to overcome the bottlenecks and enable better scalability in inference-time knowledge integration without longer-context training. Benchmarked with our enhanced multi-hop question answering test, $boldsymbol{inftyBench+}, and other public test sets including long survey generation, ExtAgents significantly enhances the performance over existing non-training methods with the same amount of external knowledge input, regardless of whether it falls within or exceeds the context window$. Moreover, the method maintains high efficiency due to high parallelism. Further study in the coordination of LLM agents on increasing external knowledge input could benefit real-world applications.