心智搜索：模拟人类思维引发深度人工智能搜索者

MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

July 29, 2024

作者: Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, Feng Zhao

cs.AI

摘要

信息搜索和整合是一项复杂的认知任务，耗费大量时间和精力。受到大型语言模型（LLMs）的显著进展的启发，最近的研究尝试通过将LLMs和搜索引擎结合来解决这一任务。然而，由于三个挑战，这些方法仍然表现不佳：（1）复杂请求通常无法被搜索引擎准确完整地检索，（2）要整合的相应信息分布在多个网页上，伴随着大量噪音，（3）大量内容较长的网页可能很快超出LLMs的最大上下文长度。受到人类解决这些问题时的认知过程的启发，我们引入MindSearch来模仿人类在网络信息搜索和整合中的思维，这可以通过一个简单而有效的基于LLMs的多智能体框架来实现。WebPlanner将多步信息搜索的人类思维建模为动态图构建过程：它将用户查询分解为图中的原子子问题节点，并根据WebSearcher的搜索结果逐步扩展图。WebSearcher负责每个子问题，通过搜索引擎进行分层信息检索，并为WebPlanner收集有价值的信息。MindSearch的多智能体设计使整个框架能够在3分钟内并行地从更大规模（例如超过300个）的网页中搜索和整合信息，相当于人类3小时的努力。MindSearch在深度和广度方面在封闭集和开放集QA问题上显著提高了响应质量。此外，基于InternLM2.5-7B的MindSearch的响应被人类更偏好于ChatGPT-Web和Perplexity.ai应用，这意味着MindSearch已经能够为专有AI搜索引擎提供具有竞争力的解决方案。

English

Information seeking and integration is a complex cognitive task that consumes enormous time and effort. Inspired by the remarkable progress of Large Language Models, recent works attempt to solve this task by combining LLMs and search engines. However, these methods still obtain unsatisfying performance due to three challenges: (1) complex requests often cannot be accurately and completely retrieved by the search engine once (2) corresponding information to be integrated is spread over multiple web pages along with massive noise, and (3) a large number of web pages with long contents may quickly exceed the maximum context length of LLMs. Inspired by the cognitive process when humans solve these problems, we introduce MindSearch to mimic the human minds in web information seeking and integration, which can be instantiated by a simple yet effective LLM-based multi-agent framework. The WebPlanner models the human mind of multi-step information seeking as a dynamic graph construction process: it decomposes the user query into atomic sub-questions as nodes in the graph and progressively extends the graph based on the search result from WebSearcher. Tasked with each sub-question, WebSearcher performs hierarchical information retrieval with search engines and collects valuable information for WebPlanner. The multi-agent design of MindSearch enables the whole framework to seek and integrate information parallelly from larger-scale (e.g., more than 300) web pages in 3 minutes, which is worth 3 hours of human effort. MindSearch demonstrates significant improvement in the response quality in terms of depth and breadth, on both close-set and open-set QA problems. Besides, responses from MindSearch based on InternLM2.5-7B are preferable by humans to ChatGPT-Web and Perplexity.ai applications, which implies that MindSearch can already deliver a competitive solution to the proprietary AI search engine.