混合智能体增强大型语言模型能力

Mixture-of-Agents Enhances Large Language Model Capabilities

June 7, 2024

作者: Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, James Zou

cs.AI

摘要

最近对大型语言模型（LLMs）的研究取得了显著进展，展示了在自然语言理解和生成任务方面的实质性能力。随着LLMs数量的增长，如何利用多个LLMs的集体专业知识是一个令人兴奋的开放方向。为了实现这一目标，我们提出了一种新方法，通过“混合代理人”（MoA）方法利用多个LLMs的集体优势。在我们的方法中，我们构建了一个分层MoA架构，其中每一层包含多个LLM代理人。每个代理人将前一层代理人的所有输出作为辅助信息，用于生成其响应。MoA模型在AlpacaEval 2.0、MT-Bench和FLASK上实现了最先进的性能，超越了GPT-4 Omni。例如，我们仅使用开源LLMs的MoA在AlpacaEval 2.0中领先GPT-4 Omni相当大的差距，取得了65.1%的得分，而GPT-4 Omni仅为57.5%。

English

Recent advances in large language models (LLMs) demonstrate substantial capabilities in natural language understanding and generation tasks. With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction. Toward this goal, we propose a new approach that leverages the collective strengths of multiple LLMs through a Mixture-of-Agents (MoA) methodology. In our approach, we construct a layered MoA architecture wherein each layer comprises multiple LLM agents. Each agent takes all the outputs from agents in the previous layer as auxiliary information in generating its response. MoA models achieves state-of-art performance on AlpacaEval 2.0, MT-Bench and FLASK, surpassing GPT-4 Omni. For example, our MoA using only open-source LLMs is the leader of AlpacaEval 2.0 by a substantial gap, achieving a score of 65.1% compared to 57.5% by GPT-4 Omni.