CoMAS：基於互動獎勵的共演化多智能體系統

摘要

自我演化是促使基於大型語言模型（LLM）的代理在預訓練後持續提升能力的核心研究課題。近期研究見證了從無強化學習（RL）方法向基於RL方法的轉變。當前的基於RL的方法，或依賴於密集的外部獎勵信號，或從LLM自身提取內在獎勵信號。然而，這些方法與人類智能中觀察到的自我演化機制相悖，在人類智能中，個體通過相互討論與協作來學習與進步。本研究中，我們引入了共演化多代理系統（CoMAS），這是一種新穎的框架，使代理能夠在無外部監督的情況下，通過代理間互動自主學習並提升。CoMAS從豐富的討論動態中生成內在獎勵，採用LLM作為裁判的機制來制定這些獎勵，並通過RL優化每個代理的策略，從而實現去中心化且可擴展的共演化。實驗結果表明，CoMAS在大多數評估設定中均優於未經訓練的代理，並達到了最先進的性能。消融研究證實了基於互動的獎勵信號的必要性，並揭示了隨著代理數量與多樣性的增加，其可擴展性前景廣闊。這些發現確立了CoMAS作為LLM基代理自我演化的一種新穎且有效的範式。

English

Self-evolution is a central research topic in enabling large language model (LLM)-based agents to continually improve their capabilities after pretraining. Recent research has witnessed a transition from reinforcement learning (RL)-free to RL-based methods. Current RL-based methods either rely on dense external reward signals or extract intrinsic reward signals from LLMs themselves. However, these approaches diverge from the self-evolution mechanisms observed in human intelligence, where individuals learn and improve through mutual discussion and collaboration. In this work, we introduce Co-Evolving Multi-Agent Systems (CoMAS), a novel framework that enables agents to improve autonomously by learning from inter-agent interactions without external supervision. CoMAS generates intrinsic rewards from rich discussion dynamics, employs an LLM-as-a-judge mechanism to formulate these rewards, and optimizes each agent's policy through RL, thereby enabling decentralized and scalable co-evolution. Experimental results demonstrate that CoMAS consistently outperforms untrained agents and achieves state-of-the-art performance across most evaluation settings. Ablation studies confirm the necessity of interaction-based reward signals and reveal promising scalability as the number and diversity of agents increase. These findings establish CoMAS as a novel and effective paradigm for self-evolution in LLM-based agents.

CoMAS：基於互動獎勵的共演化多智能體系統

CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards

摘要

Support