携手共进：面向协作大语言模型的同策略强化学习

摘要

多智能体系统（MAS）与强化学习（RL）被广泛应用于提升大型语言模型（LLMs）的代理能力。MAS通过基于角色的编排优化任务执行，而RL则利用环境奖励学习更优策略，如GRPO风格的优化方法。然而，将在线策略RL应用于MAS仍属探索不足的领域，并面临独特挑战。算法层面，由于提示信息随角色和轮次变化，标准的GRPO分组假设不再适用。系统层面，训练框架需支持MAS工作流的展开及对单策略与多策略模型的在线策略更新。我们提出了AT-GRPO，它包括：(i) 一种专为MAS设计的、按智能体和轮次分组的RL算法；(ii) 一个支持单策略与多策略模式的训练系统。在游戏、规划、编程及数学任务中，AT-GRPO均带来显著提升。在长期规划任务上，它将单智能体RL基准的准确率从14.0%至47.0%提升至96.0%至99.5%。同时，它增强了推理性能，在编程任务上平均提升3.87%至7.62%，在数学任务上提升9.0%至17.93%。代码及环境已发布于：https://github.com/pettingllms-ai/PettingLLMs。

English

Multi-agent systems (MAS) and reinforcement learning (RL) are widely used to enhance the agentic capabilities of large language models (LLMs). MAS improves task performance through role-based orchestration, while RL uses environmental rewards to learn stronger policies, such as GRPO-style optimization. However, applying on-policy RL to MAS remains underexplored and presents unique challenges. Algorithmically, standard GRPO grouping assumptions break down because prompts vary by role and by turn. System-wise, the training stack must support MAS-workflow rollouts and on-policy updates for both single-policy and multi-policy models. We propose AT-GRPO, which includes (i) an agent- and turn-wise grouped RL algorithm tailored to MAS and (ii) a training system that supports both single- and multi-policy regimes. Across game, planning, coding, and math tasks, AT-GRPO delivers substantial gains. On long-horizon planning, it increases accuracy from a 14.0 to 47.0 percent single-agent RL baseline to 96.0 to 99.5 percent. It also improves reasoning performance, with average gains of 3.87 to 7.62 percent on coding tasks and 9.0 to 17.93 percent on math. Code and environments are available at: https://github.com/pettingllms-ai/PettingLLMs.

携手共进：面向协作大语言模型的同策略强化学习

Stronger Together: On-Policy Reinforcement Learning for Collaborative LLMs

摘要

Support