强化智能体模型中的行为知识融合

摘要

强化学习（RL）在模型后训练中具有核心地位，尤其对于需要专业推理行为的智能体模型而言。在此背景下，模型融合提供了一种实用机制，可将来自不同任务的多个RL训练智能体整合为单一通用模型。然而，现有融合方法专为监督微调（SFT）设计，在保留RL训练智能体模型的特定任务能力方面存在不足。其根本原因在于RL与SFT之间存在任务向量失配：同策略RL产生的任务向量具有高度稀疏性和异质性，而SFT式融合隐式假设任务向量具备稠密性和全局可比性。当在这种失配情况下应用标准全局平均法时，RL中编码关键任务特定行为的非重叠任务向量会被削弱，参数更新也随之稀释。为解决该问题，我们提出强化智能体融合（RAM）——专为RL训练智能体模型设计的分布感知融合框架。RAM通过解耦共享参数更新与任务特异性独有参数更新，对共享组件进行平均处理，同时选择性保留并重新缩放独有组件以抵消参数更新稀释。跨多个智能体领域和模型架构的实验表明，RAM不仅超越了现有融合基线，更能释放智能体间的协同潜力，实现优于各领域专用智能体的性能表现。

English

Reinforcement learning (RL) is central to post-training, particularly for agentic models that require specialized reasoning behaviors. In this setting, model merging offers a practical mechanism for integrating multiple RL-trained agents from different tasks into a single generalist model. However, existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models. The root is a task-vector mismatch between RL and SFT: on-policy RL induces task vectors that are highly sparse and heterogeneous, whereas SFT-style merging implicitly assumes dense and globally comparable task vectors. When standard global averaging is applied under this mismatch, RL's non-overlapping task vectors that encode critical task-specific behaviors are reduced and parameter updates are diluted. To address this issue, we propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models. RAM disentangles shared and task-specific unique parameter updates, averaging shared components while selectively preserving and rescaling unique ones to counteract parameter update dilution. Experiments across multiple agent domains and model architectures demonstrate that RAM not only surpasses merging baselines, but also unlocks synergistic potential among agents to achieve performance superior to that of specialized agents in their domains.

强化智能体模型中的行为知识融合

Behavior Knowledge Merge in Reinforced Agentic Models

摘要

Support