从专有到开源：通过多智能体协议蒸馏弥合智能体搜索中的分布差距

摘要

智能体搜索通过将多步推理与检索交错进行，使大语言模型能够解决知识密集型任务，然而使用基于结果的强化学习对此进行优化仅能提供稀疏的监督信号。知识蒸馏能够提供更密集的指导，而具有强大推理能力的高级专有模型则是极具潜力的教师模型。尽管从专有模型进行蒸馏可强化这一监督信号，但传统的对数几率匹配由于隐藏的对数几率和不匹配的词元化器而无法实现，而原始自然语言轨迹模仿则只是传递了浅层的文体伪影，而非核心推理能力。为解决异构蒸馏问题并弥合分布差距，我们提出了多智能体协议蒸馏（MAPD），这是一种联合蒸馏与强化学习框架，使用结构化的、风格标准化的协议作为中间表示。一个离线多智能体系统（MAS）对每个查询进行分解，检索支持性证据，修复失败的搜索，并将生成的探索轨迹转化为包含任务类型、推理计划和可提取的事实依据的JSON协议。在训练过程中，该协议仅提供给学生策略的一个特权分支，其词元分布在与稀疏强化学习目标并列的同时，提供了密集的蒸馏信号。在七个问答基准上的广泛评估表明，MAPD始终优于具有竞争力的蒸馏与强化学习方法，在Qwen3-1.7B上平均成功率达到39.4%，在Qwen3-4B上达到44.4%。关键是，该框架在不同专有教师模型间具有稳健的泛化能力，同时有效缓解了学生策略中的风格漂移和冗长退化问题。

English

Agentic search enables large language models to solve knowledge-intensive tasks by interleaving multi-step reasoning with retrieval, yet optimizing this with outcome-based reinforcement learning (RL) provides only sparse supervision. Knowledge distillation can supply denser guidance, and advanced proprietary models with their strong reasoning capabilities are promising teachers. While distilling from proprietary models can densify this supervisory signal, conventional logit-matching is precluded by hidden logits and mismatched tokenizers, whereas raw natural language trajectory imitation transfers superficial stylistic artifacts rather than core reasoning competence. To address the heterogeneous distillation problem and bridge the distribution gap, we propose Multi-Agent Protocol Distillation (MAPD), a joint distillation and RL framework uses a structured, style-normalized protocol as an intermediate representation. An offline multi-agent system (MAS) decomposes each query, retrieves supporting evidence, repairs failed searches, and converts the resulting exploration trace into a JSON protocol containing the task type, reasoning plan, and extractive grounding facts. During training, the protocol is provided only to a privileged branch of the student policy, whose token distributions furnish a dense distillation signal alongside the sparse RL objective. Extensive evaluations across seven QA benchmarks demonstrate that MAPD consistently outperforms competitive distillation and RL, achieving average success rates of 39.4\% on Qwen3-1.7B and 44.4\% on Qwen3-4B. Crucially, the framework generalizes robustly across diverse proprietary teachers while effectively mitigating the student policy from style drift and verbosity degeneration.