從專有到開源：透過智能體搜索中的多智能體協議蒸餾彌合分佈差距

摘要

代理搜索通過將多步驟推理與檢索交織，使大型語言模型能夠解決知識密集型任務，然而以結果為基礎的強化學習（RL）進行優化僅能提供稀疏的監督信號。知識蒸餾則能提供更密集的引導，而具備強大推理能力的高階專有模型正是理想的教師模型。雖然從專有模型進行蒸餾可以強化此監督信號，但傳統的logit匹配因隱藏logit與不匹配的分詞器而無法實現，而純粹的自然語言軌跡模仿僅會傳輸表面的風格痕跡，而非核心推理能力。為了解決異質蒸餾問題並彌合分佈差距，我們提出多智能體協議蒸餾（Multi-Agent Protocol Distillation, MAPD），這是一個結合蒸餾與RL的框架，採用結構化且風格標準化的協議作為中間表示。一個離線多智能體系統（MAS）會將每個查詢分解、檢索支持證據、修復失敗的搜索，並將最終的探索軌跡轉換為包含任務類型、推理計畫及提取式基礎事實的JSON協議。在訓練過程中，該協議僅提供給學生策略中的特權分支，其token分佈在稀疏的RL目標之外，提供了密集的蒸餾信號。在七個問答基準測試上的廣泛評估顯示，MAPD始終優於具競爭力的蒸餾與RL方法，在Qwen3-1.7B上平均成功率達39.4%，在Qwen3-4B上則達44.4%。關鍵在於，該框架能穩健地泛化至多種不同的專有教師模型，同時有效減輕學生策略的風格漂移與冗餘退化問題。

English

Agentic search enables large language models to solve knowledge-intensive tasks by interleaving multi-step reasoning with retrieval, yet optimizing this with outcome-based reinforcement learning (RL) provides only sparse supervision. Knowledge distillation can supply denser guidance, and advanced proprietary models with their strong reasoning capabilities are promising teachers. While distilling from proprietary models can densify this supervisory signal, conventional logit-matching is precluded by hidden logits and mismatched tokenizers, whereas raw natural language trajectory imitation transfers superficial stylistic artifacts rather than core reasoning competence. To address the heterogeneous distillation problem and bridge the distribution gap, we propose Multi-Agent Protocol Distillation (MAPD), a joint distillation and RL framework uses a structured, style-normalized protocol as an intermediate representation. An offline multi-agent system (MAS) decomposes each query, retrieves supporting evidence, repairs failed searches, and converts the resulting exploration trace into a JSON protocol containing the task type, reasoning plan, and extractive grounding facts. During training, the protocol is provided only to a privileged branch of the student policy, whose token distributions furnish a dense distillation signal alongside the sparse RL objective. Extensive evaluations across seven QA benchmarks demonstrate that MAPD consistently outperforms competitive distillation and RL, achieving average success rates of 39.4\% on Qwen3-1.7B and 44.4\% on Qwen3-4B. Crucially, the framework generalizes robustly across diverse proprietary teachers while effectively mitigating the student policy from style drift and verbosity degeneration.