多教師同策略蒸餾中的工具調用邊界漂移診斷與校準

摘要

具有代理能力的語言模型必須學會何時呼叫工具、何時消費工具回應，以及何時直接回答。這使得多教師在策略蒸餾成為一種自然的訓練策略：一位教師可專精於工具呼叫，另一位專精於直接回答，而學生可以從兩者在其自身生成的分佈上學習。我們證明，此策略可能引發一種從聚合損失中無法察覺的行為偏移。在雙教師工具使用設定中，標準的廣義知識蒸餾提升了工具呼叫的召回率，但也使模型傾向於過度呼叫，即在應該直接回答的範例上呼叫工具。聚合解釋是不夠的：工具呼叫樣本並未獲得更多的詞元曝光，且工具呼叫教師的全序列逐詞元散度並未更大。我們反其道而行，分析行為槓桿不平衡：在模式進入和結構位置（如<tool_call>和函數名稱）的局部詞元層級信號，可能對全局生成模式產生不成比例的影響。我們提出Soft Clamp，一種逐詞元散度校準方法，能動態壓縮極端的詞元層級JS散度，同時保留非零梯度。在APIGen-MT上，相較於標準GKD，Soft Clamp將過度呼叫從13.7%降至9.0%，同時保持決策準確性相當。在BFCL多輪診斷中，它還降低了GKD變體中的工具呼叫循環和重複呼叫。這些結果表明，多教師OPD應監控教師信號作用的位置，而不僅僅是其聚合的大小。

English

Agentic language models must learn when to call tools, when to consume tool responses, and when to answer directly. This makes multi-teacher on-policy distillation a natural training strategy: one teacher can specialize in tool calls, another in direct responses, and the student can learn from both on its own generated distribution. We show that this strategy can induce a behavior shift that is invisible from aggregate losses alone. In a two-teacher tool-use setting, vanilla generalized knowledge distillation improves tool-call recall but also moves the model toward over-calling, where it calls tools on examples that should be answered directly. Aggregate explanations are insufficient: tool-call samples do not receive more token exposure, and full-sequence per-token divergence is not larger for the tool-call teacher. We instead analyze behavior leverage imbalance: local token-level signals at mode- entry and structural positions, such as <tool_call> and function names, can have disproportionate control over the global generation mode. We propose Soft Clamp, a per-token divergence calibration method that dynamically compresses extreme token-level Jensen-Shannon divergence while preserving nonzero gradients. On APIGen-MT, Soft Clamp reduces over-calling from 13.7% to 9.0% relative to vanilla GKD while matching its decision accuracy. In a BFCL multi-turn diagnostic, it also lowers tool-call loops and repeated calls among GKD variants. These results suggest that multi-teacher OPD should monitor where teacher signals act, not only how large they are in aggregate.