多教师在策略蒸馏中的工具调用边界漂移诊断与校准

摘要

智能体语言模型必须学会何时调用工具、何时处理工具响应、以及何时直接作答。这使得多教师同策略蒸馏成为一种自然的训练策略：一位教师可专注于工具调用，另一位专注于直接回应，而学生则能基于自身生成的分布同时向两者学习。我们发现，这种策略可能引发仅从聚合损失中无法察觉的行为偏移。在双教师工具使用场景下，标准广义知识蒸馏虽能提升工具调用召回率，但也会导致模型过度调用——即在本应直接回答的示例中调用工具。聚合指标无法充分解释这种现象：工具调用样本并未获得更多令牌曝光量，且工具调用教师对应的全序列逐令牌散度也并非更大。为此，我们分析了行为杠杆不平衡现象：在模式入口和结构位置（如<tool_call>和函数名称）处的局部令牌级信号，可能对全局生成模式产生不成比例的控制力。我们提出Soft Clamp方法，这是一种逐令牌散度校准技术，能在保留非零梯度的同时动态压缩极端的令牌级詹森-香农散度。在APIGen-MT数据集上，Soft Clamp将过度调用率从标准GKD的13.7%降至9.0%，同时保持决策准确率。在多轮对话诊断任务BFCL中，该方法相较于其他GKD变体，进一步减少了工具调用循环和重复调用次数。这些结果表明，多教师同策略蒸馏应当关注教师信号的作用位置，而不仅是其聚合量级。

English

Agentic language models must learn when to call tools, when to consume tool responses, and when to answer directly. This makes multi-teacher on-policy distillation a natural training strategy: one teacher can specialize in tool calls, another in direct responses, and the student can learn from both on its own generated distribution. We show that this strategy can induce a behavior shift that is invisible from aggregate losses alone. In a two-teacher tool-use setting, vanilla generalized knowledge distillation improves tool-call recall but also moves the model toward over-calling, where it calls tools on examples that should be answered directly. Aggregate explanations are insufficient: tool-call samples do not receive more token exposure, and full-sequence per-token divergence is not larger for the tool-call teacher. We instead analyze behavior leverage imbalance: local token-level signals at mode- entry and structural positions, such as <tool_call> and function names, can have disproportionate control over the global generation mode. We propose Soft Clamp, a per-token divergence calibration method that dynamically compresses extreme token-level Jensen-Shannon divergence while preserving nonzero gradients. On APIGen-MT, Soft Clamp reduces over-calling from 13.7% to 9.0% relative to vanilla GKD while matching its decision accuracy. In a BFCL multi-turn diagnostic, it also lowers tool-call loops and repeated calls among GKD variants. These results suggest that multi-teacher OPD should monitor where teacher signals act, not only how large they are in aggregate.