结合在策略内在知识边界增强的高效智能体强化学习

摘要

智能体强化学习（agentic RL）已被证明在训练具备外部工具使用能力的大语言模型（LLM）智能体方面卓有成效。然而，我们发现智能体强化学习训练会导致冗余工具调用增加，并模糊模型的内在知识边界——即模型无法区分何时需要调用工具、何时仅凭参数化知识即可作答。现有基于奖励塑形的方法提供了粗粒度的优化目标，往往倾向于不加区分地抑制工具调用，从而引发奖励作弊。本文提出AKBE（智能体知识边界增强），这是一种在训练期间通过双路径（含工具路径与无工具路径）回滚动态探测模型内在知识边界的在策略方法。我们将知识边界定义为：针对每个实例，判断是否需要工具以及所需的最小工具调用次数。通过比较不同路径的正确性，AKBE对轨迹进行分类，并构建目标明确的监督信号，引导每个问题形成高效的工具使用模式。这些信号无缝集成到智能体强化学习训练循环中。在七个问答基准上的实验表明，与标准智能体强化学习相比，AKBE平均任务准确率提升+1.85，工具调用次数减少18%，工具生产力提高25%，且未对准确率与效率造成权衡折衷。进一步分析表明，该方法在不同强化学习算法上具有即插即用的兼容性，并揭示了各信号类别的作用机制。我们的代码已开源至https://github.com/CuSO4-Chen/AKBE。

English

Agentic reinforcement learning (RL) has proven effective for training LLM-based agents with external tool-use capabilities. However, we identify that agentic RL training induces increasing redundant tool calls and blurs the model's intrinsic knowledge boundary, where the model fails to distinguish when tools are needed versus when parametric knowledge suffices. Existing solutions based on reward shaping create coarse-grained optimization targets that tend to incentivize indiscriminate tool-call suppression, leading to reward hacking. In this paper, we propose AKBE (Agentic Knowledge Boundary Enhancement), an on-policy method that dynamically probes the model's intrinsic knowledge boundary through dual-path (with-tool and no-tool) rollouts during training. We define the knowledge boundary as the per-instance determination of whether tools are required and the minimum tool calls necessary. By comparing correctness across paths, AKBE categorizes trajectories and constructs targeted supervisory signals that guide efficient tool-use patterns for each question. These signals are integrated seamlessly into the agentic RL training loop. Experiments on seven QA benchmarks demonstrate that AKBE improves task accuracy by +1.85 on average and reduces tool calls by 18% over standard agentic RL, yielding 25% higher tool productivity without any accuracy-efficiency trade-off. Further analysis suggests its plug-and-play compatibility across different RL algorithms and the mechanism of each signal category. Our code is available at https://github.com/CuSO4-Chen/AKBE.