SkillGate：长时域智能体的策略内技能选择训练

摘要

智能体框架日益将程序性知识封装为技能：智能体按需读取的指令文件，而公共库中现已存储数千个此类技能。因此，读取哪个技能成为策略在回合中途自行做出的决策，然而目前没有任何现有信号训练这一决策。我们证明，默认的补救方案——在候选技能列表上进行结果奖励强化学习——无法教会该决策，其原因是一种结构性缺陷，我们将其识别并命名为“选择器信用匮乏”：在广播式的序列级优势下，命名所选技能的少数词元只承担趋近于零的损失份额，并且随着轨迹变长，它们继承的信用越发趋向错误符号。只要该选择之后的执行失败，即使这一选择本身是轨迹中最有价值的决策之一，它也会受到惩罚。对一次完整运行自身的训练产物进行审计，证实了上述三个性质，且每个性质都随轨迹长度单调恶化。SkillGate 从构造上消除了这一失效：它将词元支撑集划分为两个不相交的信用通道——结果信用只到达执行词元，而独立的动作局部优势恰好到达命名技能的词元，并且仅当轨迹中唯一一次读取是正确时才为正。在五个智能体基准测试中，面对16个候选技能的列表，SkillGate 将 9B 参数策略的试验成功率从 40.8% 提升至 53.2%，远超将相同预算仅用于结果奖励的效果，同时将暴露于误导性候选的比例削减三分之二，且读取的技能更少。

English

Agent frameworks increasingly package procedural knowledge as skills: instruction files an agent reads on demand, while public libraries now hold thousands of them. Which skill to read has thus become a decision the policy itself makes in the middle of an episode, yet no existing signal trains it. We show that the default remedy, outcome-rewarded RL over the candidate slate, cannot teach it, for a structural reason we identify and name selector credit starvation: under a broadcast, sequence-level advantage, the few tokens that name the chosen skill carry a vanishing share of the loss, and the credit they inherit is increasingly wrong-signed as trajectories lengthen. A correct choice is punished whenever the execution after it fails, even though the choice itself is among the most valuable decisions in the trajectory. Auditing a completed run's own training artifacts confirms all three properties, each worsening monotonically with horizon. SkillGate removes the failure by construction: it partitions the token support into two disjoint credit channels, outcome credit reaching only execution tokens, and a separate action-local advantage reaching exactly the skill-naming tokens, positive only when a trajectory's single read is the correct one. On five agentic benchmarks under a 16-candidate slate, SkillGate lifts a 9B policy from 40.8% to 53.2% trial success, well ahead of the identical budget spent on outcome reward alone, while cutting exposure to misleading candidates by two thirds and reading fewer skills.