SkillGate：長時程智能體中的策略內技能選擇訓練

摘要

Agent框架日益將程序性知識封裝為技能：智能體按需讀取的指令檔案，而公開函式庫中現已存放數千個此類技能。因此，讀取哪個技能已成為策略本身在回合中途做出的決策，然而現有信號並未對其進行訓練。我們表明，預設的補救措施——對候選清單進行結果獎勵強化學習——無法教會這項決策，原因在於一個我們識別並命名為「選擇器信用飢餓」的結構性問題。在廣播式的序列級優勢下，少數命名所選技能的詞元承擔了損失中微乎其微的份額，而且隨著軌跡變長，它們繼承的信用在符號上越來越錯誤。只要其後的執行失敗，正確的選擇就會受到懲罰，儘管該選擇本身是軌跡中最有價值的決策之一。對一次完整運行自身的訓練產物進行審查，證實了這三個特性，且每個特性都隨軌跡長度單調惡化。SkillGate從構造上消除了這一失敗：它將詞元支撐集分割為兩個不相交的信用通道——結果信用僅到達執行詞元，而一個獨立的動作局部優勢則恰好到達命名技能的詞元，且僅當軌跡中唯一一次讀取是正確的時為正。在五個智能體基準上，面對16個候選的清單，SkillGate將一個9B策略的試驗成功率從40.8%提升至53.2%，遠超將相同預算僅用於結果獎勵的做法，同時將對誤導候選的接觸削減三分之二，並讀取更少的技能。

English

Agent frameworks increasingly package procedural knowledge as skills: instruction files an agent reads on demand, while public libraries now hold thousands of them. Which skill to read has thus become a decision the policy itself makes in the middle of an episode, yet no existing signal trains it. We show that the default remedy, outcome-rewarded RL over the candidate slate, cannot teach it, for a structural reason we identify and name selector credit starvation: under a broadcast, sequence-level advantage, the few tokens that name the chosen skill carry a vanishing share of the loss, and the credit they inherit is increasingly wrong-signed as trajectories lengthen. A correct choice is punished whenever the execution after it fails, even though the choice itself is among the most valuable decisions in the trajectory. Auditing a completed run's own training artifacts confirms all three properties, each worsening monotonically with horizon. SkillGate removes the failure by construction: it partitions the token support into two disjoint credit channels, outcome credit reaching only execution tokens, and a separate action-local advantage reaching exactly the skill-naming tokens, positive only when a trajectory's single read is the correct one. On five agentic benchmarks under a 16-candidate slate, SkillGate lifts a 9B policy from 40.8% to 53.2% trial success, well ahead of the identical budget spent on outcome reward alone, while cutting exposure to misleading candidates by two thirds and reading fewer skills.