一層就夠了嗎？訓練單一Transformer層可媲美全參數強化學習訓練

摘要

強化學習（RL）已成為大型語言模型（LLM）後訓練的核心環節，然而目前對於RL適應性如何在Transformer層之間分佈的理解仍十分有限。現有方法通常均勻更新所有模型參數，隱含假設每一層對RL後訓練所獲得的性能提升貢獻相似。在本研究中，我們透過系統性的逐層研究來挑戰這一假設。令人驚訝的是，我們發現僅訓練單一Transformer層即可恢復全參數RL訓練所獲得的大部分增益，在某些情況下甚至能超越之。為量化此現象，我們引入了「層貢獻」這一指標，用以衡量單獨訓練某一層所恢復的全參數RL改進比例。在涵蓋兩個模型家族（Qwen3、Qwen2.5）、三種RL演算法（GRPO、GiGPO、Dr. GRPO）以及包括數學推理、程式碼生成與自主決策在內的多個任務領域的七個模型中，我們觀察到一個極為穩定的模式：RL增益高度集中於一小部分，且在許多情況下甚至僅集中於單一Transformer層。更引人注目的是，相同的結構模式始終如一地浮現：高貢獻層集中在Transformer堆疊的中間部分，而靠近輸入與輸出端的層貢獻則明顯較小。由此產生的層級排序在不同資料集、任務、模型家族與RL演算法之間均呈現高度相關性。

English

Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across transformer layers. Existing approaches typically update all model parameters uniformly, implicitly assuming that every layer contributes similarly to the gains obtained during RL post-training. In this work, we challenge this assumption through a systematic layer-wise study of RL training. Surprisingly, we find that training a single transformer layer can recover most of the gains achieved by full-parameter RL training, and in some cases even surpass it. To quantify this phenomenon, we introduce the quantity layer contribution, which measures the fraction of full RL improvement recovered by training a layer in isolation. Across seven models spanning two model families (Qwen3, Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains including mathematical reasoning, code generation, and agentic decision-making, we observe a remarkably stable pattern: RL gains are highly concentrated in a small subset of, and in many cases even a single, transformer layers. More strikingly, the same structural pattern consistently emerges: high-contribution layers concentrate in the middle of the transformer stack, while layers near the input and output ends contribute substantially less. The resulting layer rankings remain strongly correlated across datasets, tasks, model families, and RL algorithms.