单层就够了吗？训练单一Transformer层即可媲美全参数强化学习训练。

摘要

强化学习（RL）已成为大型语言模型（LLM）后训练的核心组成部分，然而关于RL适应过程如何在Transformer各层之间分布，目前仍知之甚少。现有方法通常对所有模型参数进行统一更新，隐含地假设每一层对RL后训练所获得的性能提升贡献相同。在本工作中，我们通过对RL训练进行系统性的逐层研究来挑战这一假设。令人惊讶的是，我们发现仅训练单个Transformer层就能恢复全参数RL训练所带来的大部分收益，在某些情况下甚至能超越全参数训练。为了量化这一现象，我们引入了“层贡献”这一指标，用于衡量单独训练某一层时所能恢复的全参数RL改进比例。在涵盖两个模型系列（Qwen3、Qwen2.5）、三种RL算法（GRPO、GiGPO、Dr. GRPO）以及包括数学推理、代码生成和智能决策在内的多个任务领域的七个模型上，我们观察到一个极为稳定的模式：RL的收益高度集中在少数几个Transformer层中，在许多情况下甚至仅集中于一个层。更引人注目的是，相同的结构性模式一致出现：高贡献层集中在Transformer堆叠的中部区域，而靠近输入和输出端的层贡献则显著更少。由此产生的层排名在不同数据集、任务、模型系列和RL算法之间保持高度相关。

English

Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across transformer layers. Existing approaches typically update all model parameters uniformly, implicitly assuming that every layer contributes similarly to the gains obtained during RL post-training. In this work, we challenge this assumption through a systematic layer-wise study of RL training. Surprisingly, we find that training a single transformer layer can recover most of the gains achieved by full-parameter RL training, and in some cases even surpass it. To quantify this phenomenon, we introduce the quantity layer contribution, which measures the fraction of full RL improvement recovered by training a layer in isolation. Across seven models spanning two model families (Qwen3, Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains including mathematical reasoning, code generation, and agentic decision-making, we observe a remarkably stable pattern: RL gains are highly concentrated in a small subset of, and in many cases even a single, transformer layers. More strikingly, the same structural pattern consistently emerges: high-contribution layers concentrate in the middle of the transformer stack, while layers near the input and output ends contribute substantially less. The resulting layer rankings remain strongly correlated across datasets, tasks, model families, and RL algorithms.