群熵控制策略优化

摘要

熵控制已成为大语言模型强化学习中的有效工具，有助于在对齐过程中平衡探索与利用的权衡。这种强化学习范式通常针对混合异构任务进行，这些任务在同一策略下会引发不同的熵分布，使得全局或词元级别的熵调控难以满足相应的异构探索需求。这种异质性进一步导致GRPO风格的归一化优势产生熵依赖偏差，使得不同提示组间的优势信号在统计上不可比。为解决这一问题，我们提出组熵控制策略优化（GEPO），这是对GRPO的轻量级扩展，利用从现有分组样本中估计的组熵，进行基于熵条件的非对称优势塑造。GEPO会衰减低熵组中的正优势以减少过度利用，同时抑制高熵组中的负优势以保留探索能力，并利用历史熵统计量推导自适应阈值。在涵盖数学、物理、科学、代码生成和指令跟随的十三个基准测试中，基于两个基础模型的大量实验表明，GEPO在跨任务改进上始终优于GRPO及近期提出的熵控制方法，能在整个训练过程中保持平衡的跨任务性能提升，同时维持任务特定的探索水平。

English

Entropy control has become an effective tool in reinforcement learning (RL) of large language models (LLMs), helping balance exploration-exploitation trade-off during alignment process. Such RL paradigm is often conducted on mixtures of heterogeneous tasks, which induce distinct entropy regimes under the same policy, making global or token-level entropy regulation insufficient to corresponding heterogeneous needs of exploration. This heterogeneity further makes GRPO-style normalized advantages induce an entropy-dependent bias, making advantage signals across prompt groups statistically non-comparable. To address this issue, we propose Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy, estimated from existing grouped samples to perform entropy-conditioned asymmetric advantage shaping. GEPO attenuates positive advantages in low-entropy groups to reduce over-exploitation, and negative advantages in high-entropy groups to preserve exploration, with adaptive thresholds derived from historical entropy statistics. Extensive experiments on two base models across thirteen benchmarks spanning mathematics, physics, science, code generation, and instruction following show that GEPO consistently outperforms GRPO and recent entropy-controlled methods, delivering balanced cross-task improvements while preserving task-specific exploration levels throughout training.