理解用于大语言模型推理的进化策略：比GRPO更广泛的推理覆盖

摘要

进化策略（ES）近来已成为大语言模型（LLM）推理后训练的一种内存高效范式。然而，ES的优化行为尚未得到充分研究，难以界定其相较于主流后训练范式（如组相对策略优化（GRPO））的优势范围。通过系统考察ES的动力学与机制，本文首先识别出ES相对于GRPO的性能优势，从理论和实证两个层面表明ES能够带来更广泛的推理覆盖，从而更好地发掘预训练LLM的推理能力。在理论上，我们证明ES种群中验证器投影的Jensen-Shannon散度有助于提升Pass@K性能。在实证上，与表现出熵坍缩的GRPO不同，ES在获得更高Pass@K的同时提升了Pass@1。我们进一步开发了一种顺序式GRPO-ES训练策略，将GRPO在Pass@1上的优势与ES在Pass@K上的增益相结合。其次，我们发现尽管整体模型参数漂移显著，ES的任务性能增益仅由幅度较大的更新所构成的稀疏子集贡献。这种功能稀疏性表明，较大的参数变动未必意味着广泛的功能改变，而保留集评估进一步显示，这也不必然导致灾难性遗忘。最后，我们研究了超参数设计对ES有效性的影响，证明在更大的LLM中ES需要更小的种群规模。这些发现将ES定位为一种独特的推理后训练范式，而非GRPO的一种效果较差但内存高效的替代方案。

English

Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.