理解LLM推理的演化策略：比GRPO更廣泛的推理覆蓋範圍

摘要

演化策略（ES）近年來已成為大型語言模型（LLM）推理的一種記憶體高效後訓練範式。然而，ES 的最佳化行為仍缺乏深入研究，因此難以界定其相較於主流後訓練範式（例如群組相對策略最佳化，GRPO）的優勢範圍。透過系統性探究 ES 的動態與機制，本文首先指出 ES 相較於 GRPO 具有性能優勢，並從理論與實證兩方面證明 ES 能導致更廣泛的推理覆蓋範圍，從而更充分地發揮預訓練 LLM 的推理能力。理論上，我們證明 ES 群體中經驗證器投影的 Jensen-Shannon 多樣性有助於提升 Pass@K 性能。實證上，不同於表現出熵崩潰的 GRPO，ES 在提升 Pass@1 的同時，也能達到比 GRPO 更高的 Pass@K。我們進一步開發了一種序列式 GRPO-ES 訓練策略，結合 GRPO 在 Pass@1 的優勢與 ES 在 Pass@K 的增益。其次，我們發現儘管整體模型參數漂移顯著，ES 的任務性能增益僅由稀疏的子集、較大幅度的更新所貢獻。這種功能稀疏性表明，參數的大幅移動不一定意味著廣泛的功能變化，而保留資料的評估進一步顯示，這也不必然導致災難性遺忘。最後，我們研究超參數設計如何影響 ES 的有效性，證明在較大的 LLM 中，ES 需要較小的群體規模。這些發現將 ES 定位為一種獨特的推理後訓練範式，而非 GRPO 那種效果較差、僅具記憶體效率的替代方案。

English

Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.