N-GRPO：嵌入层级的邻域混合以增强策略优化

摘要

大语言模型在数学推理中的成功高度依赖于展开阶段生成多样且有效的解答路径。然而，当前展开技术面临一个根本性权衡：词元级采样往往产生仅表述不同而实质冗余的轨迹，而利用随机噪声的嵌入级方法则频繁破坏语义连贯性。为解决这一问题，我们提出N-GRPO——一种集成于组相对策略优化（GRPO）框架中的新型探索策略。我们的方法不依赖词元级采样或原生嵌入级噪声，而是利用语义邻居混合机制。该机制通过混合锚定词元及其最近语义邻居的嵌入表示，动态构建输入表征，从而在严格遵循局部语义流形的同时注入多样性。在DeepSeek-R1-Distill-Qwen不同规模模型上的实验评估表明，N-GRPO不仅在数学推理基准测试中相较于强基线取得持续提升，而且在分布外任务上展现出稳健的泛化能力。

English

The success of Large Language Models in mathematical reasoning relies heavily on the generation of diverse and valid solution paths during the rollout phase. However, current rollout techniques face a fundamental trade-off: token-level sampling often yields redundant trajectories that differ only in rephrasing, while embedding-level methods utilizing random noise frequently disrupt semantic consistency. To resolve this, we introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization (GRPO) framework. Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing. This mechanism dynamically constructs input representations by mixing the embeddings of an anchor token and its nearest semantic neighbors, thereby injecting diversity while strictly adhering to the local semantic manifold. Experimental evaluations on the DeepSeek-R1-Distill-Qwen models across different sizes show that N-GRPO not only achieves consistent improvements over strong baselines on math reasoning benchmarks but also exhibits robust generalization capabilities on out-of-distribution tasks.