邁向小型語言模型智能體的穩健強化學習

摘要

使用强化学习对参数量在70-500M范围内的小型语言模型（SLMs）进行对齐通常被认为不稳定，但其潜在失败机制尚未得到系统性研究。在最新研究中，通过近端策略优化（PPO）训练了十五个（模型、语料库）配置组合。实验包括在TinyStories、CNN/DailyMail和Wikitext-103语料库上分别训练Pythia-70M、160M、410M以及SmolLM2-135M、360M。在小型语言模型中识别出三种可复现的失败模式：标准PEFT/TRL流程中LoRA参数的静默冻结、使用bfloat16时重要性比率数值溢出，以及由奖励模型误差引发的灾难性策略崩溃。针对这些问题，采用合并与重新初始化的适配器技术、在PPO更新中使用float32精度，以及包含奖励白化、重要性比率防护和权重回滚的三层安全机制加以解决。本文提出容量余量假说，认为PPO在SLM尺度上的性能既依赖于流畅的监督模型（PPL<20），又依赖于具备判别能力的奖励信号，而非模型参数数量。所提系统在所有实验中均稳定收敛，且在具备流畅先验和有效奖励信号的配置中，相较于SFT基线提升了偏好胜率。此外，该系统在显著减少训练数据量的同时，性能优于指令微调基线。所有检查点、偏好数据集及训练脚本均已公开发布^§。

English

The alignment of Small Language Models (SLMs) in the 70--500M parameter range using reinforcement learning is often considered unstable, though the underlying failure mechanisms have not been systematically investigated. In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora. Three reproducible failure modes were identified in small-scale language models: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error. These issues were addressed using a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model (PPL<20) and a discriminative reward signal, rather than on the number of model parameters. The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal. Furthermore, it outperformed instruction-tuned baselines while requiring significantly less training data. All checkpoints, preference datasets, and training scripts are publicly released^§.