面向强化微调的动态重要示例挖掘

摘要

强化微调（RFT）日益被用于增强大模型的推理能力，然而其有效性受限于训练数据的选择和使用方式。大多数以数据为中心的RFT方法依赖静态或启发式样本选择，隐含地假设样本的价值在训练过程中固定不变。这忽略了策略学习的非平稳动态，可能导致次优的更新。我们提出了动态重要样例挖掘（DIEM），一个原理性且全自动的框架，使数据利用在整个RFT过程中具有自适应性。DIEM在每一步优化中整合两个组成部分：（i）梯度对齐重要性估计器，高效近似每个样本对策略提升的边际贡献；（ii）约束批量重加权方案，在保持更新的梯度幅度以稳定优化的同时最大化总体效用。在多个推理基准上，DIEM始终优于强静态基线和动态基线。代码将通过 https://github.com/hrtan/DIEM 发布。

English

Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and used. Most data-centric RFT methods rely on static or heuristic sample selection, implicitly assuming a sample's value is fixed over training. This overlooks the non-stationary dynamics of policy learning and can lead to suboptimal updates. We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT. DIEM integrates two components into each optimization step: (i) a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement; and (ii) a constrained batch reweighting scheme that maximizes aggregate utility while preserving the update's gradient magnitude to stabilize optimization. Across several reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines. The code will be released via https://github.com/hrtan/DIEM.