動態重要樣本挖掘於強化微調之研究

摘要

強化微調（RFT）日益被用於強化大型模型的推理能力，然而其有效性受制於訓練資料的選擇與使用方式。多數以資料為中心的RFT方法依賴靜態或啟發式樣本選擇，隱含地假設樣本的價值在訓練過程中是固定的。這忽略了策略學習的非平穩動態，並可能導致次優的更新。我們提出動態重要樣本挖掘（DIEM），一個具有原則性且全自動的框架，使資料利用在RFT過程中具備適應性。DIEM在每個最佳化步驟中整合兩個組件：(i) 梯度對齊重要性估計器，有效近似每個樣本對策略改進的邊際貢獻；(ii) 約束批次重新加權機制，在最大化總體效用的同時保持更新的梯度幅度以穩定最佳化。在多個推理基準上，DIEM持續優於強勁的靜態與動態基線。程式碼將透過 https://github.com/hrtan/DIEM 發布。

English

Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and used. Most data-centric RFT methods rely on static or heuristic sample selection, implicitly assuming a sample's value is fixed over training. This overlooks the non-stationary dynamics of policy learning and can lead to suboptimal updates. We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT. DIEM integrates two components into each optimization step: (i) a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement; and (ii) a constrained batch reweighting scheme that maximizes aggregate utility while preserving the update's gradient magnitude to stabilize optimization. Across several reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines. The code will be released via https://github.com/hrtan/DIEM.