利用稀疏自編碼器的模型內部機制引導大型語言模型後訓練數據工程

摘要

模型內部狀態編碼了大型語言模型（LLM）如何處理其訓練數據的豐富資訊；然而，訓練後的數據工程主要依賴外部信號，忽略了模型內部狀態中蘊藏的豐富內在信號。我們提出SAERL，一個專為LLM強化學習（RL）設計的數據工程框架。該框架利用稀疏自編碼器（SAE）——一種先進的機制可解釋性工具——從模型內部狀態中提取三種內在數據屬性：多樣性、難度與品質。每個屬性都對應具體的數據工程操作：基於SAE空間的聚類搭配適度批次混合以控制批次多樣性、用於由易到難課程排序的難度代理，以及用於數據篩選的品質探針。相較於原始GRPO，SAERL平均準確率提升3.00%，並在Qwen2.5-Math-1.5B模型上以減少20%的訓練步數達到目標準確率，且在不同模型規模與RL演算法中均展現一致的增益。實驗表明，SAE能有效地跨模型家族與規模遷移，成為一個輕量且可重複使用的數據工程工具。這些結果證明，模型內部狀態是訓練後數據工程中強大且實用的信號來源。

English

Model internals encode rich information about how a large language model (LLM) processes its training data; however, post-training data engineering largely relies on external signals and ignores rich intrinsic signals lying in model internals. We propose SAERL, a data engineering framework for LLM reinforcement learning (RL). It models three intrinsic data properties: diversity, difficulty, and quality, using model internals extracted with Sparse Autoencoder (SAE), an advanced mechanistic interpretability tool. Each property grounds a concrete data engineering operation: SAE-space clustering with moderate batch mixing for batch diversity control, a difficulty proxy for easy-to-hard curriculum ordering, and a quality probe for data filtering. SAERL improves average accuracy by 3.00% over vanilla GRPO and reaches target accuracy with 20% fewer training steps on Qwen2.5-Math-1.5B, with consistent gains across model scales and RL algorithms. Experiments show that SAE transfers effectively across model families and scales, serving as a lightweight and reusable data engineering tool. These results demonstrate that model internals are a powerful and practical source of signals for post-training data engineering.