利用稀疏自编码器的模型内部机制指导大语言模型后训练数据工程

摘要

模型内部蕴含着丰富的信息，揭示了大型语言模型（LLM）如何处理其训练数据；然而，后训练阶段的数据工程主要依赖外部信号，忽视了模型内部蕴含的丰富内在信号。我们提出SAERL——一种面向LLM强化学习（RL）的数据工程框架。该框架利用稀疏自编码器（SAE）这一先进的机械可解释性工具提取的模型内部表征，建模三种内在数据属性：多样性、难度与质量。每种属性对应一项具体的数据工程操作：基于SAE空间聚类并结合适度批次混合以实现批次多样性控制；构建难度代理指标以支持由易到难的课程排序；以及设计质量探针用于数据过滤。在Qwen2.5-Math-1.5B模型上，SAERL相比原始GRPO方法平均准确率提升3.00%，并在达到目标准确率时减少20%的训练步数；该增益在多种模型规模与RL算法上保持一致。实验表明，SAE能够跨模型系列与规模有效迁移，成为轻量级且可复用的数据工程工具。这些结果证明，模型内部表征为后训练阶段的数据工程提供了强大且实用的信号源。

English

Model internals encode rich information about how a large language model (LLM) processes its training data; however, post-training data engineering largely relies on external signals and ignores rich intrinsic signals lying in model internals. We propose SAERL, a data engineering framework for LLM reinforcement learning (RL). It models three intrinsic data properties: diversity, difficulty, and quality, using model internals extracted with Sparse Autoencoder (SAE), an advanced mechanistic interpretability tool. Each property grounds a concrete data engineering operation: SAE-space clustering with moderate batch mixing for batch diversity control, a difficulty proxy for easy-to-hard curriculum ordering, and a quality probe for data filtering. SAERL improves average accuracy by 3.00% over vanilla GRPO and reaches target accuracy with 20% fewer training steps on Qwen2.5-Math-1.5B, with consistent gains across model scales and RL algorithms. Experiments show that SAE transfers effectively across model families and scales, serving as a lightweight and reusable data engineering tool. These results demonstrate that model internals are a powerful and practical source of signals for post-training data engineering.