分布偏移下忠实生成的Token级离策略学习

摘要

我们提出Token级离策略标注（TOPL），一种将后训练重新定义为token级正确性预测任务的离策略训练范式。其核心直觉在于：通过训练模型区分回复中的优质与劣质token，自然引导模型生成优质token，同时避开直接训练模型生成离策略token所伴随的缺陷。在文档摘要任务上的实验表明，TOPL在11个数据集上取得了强大的分布外泛化性能，优于一系列序列级和token级基线方法。我们进一步证明TOPL可有效迁移至机器翻译任务，表明其优势能泛化至不同的忠实性生成任务。通过消融研究，我们确认token级学习信号对优秀性能至关重要；序列级对应方法无法带来类似收益。最后，我们展示TOPL能产生可解释的模型更新：通过TOPL学习的LoRA适配器可充当线性分类头与引导向量。

English

We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task. Our key intuition is that by training the model to distinguish good and bad tokens in a response, we naturally guide the model towards generating good tokens, while avoiding the pitfalls that come with directly training the model to generate off-policy tokens. Experiments on document summarization tasks show that TOPL achieves strong out-of-distribution generalization across 11 datasets against a diverse set of sequence-level and token-level baselines. We further demonstrate that TOPL transfers effectively to machine translation, suggesting that its benefits generalize across different faithful generation tasks. Through ablation studies, we confirm that our token-level learning signal is critical to good performance; sequence-level analogues do not confer similar benefits. Finally, we show that TOPL induces interpretable model updates: the LoRA adapters learned through TOPL function as linear classification heads and steering vectors.