分佈偏移下針對忠實生成的詞元層級離策略學習

摘要

我們提出了詞元級離策略標註（TOPL），這是一種將後訓練重新定義為詞元級正確性預測任務的離策略訓練範式。我們的核心直覺是，透過訓練模型區分回應中的好詞元與壞詞元，自然地引導模型生成好詞元，同時避免因直接訓練模型生成離策略詞元而產生的隱患。在文件摘要任務上的實驗表明，TOPL 在11個數據集上實現了強分佈外泛化，超越了多種序列級和詞元級基線方法。我們進一步證明，TOPL 可高效遷移至機器翻譯任務，表明其優勢能泛化至不同的忠實生成任務。透過消融實驗，我們確認詞元級學習信號對性能至關重要；序列級對應方法無法提供類似收益。最後，我們證明 TOPL 能產生可解釋的模型更新：透過 TOPL 學習的 LoRA 適配器可充當線性分類頭和轉向向量。

English

We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task. Our key intuition is that by training the model to distinguish good and bad tokens in a response, we naturally guide the model towards generating good tokens, while avoiding the pitfalls that come with directly training the model to generate off-policy tokens. Experiments on document summarization tasks show that TOPL achieves strong out-of-distribution generalization across 11 datasets against a diverse set of sequence-level and token-level baselines. We further demonstrate that TOPL transfers effectively to machine translation, suggesting that its benefits generalize across different faithful generation tasks. Through ablation studies, we confirm that our token-level learning signal is critical to good performance; sequence-level analogues do not confer similar benefits. Finally, we show that TOPL induces interpretable model updates: the LoRA adapters learned through TOPL function as linear classification heads and steering vectors.