Transformer Copilot：基於LLM微調中的錯誤日誌進行學習

Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuning

May 22, 2025

作者: Jiaru Zou, Yikun Ban, Zihao Li, Yunzhe Qi, Ruizhong Qiu, Ling Yang, Jingrui He

cs.AI

摘要

大型語言模型通常通過對特定領域數據進行監督式微調來適應下游任務。雖然標準的微調主要聚焦於最小化生成損失以優化模型參數，但我們更進一步，保留並利用模型自身的學習信號，這類似於人類學習者通過反思過去的錯誤來提升未來表現。我們首先引入了「錯誤日誌」的概念，以系統地追蹤模型在微調過程中的學習行為和重複出現的錯誤。將原始的基於Transformer的模型視為「主駕駛」，我們相應地設計了一個「副駕駛」模型，通過對數概率的校正來提升主駕駛的推理性能。我們將這一整體的主副駕駛框架命名為「Transformer副駕駛」，它引入了：(i) 一種新穎的副駕駛模型設計，(ii) 一種聯合訓練範式，其中副駕駛持續從不斷演化的錯誤日誌中學習，與主駕駛並行，(iii) 一種融合推理範式，其中副駕駛校正主駕駛的對數概率以增強生成效果。我們對這一新的學習框架進行了理論和實證分析。在涵蓋常識、算術和推薦任務的12個基準測試中，實驗表明Transformer副駕駛持續提升了性能，最高可達34.5%，同時僅為主駕駛模型引入了微小的計算開銷，並展現出強大的可擴展性和可遷移性。

English

Large language models are typically adapted to downstream tasks through supervised fine-tuning on domain-specific data. While standard fine-tuning focuses on minimizing generation loss to optimize model parameters, we take a deeper step by retaining and leveraging the model's own learning signals, analogous to how human learners reflect on past mistakes to improve future performance. We first introduce the concept of Mistake Log to systematically track the model's learning behavior and recurring errors throughout fine-tuning. Treating the original transformer-based model as the Pilot, we correspondingly design a Copilot model to refine the Pilot's inference performance via logits rectification. We name the overall Pilot-Copilot framework the Transformer Copilot, which introduces (i) a novel Copilot model design, (ii) a joint training paradigm where the Copilot continuously learns from the evolving Mistake Log alongside the Pilot, and (iii) a fused inference paradigm where the Copilot rectifies the Pilot's logits for enhanced generation. We provide both theoretical and empirical analyses on our new learning framework. Experiments on 12 benchmarks spanning commonsense, arithmetic, and recommendation tasks demonstrate that Transformer Copilot consistently improves performance by up to 34.5%, while introducing marginal computational overhead to Pilot models and exhibiting strong scalability and transferability.