Transformer Copilot：基于大语言模型微调中的错误日志学习

Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuning

May 22, 2025

作者: Jiaru Zou, Yikun Ban, Zihao Li, Yunzhe Qi, Ruizhong Qiu, Ling Yang, Jingrui He

cs.AI

摘要

大型语言模型通常通过在特定领域数据上进行监督微调来适应下游任务。标准的微调方法主要关注最小化生成损失以优化模型参数，而我们则更进一步，保留并利用模型自身的学习信号，这类似于人类学习者通过反思过往错误来提升未来表现。我们首先引入了“错误日志”这一概念，以系统性地追踪模型在微调过程中的学习行为及重复错误。将原始的基于Transformer的模型视为“领航员”，我们相应地设计了一个“副驾驶”模型，通过修正logits来优化领航员的推理性能。我们将这一整体框架命名为“Transformer副驾驶”，它包含三个创新点：(i) 一种新颖的副驾驶模型设计，(ii) 一种联合训练范式，其中副驾驶持续从不断演变的错误日志中学习，与领航员同步进步，以及(iii) 一种融合推理范式，副驾驶通过修正领航员的logits来增强生成效果。我们为这一新学习框架提供了理论与实证分析。在涵盖常识、算术及推荐任务的12个基准测试中，实验表明Transformer副驾驶能持续提升性能，最高可达34.5%，同时仅对领航员模型引入边际计算开销，并展现出良好的可扩展性与迁移能力。

English

Large language models are typically adapted to downstream tasks through supervised fine-tuning on domain-specific data. While standard fine-tuning focuses on minimizing generation loss to optimize model parameters, we take a deeper step by retaining and leveraging the model's own learning signals, analogous to how human learners reflect on past mistakes to improve future performance. We first introduce the concept of Mistake Log to systematically track the model's learning behavior and recurring errors throughout fine-tuning. Treating the original transformer-based model as the Pilot, we correspondingly design a Copilot model to refine the Pilot's inference performance via logits rectification. We name the overall Pilot-Copilot framework the Transformer Copilot, which introduces (i) a novel Copilot model design, (ii) a joint training paradigm where the Copilot continuously learns from the evolving Mistake Log alongside the Pilot, and (iii) a fused inference paradigm where the Copilot rectifies the Pilot's logits for enhanced generation. We provide both theoretical and empirical analyses on our new learning framework. Experiments on 12 benchmarks spanning commonsense, arithmetic, and recommendation tasks demonstrate that Transformer Copilot consistently improves performance by up to 34.5%, while introducing marginal computational overhead to Pilot models and exhibiting strong scalability and transferability.