PLaD：基于假想偏好对的偏好驱动大型语言模型蒸馏

PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs

June 5, 2024

作者: Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Haorui Wang, Zhen Qin, Feng Han, Jialu Liu, Simon Baumgartner, Michael Bendersky, Chao Zhang

cs.AI

摘要

大型语言模型（LLMs）在各种任务中展现出令人印象深刻的能力，然而其庞大的参数规模限制了它们在资源受限环境中的适用性。知识蒸馏（KD）通过将大型教师模型的专业知识转移给紧凑的学生模型，提供了一种可行的解决方案。然而，传统的知识蒸馏技术在应用于LLMs时面临特定挑战，包括对LLM输出的访问受限、显著的师生容量差距以及继承的误校准问题。在这项工作中，我们提出了PLaD，一种新颖的基于偏好的LLM蒸馏框架。PLaD利用师生容量差异生成伪偏好对，其中师生输出中更倾向于师傅输出。然后，PLaD利用排名损失重新校准学生对序列可能性的估计，引导学生的注意力关注输出的相对质量，而不仅仅是模仿老师。PLaD避免了需要访问教师LLM内部状态的需求，解决了学生表达能力的限制，并缓解了学生的误校准问题。通过在两个序列生成任务上对各种LLMs进行大量实验，我们展示了我们提出的PLaD框架的有效性。

English

Large Language Models (LLMs) have exhibited impressive capabilities in various tasks, yet their vast parameter sizes restrict their applicability in resource-constrained settings. Knowledge distillation (KD) offers a viable solution by transferring expertise from large teacher models to compact student models. However, traditional KD techniques face specific challenges when applied to LLMs, including restricted access to LLM outputs, significant teacher-student capacity gaps, and the inherited mis-calibration issue. In this work, we present PLaD, a novel preference-based LLM distillation framework. PLaD exploits the teacher-student capacity discrepancy to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. Then, PLaD leverages a ranking loss to re-calibrate student's estimation of sequence likelihood, which steers the student's focus towards understanding the relative quality of outputs instead of simply imitating the teacher. PLaD bypasses the need for access to teacher LLM's internal states, tackles the student's expressivity limitations, and mitigates the student mis-calibration issue. Through extensive experiments on two sequence generation tasks and with various LLMs, we demonstrate the effectiveness of our proposed PLaD framework.