根据学生偏好调整教师以定制训练数据生成

Aligning Teacher with Student Preferences for Tailored Training Data Generation

June 27, 2024

作者: Yantao Liu, Zhao Zhang, Zijun Yao, Shulin Cao, Lei Hou, Juanzi Li

cs.AI

摘要

大型语言模型（LLMs）已显示出在各种任务中作为副驾驶员的显著潜力。在处理涉及隐私数据或延迟敏感任务时，LLMs在边缘设备上的本地部署是必要的。这类设备的计算约束使得直接部署强大的大规模LLMs变得不切实际，因此需要从大规模模型到轻量级模型的知识蒸馏。已经开展了许多工作以从LLMs中获取多样性和高质量的训练样本，但很少关注根据学生偏好调整教师指导内容，类似于教学法中的“响应式教学”。因此，我们提出了ARTE，即Aligning TeacheR with StudenT PreferencEs，这是一个框架，用于将教师模型与学生偏好对齐，以生成定制的知识蒸馏训练样本。具体来说，我们从教师模型中获取草案问题和原理，然后利用学生在上下文学习中的表现作为代理收集这些问题和原理的学生偏好，最后将教师模型与学生偏好对齐。最后，我们使用对齐的教师模型重复第一步，为目标任务上的学生模型获取定制的训练样本。在学术基准测试上进行的大量实验表明，ARTE相对于从强大的LLMs中提炼的现有指导调整数据集具有优越性。此外，我们深入研究了ARTE的泛化能力，包括在推理能力方面对经过微调的学生模型和对齐的教师模型在跨任务和学生间生成定制训练数据的泛化。总之，我们的贡献在于提出了一个新颖的定制训练样本生成框架，展示了其在实验中的有效性，并调查了ARTE中学生和对齐教师模型的泛化能力。

English

Large Language Models (LLMs) have shown significant promise as copilots in various tasks. Local deployment of LLMs on edge devices is necessary when handling privacy-sensitive data or latency-sensitive tasks. The computational constraints of such devices make direct deployment of powerful large-scale LLMs impractical, necessitating the Knowledge Distillation from large-scale models to lightweight models. Lots of work has been done to elicit diversity and quality training examples from LLMs, but little attention has been paid to aligning teacher instructional content based on student preferences, akin to "responsive teaching" in pedagogy. Thus, we propose ARTE, dubbed Aligning TeacheR with StudenT PreferencEs, a framework that aligns the teacher model with student preferences to generate tailored training examples for Knowledge Distillation. Specifically, we elicit draft questions and rationales from the teacher model, then collect student preferences on these questions and rationales using students' performance with in-context learning as a proxy, and finally align the teacher model with student preferences. In the end, we repeat the first step with the aligned teacher model to elicit tailored training examples for the student model on the target task. Extensive experiments on academic benchmarks demonstrate the superiority of ARTE over existing instruction-tuning datasets distilled from powerful LLMs. Moreover, we thoroughly investigate the generalization of ARTE, including the generalization of fine-tuned student models in reasoning ability and the generalization of aligned teacher models to generate tailored training data across tasks and students. In summary, our contributions lie in proposing a novel framework for tailored training example generation, demonstrating its efficacy in experiments, and investigating the generalization of both student & aligned teacher models in ARTE.