LLM能透過教學來學習嗎？初步研究

Can LLMs Learn by Teaching? A Preliminary Study

June 20, 2024

作者: Xuefei Ning, Zifu Wang, Shiyao Li, Zinan Lin, Peiran Yao, Tianyu Fu, Matthew B. Blaschko, Guohao Dai, Huazhong Yang, Yu Wang

cs.AI

摘要

教學以改進學生模型（例如知識蒸餾）是LLM中被廣泛研究的方法論。然而，對於人類而言，教學不僅改進了學生，也改進了教師。我們提出：LLM是否也可以透過教學來學習（LbT）？如果是，我們可能可以解鎖在不僅依賴人類產生的數據或更強大模型的情況下持續推進模型的可能性。在本文中，我們對這一雄心勃勃的議程進行了初步探索。我們展示了LbT理念可以融入現有的LLM訓練/提示流程中並帶來顯著改進。具體而言，我們設計了三種方法，分別模仿人類LbT的三個層次：觀察學生的反饋、從反饋中學習以及迭代學習，旨在提高答案準確性而無需訓練，並通過微調提高模型的固有能力。研究結果令人鼓舞。例如，類似於人類的LbT，我們發現：（1）LbT可以引發從弱到強的泛化：強大模型可以通過教導其他弱模型來改進自身；（2）學生的多樣性可能有助於：教授多個學生可能比教授一個學生或教師本身更好。我們希望這種初步的潛力能激發對LbT的未來研究以及更廣泛地採納教育中的先進技術來改進LLM。代碼可在https://github.com/imagination-research/lbt找到。

English

Teaching to improve student models (e.g., knowledge distillation) is an extensively studied methodology in LLMs. However, for humans, teaching not only improves students but also improves teachers. We ask: Can LLMs also learn by teaching (LbT)? If yes, we can potentially unlock the possibility of continuously advancing the models without solely relying on human-produced data or stronger models. In this paper, we provide a preliminary exploration of this ambitious agenda. We show that LbT ideas can be incorporated into existing LLM training/prompting pipelines and provide noticeable improvements. Specifically, we design three methods, each mimicking one of the three levels of LbT in humans: observing students' feedback, learning from the feedback, and learning iteratively, with the goals of improving answer accuracy without training and improving models' inherent capability with fine-tuning. The findings are encouraging. For example, similar to LbT in human, we see that: (1) LbT can induce weak-to-strong generalization: strong models can improve themselves by teaching other weak models; (2) Diversity in students might help: teaching multiple students could be better than teaching one student or the teacher itself. We hope that this early promise can inspire future research on LbT and more broadly adopting the advanced techniques in education to improve LLMs. The code is available at https://github.com/imagination-research/lbt.