教导预训练语言模型通过改进型循环机制实现更深层次思考

摘要

近期深度循环语言模型的研究表明，循环结构能够将训练时的计算量与参数量同测试时的计算需求解耦。本文探索了如何将现有预训练的非循环语言模型转化为深度循环模型。我们发现，通过采用渐进式循环训练课程，在训练过程中逐步增加模型的有效深度，可以在保持性能的同时降低总体计算成本。在数学领域的实验中，相较于直接对原始非循环语言模型进行后训练，将预训练模型转化为循环结构能在相同计算预算下获得更优的性能表现。

English

Recent advances in depth-recurrent language models show that recurrence can decouple train-time compute and parameter count from test-time compute. In this work, we study how to convert existing pretrained non-recurrent language models into depth-recurrent models. We find that using a curriculum of recurrences to increase the effective depth of the model over the course of training preserves performance while reducing total computational cost. In our experiments, on mathematics, we observe that converting pretrained models to recurrent ones results in better performance at a given compute budget than simply post-training the original non-recurrent language model.

教导预训练语言模型通过改进型循环机制实现更深层次思考

Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence

摘要

Support