大型语言模型重新评分的低秩调整，用于参数高效的语音识别

摘要

我们提出了一种基于低秩适应（LoRA）的神经语言建模系统，用于语音识别输出重评分。尽管像BERT这样的预训练语言模型在二次重评分中表现出优越性能，但是扩展预训练阶段的高计算成本以及将预训练模型调整到特定领域的限制限制了它们在重评分中的实际应用。在这里，我们提出了一种基于低秩分解的方法，用于训练一个重评分的BERT模型，并仅使用预训练参数的一小部分（0.08%）来将其调整到新领域。这些插入的矩阵通过一个判别式训练目标以及基于相关性的正则化损失进行优化。提出的低秩适应Rescore-BERT（LoRB）架构在LibriSpeech和内部数据集上进行了评估，训练时间缩短了5.4到3.6倍。

English

We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained language models (LMs) like BERT have shown superior performance in second-pass rescoring, the high computational cost of scaling up the pretraining stage and adapting the pretrained models to specific domains limit their practical use in rescoring. Here we present a method based on low-rank decomposition to train a rescoring BERT model and adapt it to new domains using only a fraction (0.08%) of the pretrained parameters. These inserted matrices are optimized through a discriminative training objective along with a correlation-based regularization loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is evaluated on LibriSpeech and internal datasets with decreased training times by factors between 5.4 and 3.6.

大型语言模型重新评分的低秩调整，用于参数高效的语音识别

Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition

摘要

Support