利用示範檢索進行元訓練，以實現高效的少樣本學習

Meta-training with Demonstration Retrieval for Efficient Few-shot Learning

June 30, 2023

作者: Aaron Mueller, Kanika Narang, Lambert Mathias, Qifan Wang, Hamed Firooz

cs.AI

摘要

大型語言模型在少樣本自然語言處理任務上展現出令人印象深刻的結果。然而，這些模型需要大量記憶體和計算資源。元訓練使人們能夠以通用領域和任務無關的方式利用較小的模型進行少樣本泛化；然而，僅靠這些方法得到的模型可能缺乏足夠的參數化或知識，無法快速適應各種任務。為了克服這個問題，我們提出了具有示範檢索的元訓練方法，其中我們使用密集的段落檢索器來檢索與每個示例相似語義的標記示範，以獲得更多樣化的監督。通過將外部知識與模型參數分開，我們可以使用元訓練來訓練參數高效的模型，在更多種類的任務上實現良好的泛化。我們從UnifiedQA和CrossFit構建了一個元訓練集，並提出了一個基於UnifiedQA任務的示範庫。據我們所知，我們的工作是首個將檢索與元訓練相結合，使用DPR模型檢索示範，並同時利用來自多個任務的示範，而不是從目標任務的訓練集中隨機抽樣示範。我們的方法在問答、自然語言推理和文本分類任務（包括SQuAD、QNLI和TREC）上優於各種針對性的參數高效和檢索增強的少樣本方法。我們的方法可以在單個GPU上快速進行元訓練和微調。

English

Large language models show impressive results on few-shot NLP tasks. However, these models are memory and computation-intensive. Meta-training allows one to leverage smaller models for few-shot generalization in a domain-general and task-agnostic manner; however, these methods alone results in models that may not have sufficient parameterization or knowledge to adapt quickly to a large variety of tasks. To overcome this issue, we propose meta-training with demonstration retrieval, where we use a dense passage retriever to retrieve semantically similar labeled demonstrations to each example for more varied supervision. By separating external knowledge from model parameters, we can use meta-training to train parameter-efficient models that generalize well on a larger variety of tasks. We construct a meta-training set from UnifiedQA and CrossFit, and propose a demonstration bank based on UnifiedQA tasks. To our knowledge, our work is the first to combine retrieval with meta-training, to use DPR models to retrieve demonstrations, and to leverage demonstrations from many tasks simultaneously, rather than randomly sampling demonstrations from the training set of the target task. Our approach outperforms a variety of targeted parameter-efficient and retrieval-augmented few-shot methods on QA, NLI, and text classification tasks (including SQuAD, QNLI, and TREC). Our approach can be meta-trained and fine-tuned quickly on a single GPU.