具有大型語言模型的上下文中意義模糊感知學習

摘要

在上下文學習（ICL）中，僅向LLM展示少量特定任務的示範即可實現下游收益，而無需進行特定任務的微調。然而，LLM對提示的選擇敏感，因此一個關鍵的研究問題是如何為ICL選擇良好的示範。一種有效的策略是利用ICL示範與測試輸入之間的語義相似性，使用文本檢索器，然而這種方法並不理想，因為它並未考慮LLM對該任務的現有知識。根據先前的研究（Min等，2022），我們已經知道與示範配對的標籤會對模型預測產生偏見。這引出了我們的假設，即考慮LLM對任務的現有知識，特別是關於輸出標籤空間，是否有助於更好地選擇示範。通過對三個文本分類任務進行廣泛實驗，我們發現不僅選擇語義相似的ICL示範有益，還選擇那些有助於解決測試示例周圍固有標籤模糊性的示範。有趣的是，我們發現包括LLM先前錯誤分類並且也位於測試示例的決策邊界上的示範，帶來了最大的性能增益。

English

In-context learning (ICL) i.e. showing LLMs only a few task-specific demonstrations has led to downstream gains with no task-specific fine-tuning required. However, LLMs are sensitive to the choice of prompts, and therefore a crucial research question is how to select good demonstrations for ICL. One effective strategy is leveraging semantic similarity between the ICL demonstrations and test inputs by using a text retriever, which however is sub-optimal as that does not consider the LLM's existing knowledge about that task. From prior work (Min et al., 2022), we already know that labels paired with the demonstrations bias the model predictions. This leads us to our hypothesis whether considering LLM's existing knowledge about the task, especially with respect to the output label space can help in a better demonstration selection strategy. Through extensive experimentation on three text classification tasks, we find that it is beneficial to not only choose semantically similar ICL demonstrations but also to choose those demonstrations that help resolve the inherent label ambiguity surrounding the test example. Interestingly, we find that including demonstrations that the LLM previously mis-classified and also fall on the test example's decision boundary, brings the most performance gain.

具有大型語言模型的上下文中意義模糊感知學習

Ambiguity-Aware In-Context Learning with Large Language Models

摘要

Support