具有大型语言模型的上下文中的歧义感知学习

摘要

在上下文学习（ICL）中，即仅向LLM展示少量特定任务的演示，已经导致下游收益，而无需进行特定任务的微调。然而，LLM对提示的选择很敏感，因此一个关键的研究问题是如何为ICL选择好的演示。一种有效的策略是利用ICL演示和测试输入之间的语义相似性，通过使用文本检索器，然而这种方法并不理想，因为它并未考虑LLM对该任务的现有知识。根据之前的研究（Min等，2022），我们已经知道与演示配对的标签会影响模型的预测。这引出了我们的假设，即考虑LLM对任务的现有知识，特别是关于输出标签空间，是否可以帮助改进演示选择策略。通过对三个文本分类任务进行广泛实验，我们发现，不仅选择语义相似的ICL演示有益，还选择那些有助于解决围绕测试示例存在的内在标签模糊性的演示也是有益的。有趣的是，我们发现包括LLM先前误分类的演示，并且也落在测试示例的决策边界上，带来了最大的性能提升。

English

In-context learning (ICL) i.e. showing LLMs only a few task-specific demonstrations has led to downstream gains with no task-specific fine-tuning required. However, LLMs are sensitive to the choice of prompts, and therefore a crucial research question is how to select good demonstrations for ICL. One effective strategy is leveraging semantic similarity between the ICL demonstrations and test inputs by using a text retriever, which however is sub-optimal as that does not consider the LLM's existing knowledge about that task. From prior work (Min et al., 2022), we already know that labels paired with the demonstrations bias the model predictions. This leads us to our hypothesis whether considering LLM's existing knowledge about the task, especially with respect to the output label space can help in a better demonstration selection strategy. Through extensive experimentation on three text classification tasks, we find that it is beneficial to not only choose semantically similar ICL demonstrations but also to choose those demonstrations that help resolve the inherent label ambiguity surrounding the test example. Interestingly, we find that including demonstrations that the LLM previously mis-classified and also fall on the test example's decision boundary, brings the most performance gain.

具有大型语言模型的上下文中的歧义感知学习

Ambiguity-Aware In-Context Learning with Large Language Models

摘要

Support