技能上下文提示：解锁大型语言模型中的组合性

Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models

August 1, 2023

作者: Jiaao Chen, Xiaoman Pan, Dian Yu, Kaiqiang Song, Xiaoyang Wang, Dong Yu, Jianshu Chen

cs.AI

摘要

我们考虑如何通过一种新颖的提示策略，在大型语言模型（LLMs）中引发组合泛化能力的问题。组合泛化赋予LLMs解决比它们所见过的更难的问题的能力（即易到难的泛化），这是类似人类智能的关键推理能力。然而，即使是当前最先进的LLMs仍然在这种推理形式上遡步难行。为了弥合这一差距，我们提出了上下文技能（SKiC）提示，指导LLMs如何组合基本技能来解决更复杂的问题。我们发现，在同一提示上下文中展示技能和组合示例是至关重要的。通过仅有两个示例，我们的SKiC提示激发了技能和它们的组合能力之间的强大协同作用。值得注意的是，它赋予LLMs解决需要创新技能组合的未见问题的能力，在各种具有挑战性的组合任务上实现了近乎完美的泛化。有趣的是，SKiC提示释放了LLMs的潜在潜力，使它们能够利用在早期预训练阶段获得的内部技能，即使这些技能在提示上下文中没有明确呈现。这导致LLMs能够通过激活和组合内部能力来解决未见的复杂问题。凭借这些显著特点，SKiC提示能够在具有挑战性的数学推理基准测试（例如MATH）上取得最先进的性能。

English

We consider the problem of eliciting compositional generalization capabilities in large language models (LLMs) with a novel type of prompting strategy. Compositional generalization empowers the LLMs to solve problems that are harder than the ones they have seen (i.e., easy-to-hard generalization), which is a critical reasoning capability of human-like intelligence. However, even the current state-of-the-art LLMs still struggle with this form of reasoning. To bridge this gap, we propose skills-in-context (SKiC) prompting, which instructs LLMs how to compose basic skills to resolve more complex problems. We find that it is crucial to demonstrate both the skills and the compositional examples within the same prompting context. With as few as two examplars, our SKiC prompting initiates strong synergies between skills and their composition capabilities. Notably, it empowers LLMs to solve unseen problems that require innovative skill compositions, achieving near-perfect generalization on a broad range of challenging compositionality tasks. Intriguingly, SKiC prompting unlocks the latent potential of LLMs, enabling them to leverage pre-existing internal skills acquired during earlier pre-training stages, even when these skills are not explicitly presented in the prompting context. This results in the capability of LLMs to solve unseen complex problems by activating and composing internal competencies. With such prominent features, SKiC prompting is able to achieve state-of-the-art performance on challenging mathematical reasoning benchmarks (e.g., MATH).