在奖励模型、参数更新和上下文提示之间的转换中

On the Transformations across Reward Model, Parameter Update, and In-Context Prompt

June 24, 2024

作者: Deng Cai, Huayang Li, Tingchen Fu, Siheng Li, Weiwen Xu, Shuaiyi Li, Bowen Cao, Zhisong Zhang, Xinting Huang, Leyang Cui, Yan Wang, Lemao Liu, Taro Watanabe, Shuming Shi

cs.AI

摘要

尽管预训练大型语言模型（LLMs）具有一般的能力，但它们仍需要进一步适应以更好地服务于实际应用。在本文中，我们展示了三种流行且独特的适应工具：参数更新、奖励建模和上下文提示的互换性。这种互换性建立了一个三角形框架，具有六个转换方向，每个方向都促进了各种应用。我们的工作提供了一个统一众多现有研究的整体视角，并提出了潜在的研究方向。我们设想我们的工作将成为未来LLMs研究的有用路线图。

English

Despite the general capabilities of pre-trained large language models (LLMs), they still need further adaptation to better serve practical applications. In this paper, we demonstrate the interchangeability of three popular and distinct adaptation tools: parameter updating, reward modeling, and in-context prompting. This interchangeability establishes a triangular framework with six transformation directions, each of which facilitates a variety of applications. Our work offers a holistic view that unifies numerous existing studies and suggests potential research directions. We envision our work as a useful roadmap for future research on LLMs.