迈向大型语言模型偏好学习的统一视角：综述

摘要

大型语言模型（LLMs）展现出非常强大的能力。实现成功的关键因素之一是将LLM的输出与人类偏好对齐。这种对齐过程通常只需要少量数据就能有效地提升LLM的性能。虽然有效，但这一领域的研究涉及多个领域，所涉及的方法相对复杂难以理解。不同方法之间的关系尚未得到充分探讨，限制了偏好对齐的发展。鉴此，我们将现有流行的对齐策略分解为不同组成部分，并提供一个统一框架来研究当前的对齐策略，从而建立它们之间的联系。在这项调查中，我们将所有偏好学习策略分解为四个组成部分：模型、数据、反馈和算法。这种统一视角提供了对现有对齐算法的深入理解，同时也为协同不同策略的优势打开了可能性。此外，我们提供了流行现有算法的详细工作示例，以便读者全面了解。最后，基于我们的统一视角，我们探讨了将大型语言模型与人类偏好对齐的挑战和未来研究方向。

English

Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of the crucial factors to achieve success is aligning the LLM's output with human preferences. This alignment process often requires only a small amount of data to efficiently enhance the LLM's performance. While effective, research in this area spans multiple domains, and the methods involved are relatively complex to understand. The relationships between different methods have been under-explored, limiting the development of the preference alignment. In light of this, we break down the existing popular alignment strategies into different components and provide a unified framework to study the current alignment strategies, thereby establishing connections among them. In this survey, we decompose all the strategies in preference learning into four components: model, data, feedback, and algorithm. This unified view offers an in-depth understanding of existing alignment algorithms and also opens up possibilities to synergize the strengths of different strategies. Furthermore, we present detailed working examples of prevalent existing algorithms to facilitate a comprehensive understanding for the readers. Finally, based on our unified perspective, we explore the challenges and future research directions for aligning large language models with human preferences.

迈向大型语言模型偏好学习的统一视角：综述

Towards a Unified View of Preference Learning for Large Language Models: A Survey

摘要

Summary

Support

Support