白川对齐技术报告

Baichuan Alignment Technical Report

October 19, 2024

作者: Mingan Lin, Fan Yang, Yanjun Shen, Haoze Sun, Tianpeng Li, Tao Zhang, Chenzheng Zhu, Tao Zhang, Miao Zheng, Xu Li, Yijie Zhou, Mingyang Chen, Yanzhao Qin, Youquan Li, Hao Liang, Fei Li, Yadong Li, Mang Wang, Guosheng Dong, Kun Fang, Jianhua Xu, Bin Cui, Wentao Zhang, Zenan Zhou, Weipeng Chen

cs.AI

摘要

我们介绍了Baichuan Alignment，对Baichuan系列模型中采用的对齐技术进行了详细分析。这代表了行业首次全面记录对齐方法论，为推进人工智能研究提供了宝贵的见解。我们研究了在对齐过程中增强模型性能的关键组成部分，包括优化方法、数据策略、能力增强和评估流程。该过程涵盖了三个关键阶段：Prompt增强系统（PAS）、监督微调（SFT）和偏好对齐。记录了所遇到的问题、应用的解决方案和所做的改进。 通过与成熟基准的比较，我们突出了Baichuan Alignment带来的技术进步。Baichuan-Instruct是一个内部模型，而Qwen2-Nova-72B和Llama3-PBM-Nova-70B是Qwen2-72B和Llama-3-70B基础模型的指导版本，通过Baichuan Alignment进行了优化。Baichuan-Instruct在核心能力方面取得了显著改进，用户体验提升在17%至28%之间，并在专门基准上表现出色。在开源基准评估中，无论是Qwen2-Nova-72B还是Llama3-PBM-Nova-70B，在几乎所有数据集上始终优于各自的官方指导版本。本报告旨在阐明对齐过程背后的关键技术，促进社区对此有更深入的理解。 Llama3-PBM-Nova-70B模型可在https://huggingface.co/PKU-Baichuan-MLSystemLab/Llama3-PBM-Nova-70B找到。

English

We introduce Baichuan Alignment, a detailed analysis of the alignment techniques employed in the Baichuan series of models. This represents the industry's first comprehensive account of alignment methodologies, offering valuable insights for advancing AI research. We investigate the critical components that enhance model performance during the alignment process, including optimization methods, data strategies, capability enhancements, and evaluation processes. The process spans three key stages: Prompt Augmentation System (PAS), Supervised Fine-Tuning (SFT), and Preference Alignment. The problems encountered, the solutions applied, and the improvements made are thoroughly recorded. Through comparisons across well-established benchmarks, we highlight the technological advancements enabled by Baichuan Alignment. Baichuan-Instruct is an internal model, while Qwen2-Nova-72B and Llama3-PBM-Nova-70B are instruct versions of the Qwen2-72B and Llama-3-70B base models, optimized through Baichuan Alignment. Baichuan-Instruct demonstrates significant improvements in core capabilities, with user experience gains ranging from 17% to 28%, and performs exceptionally well on specialized benchmarks. In open-source benchmark evaluations, both Qwen2-Nova-72B and Llama3-PBM-Nova-70B consistently outperform their respective official instruct versions across nearly all datasets. This report aims to clarify the key technologies behind the alignment process, fostering a deeper understanding within the community. Llama3-PBM-Nova-70B model is available at https://huggingface.co/PKU-Baichuan-MLSystemLab/Llama3-PBM-Nova-70B.