NeMo-Aligner：用于高效模型对齐的可扩展工具包

NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment

May 2, 2024

作者: Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, Oleksii Kuchaiev

cs.AI

摘要

将大型语言模型（LLMs）与人类价值观和偏好保持一致对于使其有益且安全至关重要。然而，构建有效的工具来执行对齐可能具有挑战性，特别是对于通常包含数百亿参数的最大和最具竞争力的LLMs。我们创建了NeMo-Aligner，这是一个用于模型对齐的工具包，可以高效地扩展到使用数百个GPU进行训练。NeMo-Aligner配备了针对模型对齐主要范式的高度优化和可扩展的实现，例如：从人类反馈中进行强化学习（RLHF）、直接偏好优化（DPO）、SteerLM和自我对弈微调（SPIN）。此外，我们的工具包支持在参数高效微调（PEFT）设置中运行大多数对齐技术。NeMo-Aligner设计用于可扩展性，允许以最小的努力支持其他对齐技术。它以Apache 2.0许可证开源，并欢迎社区在https://github.com/NVIDIA/NeMo-Aligner进行贡献。

English

Aligning Large Language Models (LLMs) with human values and preferences is essential for making them helpful and safe. However, building efficient tools to perform alignment can be challenging, especially for the largest and most competent LLMs which often contain tens or hundreds of billions of parameters. We create NeMo-Aligner, a toolkit for model alignment that can efficiently scale to using hundreds of GPUs for training. NeMo-Aligner comes with highly optimized and scalable implementations for major paradigms of model alignment such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally, our toolkit supports running most of the alignment techniques in a Parameter Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for extensibility, allowing support for other alignment techniques with minimal effort. It is open-sourced with Apache 2.0 License and we invite community contributions at https://github.com/NVIDIA/NeMo-Aligner