**NeMo-Aligner：高效模型對齊的可擴充套件工具包**

摘要

對齊大型語言模型與人類價值觀及偏好，是使其具備輔助性與安全性的關鍵。然而建構高效能對齊工具面臨挑戰，尤其針對參數量達數百億甚至數千億級別的最大規模、最強效能模型。我們開發了NeMo-Aligner工具包，這套模型對齊解決方案能高效擴展至數百個GPU的訓練規模。該工具包針對主流模型對齊範式提供高度優化且可擴展的實現方案，包括：人類回饋強化學習、直接偏好優化、SteerLM技術以及自博弈微調。此外，我們的工具包支援在多數對齊技術中採用參數高效微調模式。NeMo-Aligner採用可擴展架構設計，能透過最小化開發成本支援其他對齊技術。本工具基於Apache 2.0開源協議開放原始碼，誠邀社群參與協作：https://github.com/NVIDIA/NeMo-Aligner

English

Aligning Large Language Models (LLMs) with human values and preferences is essential for making them helpful and safe. However, building efficient tools to perform alignment can be challenging, especially for the largest and most competent LLMs which often contain tens or hundreds of billions of parameters. We create NeMo-Aligner, a toolkit for model alignment that can efficiently scale to using hundreds of GPUs for training. NeMo-Aligner comes with highly optimized and scalable implementations for major paradigms of model alignment such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally, our toolkit supports running most of the alignment techniques in a Parameter Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for extensibility, allowing support for other alignment techniques with minimal effort. It is open-sourced with Apache 2.0 License and we invite community contributions at https://github.com/NVIDIA/NeMo-Aligner

NeMo-Aligner：高效模型對齊的可擴充套件工具包

NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment

摘要

Support