OpenRLHF：一個易於使用、可擴展且高性能的RLHF框架

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

May 20, 2024

作者: Jian Hu, Xibin Wu, Weixun Wang, Xianyu, Dehao Zhang, Yu Cao

cs.AI

摘要

隨著大型語言模型（LLMs）不斷按照規模定律增長，從人類反饋中進行強化學習（RLHF）因其出色的表現而受到重視。然而，與單個模型的預訓練或微調不同，對於訓練大型語言模型來說，對從人類反饋中進行強化學習（RLHF）進行規模化存在著跨越四個模型的協調挑戰。我們提出了OpenRLHF，這是一個開源框架，可以實現有效的RLHF規模化。與現有的RLHF框架不同，這些框架將四個模型放置在同一個GPU上，OpenRLHF通過使用Ray、vLLM和DeepSpeed重新設計模型的排程，從而克服了超過70B參數的挑戰，並利用了改進的資源利用率和多樣的訓練方法。OpenRLHF與Hugging Face無縫集成，提供了一個開箱即用的解決方案，具有優化的算法和啟動腳本，確保了用戶友好性。OpenRLHF實現了RLHF、DPO、拒絕採樣和其他對齊技術。作為最先進的LLM開發工具，OpenRLHF的代碼可在https://github.com/OpenLLMAI/OpenRLHF 上找到。

English

As large language models (LLMs) continue to grow by scaling laws, reinforcement learning from human feedback (RLHF) has gained significant attention due to its outstanding performance. However, unlike pretraining or fine-tuning a single model, scaling reinforcement learning from human feedback (RLHF) for training large language models poses coordination challenges across four models. We present OpenRLHF, an open-source framework enabling efficient RLHF scaling. Unlike existing RLHF frameworks that co-locate four models on the same GPUs, OpenRLHF re-designs scheduling for the models beyond 70B parameters using Ray, vLLM, and DeepSpeed, leveraging improved resource utilization and diverse training approaches. Integrating seamlessly with Hugging Face, OpenRLHF provides an out-of-the-box solution with optimized algorithms and launch scripts, which ensures user-friendliness. OpenRLHF implements RLHF, DPO, rejection sampling, and other alignment techniques. Empowering state-of-the-art LLM development, OpenRLHF's code is available at https://github.com/OpenLLMAI/OpenRLHF.