推理語言模型：藍圖

Reasoning Language Models: A Blueprint

January 20, 2025

作者: Maciej Besta, Julia Barth, Eric Schreiber, Ales Kubicek, Afonso Catarino, Robert Gerstenberger, Piotr Nyczyk, Patrick Iff, Yueling Li, Sam Houliston, Tomasz Sternal, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, Łukasz Flis, Hannes Eberhard, Hubert Niewiadomski, Torsten Hoefler

cs.AI

摘要

推理語言模型（RLMs），又稱為大型推理模型（LRMs），如OpenAI的o1和o3、DeepSeek-V3以及阿里巴巴的QwQ，通過擴展大型語言模型（LLMs）並加入先進的推理機制，重新定義了人工智慧的問題解決能力。然而，它們高昂的成本、專有性質和複雜的架構 - 獨特地結合了強化學習（RL）、搜索啟發式和LLMs - 提出了可及性和可擴展性挑戰。為了應對這些挑戰，我們提出了一項全面的藍圖，將RLM組件組織成模塊化框架，基於對所有RLM作品的調查和分析。該藍圖納入了多樣的推理結構（鏈、樹、圖和嵌套形式）、推理策略（例如蒙特卡羅樹搜索、束搜索）、RL概念（策略、價值模型等）和監督方案（基於輸出和基於過程的監督）。我們還提供了詳細的數學公式和算法規範，以簡化RLM的實施。通過展示像LLaMA-Berry、QwQ、Journey Learning和Graph of Thoughts這樣的方案如何適用作為特殊情況，我們展示了藍圖的多功能性和統一潛力。為了說明其實用性，我們引入了x1，這是一個用於快速RLM原型設計和實驗的模塊化實現。利用x1和文獻回顧，我們提供了關鍵見解，例如對策略和價值模型進行多階段訓練的重要性，以及熟悉訓練分佈的重要性。最後，我們概述了RLMs如何與更廣泛的LLM生態系統集成，包括工具和數據庫。我們的工作揭開了RLM構建的神秘面紗，使先進的推理能力民主化，並促進創新，旨在通過降低RLM開發和實驗的障礙，緩解“富裕AI”和“貧窮AI”之間的差距。

English

Reasoning language models (RLMs), also known as Large Reasoning Models (LRMs), such as OpenAI's o1 and o3, DeepSeek-V3, and Alibaba's QwQ, have redefined AI's problem-solving capabilities by extending large language models (LLMs) with advanced reasoning mechanisms. Yet, their high costs, proprietary nature, and complex architectures - uniquely combining Reinforcement Learning (RL), search heuristics, and LLMs - present accessibility and scalability challenges. To address these, we propose a comprehensive blueprint that organizes RLM components into a modular framework, based on a survey and analysis of all RLM works. This blueprint incorporates diverse reasoning structures (chains, trees, graphs, and nested forms), reasoning strategies (e.g., Monte Carlo Tree Search, Beam Search), RL concepts (policy, value models and others), and supervision schemes (Output-Based and Process-Based Supervision). We also provide detailed mathematical formulations and algorithmic specifications to simplify RLM implementation. By showing how schemes like LLaMA-Berry, QwQ, Journey Learning, and Graph of Thoughts fit as special cases, we demonstrate the blueprint's versatility and unifying potential. To illustrate its utility, we introduce x1, a modular implementation for rapid RLM prototyping and experimentation. Using x1 and a literature review, we provide key insights, such as multi-phase training for policy and value models, and the importance of familiar training distributions. Finally, we outline how RLMs can integrate with a broader LLM ecosystem, including tools and databases. Our work demystifies RLM construction, democratizes advanced reasoning capabilities, and fosters innovation, aiming to mitigate the gap between "rich AI" and "poor AI" by lowering barriers to RLM development and experimentation.