通过蒙特卡洛树搜索实现LLM自我改进：利用逐步知识与课程偏好学习。

Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learning

October 9, 2024

作者: Xiyao Wang, Linfeng Song, Ye Tian, Dian Yu, Baolin Peng, Haitao Mi, Furong Huang, Dong Yu

cs.AI

摘要

蒙特卡洛树搜索（MCTS）最近被证明是增强大型语言模型（LLMs）推理能力的强大技术。诸如SFT或DPO等技术使LLMs能够从MCTS中提炼出高质量行为，提升其推理性能。然而，现有的提炼方法未充分利用MCTS生成的丰富轨迹信息，限制了LLMs推理改进的潜力。本文提出了AlphaLLM-CPL，这是一种新颖的成对训练框架，使LLMs能够通过MCTS行为提炼自我改进。AlphaLLM-CPL通过两个关键创新有效地利用MCTS轨迹：（1）AlphaLLM-CPL从搜索树中共享相同父节点的子节点构建逐步轨迹对，为更有效的MCTS行为提炼提供步级信息。 （2）AlphaLLM-CPL引入课程偏好学习，动态调整每个离线训练时期中轨迹对的训练顺序，以优先考虑关键学习步骤并减少过拟合。在数学推理任务上的实验结果表明，AlphaLLM-CPL明显优于先前的MCTS行为提炼方法，大幅提升了LLMs的推理能力。

English

Monte Carlo Tree Search (MCTS) has recently emerged as a powerful technique for enhancing the reasoning capabilities of LLMs. Techniques such as SFT or DPO have enabled LLMs to distill high-quality behaviors from MCTS, improving their reasoning performance. However, existing distillation methods underutilize the rich trajectory information generated by MCTS, limiting the potential for improvements in LLM reasoning. In this paper, we propose AlphaLLM-CPL, a novel pairwise training framework that enables LLMs to self-improve through MCTS behavior distillation. AlphaLLM-CPL efficiently leverages MCTS trajectories via two key innovations: (1) AlphaLLM-CPL constructs stepwise trajectory pairs from child nodes sharing the same parent in the search tree, providing step-level information for more effective MCTS behavior distillation. (2) AlphaLLM-CPL introduces curriculum preference learning, dynamically adjusting the training sequence of trajectory pairs in each offline training epoch to prioritize critical learning steps and mitigate overfitting. Experimental results on mathematical reasoning tasks demonstrate that AlphaLLM-CPL significantly outperforms previous MCTS behavior distillation methods, substantially boosting the reasoning capabilities of LLMs.