Tool-Star：通过强化学习赋能具备多工具推理能力的大型语言模型

Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning

May 22, 2025

作者: Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, Ji-Rong Wen

cs.AI

摘要

近期，大规模语言模型（LLMs）通过大规模强化学习（RL）展现了卓越的推理能力。然而，如何利用RL算法来增强LLMs在多工具协作推理中的有效性，仍是一个待解决的挑战。本文提出了Tool-Star，一个基于RL的框架，旨在赋能LLMs在逐步推理过程中自主调用多种外部工具。Tool-Star整合了六类工具，并在数据合成与训练中融入了系统性设计。针对工具使用数据稀缺的问题，我们提出了一种通用的工具集成推理数据合成流程，该流程结合了工具集成提示与基于提示的采样，以自动且可扩展地生成工具使用轨迹。随后，通过质量归一化与难度感知分类过程，过滤低质量样本，并将数据集从易到难组织。此外，我们提出了一个两阶段训练框架，以增强多工具协作推理能力：首先，通过冷启动微调，引导LLMs探索工具调用反馈中的推理模式；其次，采用多工具自我批评RL算法，配合层次化奖励设计，强化奖励理解并促进工具间的有效协作。在超过10个具有挑战性的推理基准上的实验分析，凸显了Tool-Star的有效性与高效性。代码已发布于https://github.com/dongguanting/Tool-Star。

English

Recently, large language models (LLMs) have shown remarkable reasoning capabilities via large-scale reinforcement learning (RL). However, leveraging the RL algorithm to empower effective multi-tool collaborative reasoning in LLMs remains an open challenge. In this paper, we introduce Tool-Star, an RL-based framework designed to empower LLMs to autonomously invoke multiple external tools during stepwise reasoning. Tool-Star integrates six types of tools and incorporates systematic designs in both data synthesis and training. To address the scarcity of tool-use data, we propose a general tool-integrated reasoning data synthesis pipeline, which combines tool-integrated prompting with hint-based sampling to automatically and scalably generate tool-use trajectories. A subsequent quality normalization and difficulty-aware classification process filters out low-quality samples and organizes the dataset from easy to hard. Furthermore, we propose a two-stage training framework to enhance multi-tool collaborative reasoning by: (1) cold-start fine-tuning, which guides LLMs to explore reasoning patterns via tool-invocation feedback; and (2) a multi-tool self-critic RL algorithm with hierarchical reward design, which reinforces reward understanding and promotes effective tool collaboration. Experimental analyses on over 10 challenging reasoning benchmarks highlight the effectiveness and efficiency of Tool-Star. The code is available at https://github.com/dongguanting/Tool-Star.