通过预算引导调控大语言模型的思维过程

Steering LLM Thinking with Budget Guidance

June 16, 2025

作者: Junyan Li, Wenshuo Zhao, Yang Zhang, Chuang Gan

cs.AI

摘要

近期，深度思考的大型语言模型往往通过大量推理来提升性能，但如此冗长的推理并非总是理想，因为它会带来过高的推理成本，而性能提升却不成比例。因此，在不牺牲性能的前提下控制推理长度显得尤为重要，但这仍具挑战性，尤其是在严格的思考预算下。我们提出了预算引导法，这是一种简单而有效的方法，旨在无需对大型语言模型进行微调的情况下，引导其推理过程向目标预算靠拢。我们的方法引入了一个轻量级预测器，该预测器在生成下一个令牌时，对剩余思考长度建模为伽马分布。随后，这一信号被用于以柔和的、令牌级别的方式引导生成，确保整体推理轨迹遵循指定的思考预算。预算引导法实现了对思考长度的自然控制，并在具有挑战性的数学基准测试上，相较于基线方法显著提升了令牌效率。例如，在严格预算下，它在MATH-500基准测试上实现了高达26%的准确率提升，同时仅使用了全思考模型63%的思考令牌，保持了竞争力的准确率。预算引导法还泛化至更广泛的任务领域，并展现出新兴能力，如估计问题难度。源代码已发布于：https://github.com/UMass-Embodied-AGI/BudgetGuidance。

English

Recent deep-thinking large language models often reason extensively to improve performance, but such lengthy reasoning is not always desirable, as it incurs excessive inference costs with disproportionate performance gains. Controlling reasoning length without sacrificing performance is therefore important, but remains challenging, especially under tight thinking budgets. We propose budget guidance, a simple yet effective method for steering the reasoning process of LLMs toward a target budget without requiring any LLM fine-tuning. Our approach introduces a lightweight predictor that models a Gamma distribution over the remaining thinking length during next-token generation. This signal is then used to guide generation in a soft, token-level manner, ensuring that the overall reasoning trace adheres to the specified thinking budget. Budget guidance enables natural control of the thinking length, along with significant token efficiency improvements over baseline methods on challenging math benchmarks. For instance, it achieves up to a 26% accuracy gain on the MATH-500 benchmark under tight budgets compared to baseline methods, while maintaining competitive accuracy with only 63% of the thinking tokens used by the full-thinking model. Budget guidance also generalizes to broader task domains and exhibits emergent capabilities, such as estimating question difficulty. The source code is available at: https://github.com/UMass-Embodied-AGI/BudgetGuidance.