以預算指導引導大型語言模型思維

Steering LLM Thinking with Budget Guidance

June 16, 2025

作者: Junyan Li, Wenshuo Zhao, Yang Zhang, Chuang Gan

cs.AI

摘要

近期深度思考的大型語言模型常透過廣泛推理來提升效能，然而此類冗長的推理並非總是理想，因其會導致過高的推論成本，卻僅帶來不成比例的效能增益。因此，在無損效能的前提下控制推理長度至關重要，但這仍具挑戰性，特別是在嚴格的思考預算下。我們提出預算引導法，這是一種無需對大型語言模型進行微調即可引導其推理過程朝向目標預算的簡單而有效的方法。我們的方法引入了一個輕量級預測器，該預測器在生成下一個詞元時對剩餘的思考長度建模為伽馬分佈。此信號隨後用於以柔和的、詞元級別的方式引導生成，確保整體的推理軌跡符合指定的思考預算。預算引導法實現了對思考長度的自然控制，並在具挑戰性的數學基準測試上相較於基線方法顯著提升了詞元效率。例如，在嚴格預算下，它在MATH-500基準測試上實現了高達26%的準確率提升，同時僅使用完整思考模型63%的思考詞元，仍保持了競爭力的準確率。預算引導法亦能泛化至更廣泛的任務領域，並展現出如估計問題難度等新興能力。源代碼可於以下網址獲取：https://github.com/UMass-Embodied-AGI/BudgetGuidance。

English

Recent deep-thinking large language models often reason extensively to improve performance, but such lengthy reasoning is not always desirable, as it incurs excessive inference costs with disproportionate performance gains. Controlling reasoning length without sacrificing performance is therefore important, but remains challenging, especially under tight thinking budgets. We propose budget guidance, a simple yet effective method for steering the reasoning process of LLMs toward a target budget without requiring any LLM fine-tuning. Our approach introduces a lightweight predictor that models a Gamma distribution over the remaining thinking length during next-token generation. This signal is then used to guide generation in a soft, token-level manner, ensuring that the overall reasoning trace adheres to the specified thinking budget. Budget guidance enables natural control of the thinking length, along with significant token efficiency improvements over baseline methods on challenging math benchmarks. For instance, it achieves up to a 26% accuracy gain on the MATH-500 benchmark under tight budgets compared to baseline methods, while maintaining competitive accuracy with only 63% of the thinking tokens used by the full-thinking model. Budget guidance also generalizes to broader task domains and exhibits emergent capabilities, such as estimating question difficulty. The source code is available at: https://github.com/UMass-Embodied-AGI/BudgetGuidance.