知止之道：诊断并训练大语言模型以中止徒劳推理

摘要

大语言模型在超出其能力范围的任务上会产生计算成本高昂但语义空洞的推理，由此带来风险——那些听起来合理但实际错误的推导可能误导用户。我们通过系统分析刻画了这一徒劳推理现象，揭示了普遍存在的能力越界以及能力与行为之间的系统性校准失准。主要的失败模式是似是而非的推理，其输出表面上看似合理，但包含细微错误，并随任务难度增加而加剧。为解决这一问题，我们提出了CaRL（能力对齐强化学习，Capability-aligned Reinforcement Learning），该方法通过奖励塑造激励模型拒绝徒劳推理，并通过事后拒绝增强将失败转化为拒绝监督，从而使模型行为与能力边界对齐。实验表明，该方法在不同任务难度下显著减少了徒劳推理，同时保持了性能，有效实现了能力对齐行为而不牺牲效用。https://github.com/icip-cas/Knowing-When-to-Quit

English

Large language models generate computationally expensive yet semantically void reasoning on beyond-capability tasks, creating risks where plausible-sounding but incorrect derivations mislead users. We characterize this futile reasoning phenomenon through systematic analysis, revealing universal capability overreach and systematic miscalibration between capability and behavior. The dominant failure mode is specious reasoning, which outputs look superficially valid but contain subtle errors, escalating with task difficulty. To address this, we introduce CaRL (Capability-aligned Reinforcement Learning), which aligns model behavior with capability boundaries through reward shaping that incentivizes refusal over futile reasoning and hindsight refusal augmentation that converts failures into refusal supervision. Experiments demonstrate a substantial reduction in futile reasoning while preserving performance across task difficulties, effectively achieving capability-aligned behavior without sacrificing utility. https://github.com/icip-cas/Knowing-When-to-Quit