强化微调的幻觉代价

摘要

强化微调（Reinforcement Finetuning, RFT）已成为提升大型语言模型（LLMs）推理能力的标准方法。然而，其对模型可信度的影响仍待深入探究。在本研究中，我们识别并系统性地研究了RFT的一个关键副作用，我们称之为“幻觉税”：即模型在拒绝行为上的退化，导致其自信地生成无法回答问题的虚假答案。为探究此现象，我们引入了SUM（合成不可解数学问题），这是一个高质量的不可解数学问题数据集，旨在通过从信息不足或模糊的推理中测试模型识别不可解问题的能力。我们的结果显示，标准的RFT训练可能使模型拒绝率降低超过80%，显著增加了模型产生幻觉的倾向。我们进一步证明，在RFT过程中仅融入10%的SUM数据，即可大幅恢复适当的拒绝行为，且在可解任务上的准确性损失极小。至关重要的是，这种方法使LLMs能够利用推理时的计算资源来思考自身的不确定性和知识边界，不仅提升了在跨域数学问题上的泛化能力，也改善了事实性问答任务的表现。

English

Reinforcement finetuning (RFT) has become a standard approach for enhancing the reasoning capabilities of large language models (LLMs). However, its impact on model trustworthiness remains underexplored. In this work, we identify and systematically study a critical side effect of RFT, which we term the hallucination tax: a degradation in refusal behavior causing models to produce hallucinated answers to unanswerable questions confidently. To investigate this, we introduce SUM (Synthetic Unanswerable Math), a high-quality dataset of unanswerable math problems designed to probe models' ability to recognize an unanswerable question by reasoning from the insufficient or ambiguous information. Our results show that standard RFT training could reduce model refusal rates by more than 80%, which significantly increases model's tendency to hallucinate. We further demonstrate that incorporating just 10% SUM during RFT substantially restores appropriate refusal behavior, with minimal accuracy trade-offs on solvable tasks. Crucially, this approach enables LLMs to leverage inference-time compute to reason about their own uncertainty and knowledge boundaries, improving generalization not only to out-of-domain math problems but also to factual question answering tasks.

强化微调的幻觉代价

The Hallucination Tax of Reinforcement Finetuning

摘要

Summary

Support

Support