基于置信度的推理：利用不确定性头高效验证大语言模型推理步骤

Reasoning with Confidence: Efficient Verification of LLM Reasoning Steps via Uncertainty Heads

November 9, 2025

作者: Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov, Mrinmaya Sachan

cs.AI

摘要

处理复杂任务通常需要大型语言模型生成冗长的多步推理链。已有研究表明，对单个推理步骤的正确性进行验证能够进一步提升模型在此类任务中的表现与效率，并增强解决方案的可解释性。然而，现有验证方法（如过程奖励模型）存在计算成本高昂、适用领域受限或需要大规模人工/模型生成标注等局限性。为此，我们提出一种基于数据驱动不确定性评分的轻量级推理步骤验证方案。通过训练基于Transformer的不确定性量化头部模块，利用冻结LLM的内部状态来估计其生成过程中推理步骤的不确定性。该方法完全自动化：目标标签可由更大规模LLM（如DeepSeek R1）生成，或由原模型以自监督方式产生。该头部模块参数量不足1000万，兼具高效性与轻量化特性。在数学、规划、常识问答等多个领域，其性能媲美甚至超越参数量达810倍的过程奖励模型。我们的研究结果表明，LLM内部状态编码了其不确定性，可作为推理验证的可靠信号，为构建可扩展、泛化性强的自省式LLM指明了新方向。

English

Solving complex tasks usually requires LLMs to generate long multi-step reasoning chains. Previous work has shown that verifying the correctness of individual reasoning steps can further improve the performance and efficiency of LLMs on such tasks and enhance solution interpretability. However, existing verification approaches, such as Process Reward Models (PRMs), are either computationally expensive, limited to specific domains, or require large-scale human or model-generated annotations. Thus, we propose a lightweight alternative for step-level reasoning verification based on data-driven uncertainty scores. We train transformer-based uncertainty quantification heads (UHeads) that use the internal states of a frozen LLM to estimate the uncertainty of its reasoning steps during generation. The approach is fully automatic: target labels are generated either by another larger LLM (e.g., DeepSeek R1) or in a self-supervised manner by the original model itself. UHeads are both effective and lightweight, containing less than 10M parameters. Across multiple domains, including mathematics, planning, and general knowledge question answering, they match or even surpass the performance of PRMs that are up to 810x larger. Our findings suggest that the internal states of LLMs encode their uncertainty and can serve as reliable signals for reasoning verification, offering a promising direction toward scalable and generalizable introspective LLMs.