元认知反馈强化学习促使大型语言模型忠实表达不确定性

摘要

元认知是智力的关键组成部分，描述了个体监控和调节自身认知过程的能力。然而，大型语言模型（LLMs）在核心元认知能力上存在系统性缺陷：它们会以高置信度产生幻觉、无法识别知识边界，并错误表征其内部不确定性——这损害了可信度与可靠性。由于监控任务表现并相应调整行为是元认知的核心，我们推测：能够准确判断自身表现的模型更有可能改进其表现。我们通过两种新机制实现这一理念：基于元认知反馈的强化学习（RLMF），这是一种在偏好优化过程中，基于模型对自身表现判断的质量来优化完成结果排序的范式；以及元认知数据选择，它利用类似的自我判断来识别高价值训练样本，其效果优于朴素主动学习。我们将这些创新应用于忠实校准（FC）问题——该任务本身本质上具有元认知特性：目标是使表达的不确定性与内在不确定性对齐，即便对前沿LLMs而言也颇具挑战。我们采用两阶段解耦方法，首先运用这些方法校准模型自报置信分数的忠实性，随后通过定向输出编辑映射为自然且可适应上下文的语言不确定性。大量实验表明，RLMF在保持准确性的同时，能在多样化任务上实现可泛化的最先进FC效果。此外，RLMF超越标准强化学习达63%，同时增强了模型评估和表达自身能力边界的能力。这使RLMF成为增强LLM元认知、提升能力与对齐度的有前景范式，并表明元认知表现可作为有效强化学习信号，突破先前内在反馈方法的局限。

English

Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes. Yet LLMs exhibit systemic deficiencies in key metacognitive faculties: they hallucinate with high confidence, fail to recognize knowledge boundaries, and misrepresent their internal uncertainty--undermining trustworthiness and reliability. Since monitoring task performance and adapting behavior accordingly are central to metacognition, we posit that models capable of accurately judging their own performance are better positioned to improve it. We operationalize this idea via two novel mechanisms: reinforcement learning with metacognitive feedback (RLMF), a paradigm to refine completion rankings during preference optimization based on the quality of a model's self-judgments of performance, and metacognitive data selection, which uses similar self-judgments to identify high-value training examples, outperforming naive active learning. We apply these innovations to the problem of faithful calibration (FC), a task that is itself fundamentally metacognitive: the goal is to align expressed with intrinsic uncertainty, difficult even for frontier LLMs. We adopt a two-stage, decoupled approach, first using these methods to calibrate the faithfulness of models' self-reported confidence scores, then mapping to natural, context-adaptable linguistic uncertainty via targeted output editing. Extensive experiments show RLMF achieves generalizable, state-of-the-art FC on diverse tasks while preserving accuracy. Further, RLMF surpasses standard RL by up to 63% while enhancing models' ability to assess and express their own capability limits. This positions RLMF as a promising paradigm to enhance LLM metacognition toward improved abilities and alignment, and suggests metacognitive performance as an effective RL signal to overcome limits of prior intrinsic feedback methods.