結合後設認知回饋之強化學習引發大型語言模型忠實的不確定性表達

摘要

元認知是智力的關鍵組成部分，描述個體監控及調節自身認知過程的能力。然而，大型語言模型在關鍵元認知能力上存在系統性缺陷：它們會自信滿滿地產生幻覺、無法識別知識邊界，且誤報內在不確定性——從而削弱可信度與可靠性。由於監控任務表現並據此調整行為是元認知的核心，我們假定能準確判斷自身表現的模型，更可能改善其表現。我們透過兩種新機制將此概念具體化：其一為「具元認知反饋的強化學習」，這是一種在偏好最佳化過程中根據模型對自身表現的自我判斷品質來修正完成排序的範式；其二為「元認知資料選取」，利用類似的自我判斷來識別高價值訓練樣本，表現優於單純的主動學習。我們將這些創新應用於「忠實校準」問題——該任務本質上即為元認知任務：其目標在於使表達出的不確定性與內在不確定性一致，即使對前沿大型語言模型而言亦具挑戰。我們採用兩階段解耦方法：首先運用這些方法校準模型自報信心分數的忠實度，再透過目標式輸出編輯將其映射至自然、可適應情境的語言不確定性。大量實驗顯示，具元認知反饋的強化學習可在多樣化任務上達成可泛化的最先進忠實校準，同時維持準確性。此外，該方法在增強模型評估及表達自身能力界限的同時，其表現比標準強化學習高出最多63%。這使具元認知反饋的強化學習成為一個有前景的範式，可增強大型語言模型的元認知能力以提升其能力與對齊，並指出元認知表現可作為有效的強化學習信號，以克服先前內在反饋方法的限制。

English

Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes. Yet LLMs exhibit systemic deficiencies in key metacognitive faculties: they hallucinate with high confidence, fail to recognize knowledge boundaries, and misrepresent their internal uncertainty--undermining trustworthiness and reliability. Since monitoring task performance and adapting behavior accordingly are central to metacognition, we posit that models capable of accurately judging their own performance are better positioned to improve it. We operationalize this idea via two novel mechanisms: reinforcement learning with metacognitive feedback (RLMF), a paradigm to refine completion rankings during preference optimization based on the quality of a model's self-judgments of performance, and metacognitive data selection, which uses similar self-judgments to identify high-value training examples, outperforming naive active learning. We apply these innovations to the problem of faithful calibration (FC), a task that is itself fundamentally metacognitive: the goal is to align expressed with intrinsic uncertainty, difficult even for frontier LLMs. We adopt a two-stage, decoupled approach, first using these methods to calibrate the faithfulness of models' self-reported confidence scores, then mapping to natural, context-adaptable linguistic uncertainty via targeted output editing. Extensive experiments show RLMF achieves generalizable, state-of-the-art FC on diverse tasks while preserving accuracy. Further, RLMF surpasses standard RL by up to 63% while enhancing models' ability to assess and express their own capability limits. This positions RLMF as a promising paradigm to enhance LLM metacognition toward improved abilities and alignment, and suggests metacognitive performance as an effective RL signal to overcome limits of prior intrinsic feedback methods.