ReFIne：一个具备可靠性、忠实性与可解释性的可信大型推理模型框架
ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability
October 10, 2025
作者: Chung-En Sun, Ge Yan, Akshay Kulkarni, Tsui-Wei Weng
cs.AI
摘要
近期在长链思维推理（CoT）领域的进展主要聚焦于答案准确性和计算效率，却忽视了可信度这一关键维度。我们认为，实用的推理系统必须具备可信性，具体表现为三个特性：可解释性、忠实性和可靠性。为此，我们提出了ReFIne这一新型训练框架，它结合了监督微调与GRPO技术，旨在引导模型实现以下目标：(i) 通过生成结构化、基于标签的推理轨迹，并辅以高层规划，提升可解释性，使人类更易理解；(ii) 通过明确揭示指导每一步解决方案的关键信息，并保持跨部分引用的一致性，增强忠实性；(iii) 通过提供对推理过程合理性的自我评估及最终答案的置信度，促进可靠性。我们将ReFIne应用于不同规模（1.7B/4B/8B）的Qwen3模型，并在难度各异的数学基准上进行评估。实验结果表明，采用ReFIne的模型生成了更清晰、结构更优的推理轨迹（可解释性提升44.0%），更忠实地展现了其决策过程（忠实性提升18.8%），并提供了信息丰富的置信度估计（可靠性提升42.4%）。这些发现揭示了一个被忽视但至关重要的方向：推理模型的优化不应仅限于准确性，还应涵盖可信度的更广泛维度。我们的代码已公开于：https://github.com/Trustworthy-ML-Lab/Training_Trustworthy_LRM_with_Refine。
English
Recent advances in long chain-of-thought (CoT) reasoning have largely
prioritized answer accuracy and token efficiency, while overlooking aspects
critical to trustworthiness. We argue that usable reasoning systems must be
trustworthy, characterized by three properties: interpretability, faithfulness,
and reliability. To this end, we propose ReFIne, a new training framework that
integrates supervised fine-tuning with GRPO to encourage models to: (i) improve
interpretability by producing structured, tag-based traces with high-level
planning that are easier for humans to follow; (ii) enhance faithfulness by
explicitly disclosing the decisive information guiding each solution, with
consistent cross-section references; and (iii) promote reliability by providing
self-assessments of both the derivation's soundness and the confidence of the
final answer. We apply ReFIne to the Qwen3 models at multiple scales
(1.7B/4B/8B) and evaluate across mathematical benchmarks of varying difficulty.
Our experimental results show that ReFIne models generate clearer and
better-structured reasoning traces (interpretability +44.0%), more faithfully
expose their underlying decision process (faithfulness +18.8%), and offer
informative confidence estimates (reliability +42.4%). These findings highlight
an overlooked but important direction: reasoning models should be optimized not
only for accuracy, but also for broader dimensions of trustworthiness. Our code
is available at:
https://github.com/Trustworthy-ML-Lab/Training_Trustworthy_LRM_with_Refine