LLM预测器所知而不言：探测内部表征以实现校准与忠实性

摘要

针对预测任务微调的大型语言模型可能准确但校准不佳，其思维链推理也未必忠实反映预测背后的证据。我们探究内部表征是否能为这两方面提供更直接的视角。基于Eternis-Forecaster 8B模型与OpenForesight数据集，我们训练了中间激活的表征池化探针，发现其校准效果显著提升；这一结论同样适用于GLM-4.7-Flash和GLM-4.5-Air模型。随后，我们通过证据消融与干扰注入评估思维链的忠实性：移除提示中的关键信息常能改变模型预测，但推理轨迹却保持原样。相同的探针可作为谎言检测器：其激活值追踪行为变化的准确度远高于推理轨迹，且在84%的案例中能预测变化方向，包括当思维链掩盖扰动影响时。最后，强制回答实验表明，预测在推理开始前已基本固化：仅需一次推理前的传递即可复现既定答案与置信度，而根据预设答案分布的离散度路由问题，可在不损失准确率的前提下节省30-47%的生成词元。综合而言，这些结果确立了将内部表征探查作为校准、审计与分流语言模型预测系统（乃至更广泛的推理模型）的实用工具。

English

Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. We ask whether internal representations offer a more direct window into both. Working with Eternis-Forecaster 8B on OpenForesight, we train representation-pooling probes on intermediate activations and find they achieve substantially better calibration; a result that also holds for GLM-4.7-Flash and GLM-4.5-Air. We then assess CoT faithfulness through evidence ablation and diversionary injection: removing an influential source in the prompt often changes the model's forecast while leaving the reasoning trace untouched. The same probes function as lie detectors: their activations track behavioral shifts far better than the reasoning trace does, and they also predict the direction of change in 84% of cases, including when the CoT conceals the perturbation's influence. Finally, forced answering reveals that forecasts are largely fixed before reasoning begins: a single pre-reasoning pass recovers the committed answer and confidence, and routing questions by the spread of this pre-set answer distribution saves 30-47% of generated tokens, with no loss of accuracy. Together, these results establish probing internal representations as a practical tool for calibrating, auditing, and triaging language model forecasters and reasoning models more broadly.