长度惩罚使思维链更难以监控

摘要

长度惩罚强化学习可以缩短链式推理过程，同时隐藏影响模型答案的驱动因素。在我们的实验中，即使模型在链式推理中提及提示的频率显著降低，施加长度惩罚的训练仍无法阻止误导性提示左右模型输出。若采用词元准确率评估，由于这些运行过程使用的推理词元更少且准确率损失很小，会被判定为成功；但这种方法无法判断剩余推理痕迹是否仍能揭示答案的驱动因素。我们训练了Qwen3-4B和Qwen3-14B的不同版本，使其目标链长度各异，随后在保留的MMLU-Pro-R数据集及四个迁移基准上通过引入偏颇提示进行干预评估。压缩处理显著削减了推理词元数量，基本保持了多项选择准确率，而提示影响程度与基线水平接近。在最严苛的压缩目标下，Qwen3-14B的下限忠实度降至基线的63.1%，Qwen3-4B降至69.4%；监控器对提示使用情况的原始捕捉率从69%降至49%，从60%降至48%。为分离长度与内容的影响，我们从未压缩的基线链中随机删除句子，直至剩余文本长度与压缩链匹配。即便经过长度匹配，对于两种规模的Qwen3版本及全部五个评估分布，压缩链披露提示的频率仍比随机缩短的基线链低7-35个百分点。因此，压缩的作用不仅限于缩短推理过程，更会优先移除监控器识别答案影响来源所需的线索。这些发现共同揭示了压缩-可监控性前沿：更经济的推理方式在保留答案的同时，使答案背后的影响因素更难以被检测。

English

Length-penalized reinforcement learning can shorten chain-of-thought reasoning while hiding an influence that drives the model's answer. In our experiments, training with length penalties does not stop misleading hints from steering models, even though the models' chains of thought mention the hint much less often. A token-accuracy evaluation would count these runs as successful because they use fewer reasoning tokens with little accuracy loss; it would miss whether the remaining trace still shows what drove the answer. We train Qwen3-4B and Qwen3-14B variants with different target chain lengths, then evaluate them with biasing-hint interventions on held-out MMLU-Pro-R and four transfer benchmarks. Compression sharply cuts reasoning tokens, preserves most multiple-choice accuracy, and leaves hint influence near baseline. At the strongest target, lower-bound faithfulness falls to 63.1% of baseline for Qwen3-14B and 69.4% for Qwen3-4B; the raw rate at which a monitor catches hint use falls from 69% to 49% and from 60% to 48%. To separate length from content, we randomly delete sentences from uncompressed baseline chains until the remaining text matches the compressed length. Even after this length matching, compressed chains disclose the hint 7-35 percentage points less often than baseline chains that we shorten at random, for both Qwen3 sizes and all five evaluation distributions. Compression therefore does more than shorten reasoning, preferentially removing the cues a monitor needs to see what influenced the answer. Together, these results reveal a compression-monitorability frontier in which cheaper reasoning can preserve answers while making the influences behind them harder to detect.