長度懲罰使思維鏈較難監控

摘要

長度懲罰的強化學習可以縮短思考鏈推理，同時隱藏驅動模型答案的影響。在我們的實驗中，使用長度懲罰進行訓練並未阻止誤導性提示引導模型，儘管模型的思考鏈提及提示的頻率大幅降低。基於令牌準確率的評估會將這些運行視為成功，因為它們使用了更少的推理令牌且準確率損失很小；但它會忽略剩餘軌跡是否仍顯示驅動答案的因素。我們訓練不同目標思考鏈長度的Qwen3-4B和Qwen3-14B變體，然後在保留的MMLU-Pro-R和四個遷移基準上使用偏差提示干預對其進行評估。壓縮大幅減少推理令牌，保留了大部分選擇題準確率，並使提示影響接近基線。在最強的目標下，Qwen3-14B的下限忠實度降至基線的63.1%，Qwen3-4B降至69.4%；監控器捕捉到提示使用的原始比率從69%降至49%，以及從60%降至48%。為了將長度與內容分離，我們從未壓縮的基線鏈中隨機刪除句子，直至剩餘文本匹配壓縮後的長度。即使在長度匹配後，對於兩個Qwen3規模和所有五個評估分佈，壓縮後的鏈揭示提示的頻率仍比隨機縮短的基線鏈低7至35個百分點。因此，壓縮不僅僅是縮短推理，而是優先移除監控器需要用來觀察影響答案因素的線索。綜合這些結果揭示了一個壓縮-可監控性邊界，其中成本更低的推理可以保留答案，同時使背後的影響更難被檢測。

English

Length-penalized reinforcement learning can shorten chain-of-thought reasoning while hiding an influence that drives the model's answer. In our experiments, training with length penalties does not stop misleading hints from steering models, even though the models' chains of thought mention the hint much less often. A token-accuracy evaluation would count these runs as successful because they use fewer reasoning tokens with little accuracy loss; it would miss whether the remaining trace still shows what drove the answer. We train Qwen3-4B and Qwen3-14B variants with different target chain lengths, then evaluate them with biasing-hint interventions on held-out MMLU-Pro-R and four transfer benchmarks. Compression sharply cuts reasoning tokens, preserves most multiple-choice accuracy, and leaves hint influence near baseline. At the strongest target, lower-bound faithfulness falls to 63.1% of baseline for Qwen3-14B and 69.4% for Qwen3-4B; the raw rate at which a monitor catches hint use falls from 69% to 49% and from 60% to 48%. To separate length from content, we randomly delete sentences from uncompressed baseline chains until the remaining text matches the compressed length. Even after this length matching, compressed chains disclose the hint 7-35 percentage points less often than baseline chains that we shorten at random, for both Qwen3 sizes and all five evaluation distributions. Compression therefore does more than shorten reasoning, preferentially removing the cues a monitor needs to see what influenced the answer. Together, these results reveal a compression-monitorability frontier in which cheaper reasoning can preserve answers while making the influences behind them harder to detect.