思考令牌有助於安全嗎？

摘要

現今的推理模型利用思考標記（thinking tokens），在基準測試上獲得比純指令調整模型更強的表現。一般也認為，這種更為「審慎」的模式，應能透過為模型提供安全的空間，讓其評估對某請求的預定回答是否違反安全原則，從而提升對齊與安全性。我們提出的證據顯示，這一直覺並非總是正確。在涵蓋GPT-OSS、Qwen、Olmo及Phi系列等前沿開源權重推理模型中，我們發現最終的拒絕／順從結果，在可見思考過程開始之前，便可經由在第一個標記的隱藏表徵上訓練的分類頭（trained head）高準確度地預測（拒絕／順從預測的AUROC達0.84–0.95，平衡準確率約88%）。思考過程實際上更接近前綴補全，而非審慎修正；儘管在文本層面呈現出審慎表象（約74%的文本層面審慎行為發生在回應分佈已鎖定於拒絕或順從某一側時），最終結果在思考過程約前20%後便極少改變。我們也發現，現有的推論時與訓練基礎的安全干預措施，雖以誘發審慎思考為目標，卻多數使模型行為傾向過度拒絕，同時壓抑本已稀少的審慎訊號。我們的研究結果顯示，當前推理模型的安全行為遠不如普遍假設的那樣審慎，並凸顯了開發能真正引發安全審慎思考方法的需求。

English

Today's reasoning models use thinking tokens to attain stronger performance on benchmarks than their instruction-tuned counterparts. It is also generally believed that this more "deliberative" mode should improve alignment and safety, by providing the model a safe space to consider whether its planned answer to a request violates its safety principles. We present evidence that this intuition is not always correct. Across frontier open-weight reasoning models spanning GPT-OSS, Qwen, Olmo, and Phi families, we find that the eventual refusal/compliance outcome is already strongly predictable via a trained head on the first token's hidden representation (0.84-0.95 AUROC and sim88% balanced accuracy for predicting refusal/compliance) before any visible thinking. The thinking process turns out to be more akin to prefix completion than to deliberative revision, with the final outcome rarely changing after the first sim20% of thinking, despite giving the appearance of deliberation at the text level (sim74% of text-level deliberations occur when the response distribution is already locked to one refusal/compliance side). We also find that existing inference-time and training-based safety interventions, despite being motivated by the goal of inducing deliberation, largely shift model behavior toward over-refusal while suppressing already-scarce deliberation signals. Our results suggest that safety behavior in current reasoning models is much less deliberative than commonly assumed, and highlight the need for methods that induce real safety deliberation.