思維鏈的表象之下：大型語言模型中推理運算的機制性詮釋

摘要

大型語言模型中的推理透過多樣化的功能操作展開，例如問題建構、目標分解與演繹推論。儘管這些操作在文本中能被明確區分，但它們如何在表徵空間中以幾何方式組織，目前仍所知甚少。為此，我們探討不同的推理操作是否在隱藏表徵中展現出相應的幾何結構。我們發現，操作在保留的表徵中是可分離的，且可分離性在中間層達到高峰，並驗證此結構並非由詞彙或位置混淆所解釋。跨層觀察時，逐詞元的操作對齊在片段上變得更為分散，而相同的表面詞元會根據其所在區塊的操作被表徵為不同的形式。注意力遮蔽干預進一步顯示，區塊起始處的操作對齊表徵依賴於先前的推理上下文。因此，我們的研究證明語言模型在語言推理表達與其內部幾何結構之間維持著表徵對應。程式碼與專案資料可於 https://github.com/naver-ai/beneath-cot 取得。

English

Reasoning in large language models unfolds through diverse functional operations, such as problem formulation, goal decomposition, and deduction. Although these operations are explicitly distinguished in text, little is known about how they are geometrically organized in representation spaces. To this end, we investigate whether distinct reasoning operations exhibit corresponding geometric structure in hidden representations. We find that operations are separable in held-out representations, with separability peaking in middle layers, and verify that this structure is not explained by lexical or positional confounds. Across layers, token-wise operation-alignment becomes more distributed over spans, while identical surface tokens are represented differently depending on the operation of its surrounding chunk. Attention-masking interventions further show that operation-aligned representations at chunk onset depend on preceding reasoning context. Consequently, our work demonstrates that language models maintain representational correspondence between linguistic reasoning expressions and their internal geometric structures. Code and project materials are available at https://github.com/naver-ai/beneath-cot.