當圖形標記沉沒：圖形語言模型的機制分析

摘要

圖語言模型已成為將大型語言模型應用於圖學習任務的一個有潛力的方向。通過將圖拓撲結構與節點信息轉化為圖標記，圖語言模型使大型語言模型能夠同時處理結構化的圖輸入與文本指令。然而，目前仍不清楚大型語言模型內部如何解讀這些圖標記，以及圖標記是否作為圖結構有意義的載體。在本研究中，我們通過代表性圖語言模型架構中的圖標記行為，分析大型語言模型如何處理圖信息。研究結果。我們發現，圖語言模型中圖標記的內在顯著性並不等同於圖信息的利用程度。圖匯聚標記持續表現為激活級異常值：這些標記可透過在少數隱藏狀態維度上的巨量激活值來識別，並且傾向於出現在圖標記的早期位置。然而，這種激活級顯著性並不意味著這些標記是圖信息的主要載體。與語言模型及視覺-語言模型中的經典注意力匯聚點不同，圖匯聚標記不一定能吸引查詢標記的最大注意力權重。通過剪枝、重定位與交換等干預實驗，我們證明圖匯聚標記對於下游預測而言，並非最重要的語義或結構標記。影響。綜合來看，這些結果表明，當前圖語言模型將圖結構映射至大型語言模型的標記空間後，所產生的圖標表徵並未自然形成完全可用的拓撲感知內部表徵；相反，它們展現出激活級顯著性與圖語義實用性之間的分離。這種分離指出了現有圖標記構建、定位與對齊機制的局限性。

English

Graph Language Models (GLMs) have become a promising direction for adapting Large Language Models (LLMs) to graph learning tasks. By transforming graph topology and node information into graph tokens, GLMs allow LLMs to jointly process structured graph inputs and textual instructions. Yet, it remains unclear how LLMs internally interpret these graph tokens and whether graph tokens act as meaningful carriers of graph structure. In this work, we analyze how LLMs process graph information through graph-token behavior in representative GLM architectures. Findings. We find that the internal saliency of graph tokens in GLMs is not equivalent to graph information utilization. Graph sink tokens consistently emerge as activation-level outliers: they can be identified by massive activation values along a small set of hidden-state dimensions and are biased toward early graph-token positions. However, this activation-level saliency does not imply that these tokens are the main carriers of graph information. Unlike classical attention sinks in language and vision-language models, graph sink tokens do not necessarily attract the largest attention weights from query tokens. Through pruning, repositioning, and swapping interventions, we show that graph sink tokens are not the most important semantic or structural tokens for downstream prediction. Implications. Together, these results suggest that after current GLMs map graph structure into the LLM token space, the resulting graph-token representations do not naturally form a fully usable topology-aware internal representation; instead, they exhibit a decoupling between activation-level saliency and graph-semantic utility. This decoupling points to limitations in existing graph-token construction, placement, and alignment mechanisms.