語義中樞假設：語言模型在不同語言和模態之間共享語義表徵

The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities

November 7, 2024

作者: Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, Yoon Kim

cs.AI

摘要

現代語言模型能夠處理來自不同語言和模態的輸入。我們假設這些模型通過學習跨異質數據類型（例如不同語言和模態）之間的共享表示空間來獲得這種能力，該空間將語義上相似的輸入放置在一起，即使它們來自不同的模態/語言。我們稱之為語義中樞假設，這源自神經科學中的中樞-輻輪模型（Patterson等人，2007），該模型認為人類大腦中的語義知識是通過一個跨模態的語義“中樞”組織的，該中樞整合了來自各種模態特定的“輻輪”區域的信息。我們首先展示，在中間層中，不同語言中語義等效輸入的模型表示是相似的，並且可以通過模型的主要預訓練語言使用對數鏡片來解釋這個空間。這種趨勢擴展到其他數據類型，包括算術表達式、代碼和視覺/音頻輸入。對一種數據類型中的共享表示空間的干預也可以預測地影響其他數據類型中模型的輸出，這表明這種共享表示空間不僅僅是在廣泛數據上的大規模訓練的副產品，而是模型在輸入處理過程中積極利用的東西。

English

Modern language models can process inputs across diverse languages and modalities. We hypothesize that models acquire this capability through learning a shared representation space across heterogeneous data types (e.g., different languages and modalities), which places semantically similar inputs near one another, even if they are from different modalities/languages. We term this the semantic hub hypothesis, following the hub-and-spoke model from neuroscience (Patterson et al., 2007) which posits that semantic knowledge in the human brain is organized through a transmodal semantic "hub" which integrates information from various modality-specific "spokes" regions. We first show that model representations for semantically equivalent inputs in different languages are similar in the intermediate layers, and that this space can be interpreted using the model's dominant pretraining language via the logit lens. This tendency extends to other data types, including arithmetic expressions, code, and visual/audio inputs. Interventions in the shared representation space in one data type also predictably affect model outputs in other data types, suggesting that this shared representations space is not simply a vestigial byproduct of large-scale training on broad data, but something that is actively utilized by the model during input processing.