語言特定知識：模型在X語言中的表現是否優於英語？

Language Specific Knowledge: Do Models Know Better in X than in English?

May 21, 2025

作者: Ishika Agarwal, Nimet Beyza Bozdag, Dilek Hakkani-Tür

cs.AI

摘要

語碼轉換是一種在同一次表達、思考或對話中交替使用不同語言的常見現象。我們認為，人類之所以進行語碼轉換，是因為他們在談論某些主題和領域時，使用某一種語言比另一種語言更為自在。隨著知識密集型語言模型的興起，我們自然而然地提出了下一個問題：模型是否在某些主題上，使用某種語言X時掌握更多知識？更重要的是，我們能否通過改變推理所使用的語言來提升推理能力？我們創造了「語言特定知識」（Language Specific Knowledge, LSK）這一術語來描述這一現象。由於民族文化往往與不同語言共同發展，我們採用了文化特定的數據集（這些數據集包含關於文化和社會行為規範的知識）。我們發現，在某些非英語語言中，語言模型在使用思維鏈推理時表現更佳，有時甚至在低資源語言中表現更為突出。結合先前研究表明語義相似性並不等同於表徵相似性，我們假設文化特定的文本在相應語言中出現得更為頻繁，使得特定知識僅存在於特定的「專家」語言中。基於初步結果的啟發，我們設計了一種名為LSKExtractor的簡單方法，用於基準測試語言模型中存在的語言特定知識，並在推理過程中加以利用。我們在多種模型和數據集上展示了結果，顯示出準確率平均相對提升了10%。我們的研究有助於開發開源語言模型，使其更具包容性，並更貼近其部署的文化和語言背景。

English

Code-switching is a common phenomenon of alternating between different languages in the same utterance, thought, or conversation. We posit that humans code-switch because they feel more comfortable talking about certain topics and domains in one language than another. With the rise of knowledge-intensive language models, we ask ourselves the next, natural question: Could models hold more knowledge on some topics in some language X? More importantly, could we improve reasoning by changing the language that reasoning is performed in? We coin the term Language Specific Knowledge (LSK) to represent this phenomenon. As ethnic cultures tend to develop alongside different languages, we employ culture-specific datasets (that contain knowledge about cultural and social behavioral norms). We find that language models can perform better when using chain-of-thought reasoning in some languages other than English, sometimes even better in low-resource languages. Paired with previous works showing that semantic similarity does not equate to representational similarity, we hypothesize that culturally specific texts occur more abundantly in corresponding languages, enabling specific knowledge to occur only in specific "expert" languages. Motivated by our initial results, we design a simple methodology called LSKExtractor to benchmark the language-specific knowledge present in a language model and, then, exploit it during inference. We show our results on various models and datasets, showing an average relative improvement of 10% in accuracy. Our research contributes to the open-source development of language models that are inclusive and more aligned with the cultural and linguistic contexts in which they are deployed.