MultiHashFormer：基於哈希的生成式語言模型

摘要

語言模型（LM）使用嵌入矩陣來表示詞元，該矩陣的大小與詞彙量呈線性關係。為限制參數規模，先前研究提出在僅編碼器模型中將多個詞元哈希至單一向量。雖然這提升了參數效率，但多對一碰撞使其無法用於因果語言模型。本文提出 MultiHashFormer，一個允許基於哈希的自迴歸的新框架。每個詞元被表示為獨特的哈希簽章，即由多個獨立哈希函數產生的離散哈希 ID 短序列。哈希編碼器將此簽章壓縮為單一潛在向量，供 Transformer 解碼器處理；接著哈希解碼器生成下一個詞元的哈希簽章，再將其映射回文字。我們在 100M、1B 及 3B 參數規模下評估此方法，結果顯示 MultiHashFormer 在多項基準測試中持續優於標準 Transformer LM。此外，我們證明該模型能在參數規模恆定且無需任何修改的情況下處理多語言詞彙擴展。

English

Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder-only models. While this offers parameter efficiency, many-to-one collisions prevent its use in causal LMs. In this paper, we propose MultiHashFormer, a new framework that allows hash-based autoregression. Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions. A Hash Encoder compresses this signature into a single latent vector for processing by a Transformer decoder. Then, a Hash Decoder generates the hash signature of the next token, which is then mapped back to text. We evaluate our approach at the 100M, 1B and 3B parameter scales, demonstrating that MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks. Furthermore, we show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications.