MultiHashFormer: 基于哈希的生成式语言模型

摘要

语言模型（LM）通过嵌入矩阵表示词元，该矩阵规模随词汇表大小线性增长。为限制参数占用，先前研究在编码器专用模型中提出将多个词元哈希至单一向量。这种方法虽然提升了参数效率，但多对一冲突问题使其无法应用于因果语言模型。本文提出MultiHashFormer框架，首次实现基于哈希的自回归机制。每个词元被表示为独特的哈希签名（由多个独立哈希函数生成的一串离散哈希ID），经哈希编码器压缩为单个潜向量供Transformer解码器处理，再由哈希解码器生成下一词元的哈希签名，最终映射回文本。我们在1亿、10亿和30亿参数规模下进行评估，结果表明MultiHashFormer在多项基准测试中持续优于标准Transformer语言模型。此外，我们的模型支持词汇表多语言扩展，且无需任何修改即可保持恒定参数占用。

English

Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder-only models. While this offers parameter efficiency, many-to-one collisions prevent its use in causal LMs. In this paper, we propose MultiHashFormer, a new framework that allows hash-based autoregression. Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions. A Hash Encoder compresses this signature into a single latent vector for processing by a Transformer decoder. Then, a Hash Decoder generates the hash signature of the next token, which is then mapped back to text. We evaluate our approach at the 100M, 1B and 3B parameter scales, demonstrating that MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks. Furthermore, we show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications.