FoNE：透過傅立葉特徵實現精確的單令牌數字嵌入

摘要

大型語言模型（LLMs）通常使用多個標記來表示數字，這需要模型將這些標記聚合起來以解釋數值。這種碎片化使得訓練和推理都變得不那麼高效，並對模型在與數字相關的任務上的表現產生不利影響。受到預訓練LLMs內部學習數字標記的類似傅立葉特徵的觀察的啟發，我們提出了傅立葉數字嵌入（FoNE），這是一種直接將數字與其傅立葉特徵映射到嵌入空間的新方法。FoNE將每個數字編碼為一個標記，每位數僅具有兩個嵌入維度，有效捕捉數值而不會產生碎片化。這種緊湊的表示加快了訓練和推理的速度。與傳統的子詞和按位嵌入相比，FoNE不僅減少了計算開銷，還在各種數值任務（包括加法、減法和乘法）中實現了更高的準確性。在6位十進制加法中，FoNE僅需要64倍的數據即可實現99%的準確性，而使用的標記數分別比子詞和按位嵌入少3倍和6倍。此外，FoNE是唯一一種在超過100,000個測試示例中實現加法、減法和乘法100%準確性的方法。代碼和可視化可在https://fouriernumber.github.io/找到。

English

Large Language Models (LLMs) typically represent numbers using multiple tokens, which requires the model to aggregate these tokens to interpret numerical values. This fragmentation makes both training and inference less efficient and adversely affects the model's performance on number-related tasks. Inspired by the observation that pre-trained LLMs internally learn Fourier-like features for number tokens, we propose Fourier Number Embedding (FoNE), a novel method that directly maps numbers into the embedding space with their Fourier features. FoNE encodes each number as a single token with only two embedding dimensions per digit, effectively capturing numerical values without fragmentation. This compact representation accelerates both training and inference. Compared to traditional subword and digit-wise embeddings, FoNE not only reduces computational overhead but also achieves higher accuracy across various numerical tasks including addition, subtraction and multiplication. On 6-digit decimal addition, FoNE requires 64times less data to achieve 99% accuracy than subword and digit-wise embeddings while using 3times and 6times fewer tokens per number, respectively. Furthermore, FoNE is the only method that yields 100% accuracy on over 100,000 test examples for addition, subtraction, and multiplication. The codes and visualization are available at https://fouriernumber.github.io/.

FoNE：透過傅立葉特徵實現精確的單令牌數字嵌入

FoNE: Precise Single-Token Number Embeddings via Fourier Features

摘要

Summary

Support

Support