Daedalus-150M：专为CPU推理设计的卷积-注意力混合架构

摘要

小型語言模型通常先以大型模型的方式建構，之後再壓縮到 CPU 上執行。我們反其道而行：先鎖定目標——單一使用者、一次一個詞元、4 位元權重、一般 CPU——再選擇適合此目標的架構。結果，其 18 個區塊中僅有 6 個保留完整注意力機制；其餘 12 個使用短卷積，無論對話變得多長，這些卷積的記憶跨度僅有兩個時間步，因此網路中三分之二的區塊永遠不會重新讀取不斷增長的快取。該模型以 599 億個詞元從零開始訓練，在五任務基準上得分 47.31，而訓練開始前設定的比較基準為 42.20。它超越了 GPT-2 124M、Pythia-160M、OPT-125M 與 GPT-neo-125M，這些模型均以三到六倍的資料量訓練；同時也超過 MobileLLM-125M 的公開得分，儘管後者的訓練資料達一兆個詞元。驗證的每位元組位元數（bits-per-byte）為 0.8685。為了檢驗架構本身而非訓練配方，我們使用相同資料訓練了一個同等規模的傳統全注意力模型，並在評分前事先寫下勝出條件。混合模型在所選品質指標上勝出 0.81%，在下游任務上持平，產生的 4 位元檔案縮小 6.3%，且在 2048 個詞元上下文時解碼速度快 1.76 倍，對比同等規模的外部模型則快 2.08 倍。在所有測量中，速度優勢在空上下文時接近於零，並隨長度增加而增長；這正是該機制所預期的，而僅是更精簡的模型不會呈現這種現象。簡單的頻寬計算只預測 1.17 倍，因此單靠記憶體容量無法解釋這項差距。我們也報告了未奏效的部分：4 位元造成的品質代價無法緩解；約一半的卷積通道最終呈惰性，且無法移除；詞彙表大小超過此模型規模所需。

English

Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it. The result keeps full attention in only 6 of its 18 blocks. The other 12 use short convolutions whose memory is two timesteps wide no matter how long the conversation gets, so two thirds of the network never re-reads a growing cache. Trained from scratch on 59.9B tokens, the model scores 47.31 on a five-task benchmark against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M, all trained on three to six times more data, and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685. To check the architecture rather than the training recipe, we trained a conventional all-attention model of the same size on the same data, and wrote down the winning condition before scoring either. The hybrid won the chosen quality metric by 0.81%, matched it on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76x faster at 2048 tokens of context, 2.08x against an external model of similar size. In every measurement the speed advantage is near zero at an empty context and grows with length, which is what the mechanism predicts and what a merely leaner model would not show. A simple bandwidth calculation predicts only 1.17x, so memory volume alone does not explain the gap. We also report what did not work: an unmitigated 4-bit quality cost, roughly half the convolution channels ending up inert and impossible to remove, and a vocabulary larger than this model size warrants.