Graph Machine：透過邊邁向更佳的預訓練

摘要

我們提出圖機器（Graph Machine，GM），一種維護大小為 O(n) 的狀態、並透過稀疏且動態的路由存取該狀態的架構。不同於具有固定大小狀態或稀疏但靜態路由的方法，GM 在其稀疏層中保持 O(n) 複雜度，而不將可能可存取的狀態大小限制為 O(1)。相反地，GM 使用邊（edges）——類似指標的物件，這些物件透過類似指標追逐的參照機制以可微分方式更新。我們將 Qwen3-0.6B 中 75% 的稠密 Transformer 層替換為 GM 稀疏層，並從頭在 15.7B 個 token 上進行預訓練。在每個稀疏層中，每個 KV 頭僅檢索 4,096 個 token 中的 2 個時，損失僅略微退化；檢索 4 個時，最佳模型的損失略有改善。

English

We introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves O(n) complexity in its sparse layers without restricting the potentially accessible state size to O(1). Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.