FreeToken：具頻寬自適應執行能力的高效邊緣原生MoE服務

摘要

前沿開放權重模型日益普及，但服務這些模型仍大多預設在資料中心基礎設施上執行。我們提出 FreeToken，一套邊緣原生的 MoE 服務系統，它不將個人機器視為小型 GPU，而是視為一個統一且具彈性的推論平台。FreeToken 圍繞在地 AI 的兩項現實，共同設計了完整的服務堆疊，包括模型佈局與載入、專家駐留、CPU-GPU 執行、代理狀態重用，以及執行時期記憶體管理：代理工作負載會持續改變其執行模式，而邊緣硬體暴露了異質資源，其資源平衡因機器而異。FreeToken 不採用固定的卸載策略，而是持續將計算與模型狀態映射到實際可用的資源上。FreeToken 支援超過 20 個 MoE 模型，以及真實的編碼與工具使用代理，涵蓋從 8GB 筆電 GPU 到單一工作站 GPU 的各種硬體。更重要的是，它改變了這些機器實際能服務的規模：從筆電上的 35B 模型，到遊戲桌上型電腦上的 284B 模型，再到單一工作站 GPU 上的 753B GLM-5.2。FreeToken 將開放權重轉化為可部署的本地軟體，讓使用者既有的機器成為能承載前沿規模智慧的实际平台。我們在 flashml.ai 釋出此系統。

English

Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.