FreeToken：面向边缘原生的高效MoE推理服务，采用带宽自适应执行

摘要

前沿开放权重模型日益普及，但其服务部署仍主要假定依托数据中心基础设施。我们提出FreeToken，一个边缘原生的MoE服务系统，它将个人机器视为统一的弹性推理平台，而非仅仅是一块小型GPU。FreeToken围绕本地AI的两个现实协同设计整个服务栈，涵盖模型布局与加载、专家驻留、CPU--GPU执行、智能体状态复用以及运行时内存管理：智能体工作负载会持续改变其执行模式，而边缘硬件暴露出的异构资源在不同机器间存在差异。FreeToken不采用固定的卸载策略，而是持续将计算和模型状态映射到实际可用的资源上。FreeToken支持超过20个MoE模型以及真实的编码和工具使用型智能体，硬件范围从8GB笔记本电脑GPU到单块工作站GPU。更重要的是，它改变了这些机器实际可服务的模型规模：从笔记本电脑上的35B模型，到游戏台式机上的284B模型，再到单块工作站GPU上的753B GLM-5.2。FreeToken将开放权重转化为可部署的本地软件，使用户已有的机器成为前沿规模智能的实用平台。我们在flashml.ai发布该系统。

English

Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.