ELDR：專家局部性感知的解碼路由用於PD分離的MoE服務

摘要

在预填充-解码（PD）分离式大语言模型（LLM）服务中，每个请求在预填充阶段后被分配至某个解码工作节点。现有解码路由器仅平衡负载；对于混合专家（MoE）模型而言，这并不完备：负载相同的工作节点可能产生不同的延迟，因为每个解码步骤都会加载其批次内所有已激活专家的权重。本文提出ELDR，一种面向PD分离式MoE服务的专家局部性感知解码路由器。ELDR利用请求预填充阶段的专家激活信息，构建用于预测该请求在生成过程中将激活的专家的专家签名。离线阶段，平衡K-means算法在解码工作节点间划分签名空间；在线阶段，局部性带路由将每个请求发送至与其签名匹配最佳且负载最低的工作节点。签名缓存与KV缓存以KV块为粒度联合索引，可在前缀缓存机制下保持签名的精确性。ELDR已在vLLM中实现，并在多达40个GPU的部署上进行评估。相较于四种负载均衡基线中的最强方法，ELDR在三个MoE模型和两种工作负载下，将中位每个输出token生成时间（TPOT）降低了5.9%至13.9%，且模型输出保持不变。

English

In prefill-decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existing decode routers balance only load; for mixture-of-experts (MoE) models this is incomplete: equally loaded workers can differ in latency, since each decode step loads the weights of every distinct expert its batch activates. We present ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving. From a request's prefill expert activations, ELDR builds an expert signature predicting the experts it will activate during generation. Offline, balanced K-means partitions signature space across decode workers; online, locality-band routing sends each request to the least-loaded worker among those best matching its signature. A signature cache, co-indexed with the KV cache at KV-block granularity, keeps signatures exact under prefix caching. Implemented in vLLM and evaluated on deployments of up to 40 GPUs, ELDR reduces median TPOT by 5.9-13.9% over the strongest of four load-balancing baselines across three MoE models and two workloads, with model outputs unchanged.