ELDR：面向预填充-解码分离MoE服务的专家局部性感知解码路由

摘要

在预填充-解码（PD）分离式大语言模型服务中，每个请求在预填充后被分配给一个解码工作节点。现有的解码路由器仅平衡负载；对于混合专家（MoE）模型而言，这并不充分：负载相等的工作节点其延迟可能不同，因为每个解码步骤都需要加载该批次激活的所有不同专家的权重。我们提出了ELDR，一种用于PD分离式MoE服务、具有专家局部性感知能力的解码路由器。ELDR根据请求的预填充专家激活信息，构建一个专家签名，用以预测该请求在生成过程中将激活的专家。在离线阶段，平衡K-means将签名空间划分到各解码工作节点；在线阶段，局部性波段路由将每个请求发送到与其签名最匹配且负载最低的工作节点。一个与KV缓存以KV块粒度协同索引的签名缓存，能确保在前缀缓存下签名的准确性。ELDR已在vLLM中实现，并在多达40个GPU的部署上进行了评估。在三种MoE模型和两种工作负载下，与四种负载均衡基线中最强者相比，ELDR将中位TPOT降低了5.9%-13.9%，且模型输出保持不变。

English

In prefill-decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existing decode routers balance only load; for mixture-of-experts (MoE) models this is incomplete: equally loaded workers can differ in latency, since each decode step loads the weights of every distinct expert its batch activates. We present ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving. From a request's prefill expert activations, ELDR builds an expert signature predicting the experts it will activate during generation. Offline, balanced K-means partitions signature space across decode workers; online, locality-band routing sends each request to the least-loaded worker among those best matching its signature. A signature cache, co-indexed with the KV cache at KV-block granularity, keeps signatures exact under prefix caching. Implemented in vLLM and evaluated on deployments of up to 40 GPUs, ELDR reduces median TPOT by 5.9-13.9% over the strongest of four load-balancing baselines across three MoE models and two workloads, with model outputs unchanged.