GORGO：跨区域网络感知的LLM服务在线调优

摘要

越来越多的LLM推理服务将客户端请求代理到分布全球的引擎副本。负载均衡策略在优化延迟和TTFT等指标时，必须综合考虑KV-cache局部性、副本负载及网络延迟变化等因素。然而现有系统的代价模型仅评估部分因素，导致副本间负载与KV-cache分布不均。我们提出GORGO代理架构，通过可调参数全面融合网络延迟、预填充代价与排队延迟。由于LMSYS-Chat1M和WildChat-4.8M等开源聊天数据集缺乏长上下文、高前缀复用数据，我们基于长上下文生产元数据发布了合成数据集ART-Chat-2.5M。在ART-Chat-2.5M的调优窗口上，进化策略指导GORGO策略参数直接优化p95 TTFT。在保留的评估窗口上，我们固定调优所得参数值，相较于简单会话亲和与前缀缓存等基准负载均衡策略，p95 TTFT改善6.9-15.5%，p95端到端延迟改善14.3-30.9%。代码和ART-Chat-2.5M数据集见https://github.com/Arcadia-Research-Team/GORGO。

English

Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M, evolutionary strategies guide the GORGO policy's parameters to directly optimize p95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improve p95 TTFT by 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baseline load-balancing policies such as simple session affinity and prefix-cache. The code and ART-Chat-2.5M dataset can be found at https://github.com/Arcadia-Research-Team/GORGO.