GORGO：跨區域網路感知的大型語言模型服務的線上調校

摘要

越来越多地，LLM推理服务将客户端请求代理到全球分布的引擎副本。负载均衡策略在优化延迟和首Token生成时间（TTFT）等指标时，必须综合考虑KV-cache局部性、副本负载和可变网络延迟等因素。然而，现有系统在其成本模型中仅评估了这些因素的子集，导致副本间负载和KV-cache分布不均。我们提出GORGO，一种代理架构，通过可调参数全面考虑网络延迟、预填充成本和排队延迟。由于LMSYS-Chat1M和WildChat-4.8M等开源聊天数据集缺乏长上下文、高前缀复用数据，我们发布了从长上下文生产元数据中生成的合成数据集ART-Chat-2.5M。在ART-Chat-2.5M的调优窗口上，进化策略引导GORGO策略的参数直接优化p95 TTFT。在保留的评估窗口期间，我们固定从调优中学习到的参数值，与简单的会话亲和性和前缀缓存等基线负载均衡策略相比，p95 TTFT改善了6.9-15.5%，p95端到端（E2E）延迟改善了14.3-30.9%。代码和ART-Chat-2.5M数据集可在https://github.com/Arcadia-Research-Team/GORGO 获取。

English

Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M, evolutionary strategies guide the GORGO policy's parameters to directly optimize p95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improve p95 TTFT by 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baseline load-balancing policies such as simple session affinity and prefix-cache. The code and ART-Chat-2.5M dataset can be found at https://github.com/Arcadia-Research-Team/GORGO.