集群，路由，升级：成本感知的大语言模型服务的级联框架

摘要

在大型语言模型（LLM）的高效生产部署中，准确性与成本之间必然存在权衡。操作者通常默认使用单一模型，这导致简单查询成本过高，而困难查询则可能因模型能力不足而效果欠佳。为解决这一挑战，我们提出了一种两阶段级联解决方案。第一阶段对传入查询进行聚类，并将每个聚类分配给成本效益最优的模型。该路由过程的成本预算由一个可解释的超参数设定，并通过离线方式调优。第二阶段引入质量估计（QE）级联：当第一阶段输出被判定为低质量时，查询将被升级至更强的模型。这确保了只有困难或低置信度的案例才会被分配到昂贵模型。在测试数据集上，该级联系统在保留最强模型97-99%准确率的同时，降低了每输出令牌时间（TPOT）。该系统仅需任务正确性标注，并能适应模型池的变化而无需手动重新配置。

English

Efficient deployment of large language models (LLMs) in production forces a trade-off between accuracy and cost. Operators often default to a single model that is either expensive for easy queries or insufficient for hard ones. To address this challenge, we propose a two-stage cascaded solution. Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model. The cost budget for this routing process is set by an interpretable hyperparameter, tuned offline. Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model. This ensures only hard or low-confidence cases reach the expensive models. On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT). It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.