集群、路由、升階：成本感知大型語言模型服務的串聯式框架

摘要

在生產環境中高效部署大型語言模型（LLMs）時，必須在準確性與成本之間取得權衡。營運者通常預設使用單一模型，但這可能導致簡單查詢成本過高，或難題查詢效果不足。為解決此挑戰，我們提出了一個兩階段級聯解決方案。第一階段將傳入查詢進行分群，並為每個群組分配其最具成本效益的模型。此路由過程的成本預算由一個可解釋的超參數設定，並透過離線調校。第二階段則加入品質估計（QE）級聯：當第一階段的輸出被判定為低品質時，該查詢將升級至更強的模型。這確保只有困難或低信心的案例才會用到昂貴模型。在測試資料集上，此級聯系統保留了最強模型 97-99% 的準確性，同時降低了每輸出 Token 時間（TPOT）。該方法僅需任務正確性標籤，且能在無需手動重新配置的情況下適應模型池的變化。

English

Efficient deployment of large language models (LLMs) in production forces a trade-off between accuracy and cost. Operators often default to a single model that is either expensive for easy queries or insufficient for hard ones. To address this challenge, we propose a two-stage cascaded solution. Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model. The cost budget for this routing process is set by an interpretable hyperparameter, tuned offline. Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model. This ensures only hard or low-confidence cases reach the expensive models. On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT). It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.