Parrot：具有語義變量的基於LLM的應用程序的高效服務

Parrot: Efficient Serving of LLM-based Applications with Semantic Variable

May 30, 2024

作者: Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, Lili Qiu

cs.AI

摘要

大型語言模型（LLM）的崛起使LLM應用（又稱AI代理或共同飛行員）成為可能，這是一種結合LLM和傳統軟件優勢的新軟件範式。來自不同租戶的多樣化LLM應用可以使用多個LLM請求設計複雜的工作流程來完成一個任務。然而，它們必須使用當今公共LLM服務提供的過度簡化的請求級API，從而失去了重要的應用程序級信息。公共LLM服務必須盲目優化個別的LLM請求，導致LLM應用的端到端性能次優。 本文介紹了Parrot，一個專注於LLM應用端到端體驗的LLM服務系統。Parrot提出了語義變量，這是一種統一的抽象，用於向公共LLM服務公開應用程序級知識。語義變量在請求的提示中為輸入/輸出變量進行註釋，並在連接多個LLM請求時創建數據管道，提供了一種自然的方式來編程LLM應用。將語義變量公開給公共LLM服務使其能夠執行傳統數據流分析，以揭示多個LLM請求之間的相關性。這種相關性為LLM應用的端到端性能開啟了全新的優化空間。廣泛的評估表明，Parrot可以為LLM應用的熱門和實用用例實現高達一個數量級的改進。

English

The rise of large language models (LLMs) has enabled LLM-based applications (a.k.a. AI agents or co-pilots), a new software paradigm that combines the strength of LLM and conventional software. Diverse LLM applications from different tenants could design complex workflows using multiple LLM requests to accomplish one task. However, they have to use the over-simplified request-level API provided by today's public LLM services, losing essential application-level information. Public LLM services have to blindly optimize individual LLM requests, leading to sub-optimal end-to-end performance of LLM applications. This paper introduces Parrot, an LLM service system that focuses on the end-to-end experience of LLM-based applications. Parrot proposes Semantic Variable, a unified abstraction to expose application-level knowledge to public LLM services. A Semantic Variable annotates an input/output variable in the prompt of a request, and creates the data pipeline when connecting multiple LLM requests, providing a natural way to program LLM applications. Exposing Semantic Variables to the public LLM service allows it to perform conventional data flow analysis to uncover the correlation across multiple LLM requests. This correlation opens a brand-new optimization space for the end-to-end performance of LLM-based applications. Extensive evaluations demonstrate that Parrot can achieve up to an order-of-magnitude improvement for popular and practical use cases of LLM applications.