鹦鹉：利用语义变量高效提供基于LLM的应用程序

Parrot: Efficient Serving of LLM-based Applications with Semantic Variable

May 30, 2024

作者: Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, Lili Qiu

cs.AI

摘要

大型语言模型（LLM）的崛起使得基于LLM的应用（又称AI代理或副驾驶员）成为可能，这是一种结合了LLM和传统软件优势的新软件范式。来自不同租户的多样化LLM应用可以使用多个LLM请求设计复杂的工作流程来完成一个任务。然而，它们必须使用当今公共LLM服务提供的过于简化的请求级API，从而丢失了重要的应用级信息。公共LLM服务必须盲目优化单个LLM请求，导致LLM应用的端到端性能不佳。 本文介绍了Parrot，这是一个专注于LLM应用端到端体验的LLM服务系统。Parrot提出了语义变量，这是一种统一的抽象，用于向公共LLM服务公开应用级知识。语义变量在请求的提示中注释输入/输出变量，并在连接多个LLM请求时创建数据管道，为编程LLM应用提供了一种自然的方式。将语义变量暴露给公共LLM服务使其能够执行传统数据流分析，以揭示多个LLM请求之间的相关性。这种相关性为LLM应用的端到端性能开辟了全新的优化空间。广泛的评估表明，Parrot可以在流行且实际的LLM应用用例中实现高达一个数量级的改进。

English

The rise of large language models (LLMs) has enabled LLM-based applications (a.k.a. AI agents or co-pilots), a new software paradigm that combines the strength of LLM and conventional software. Diverse LLM applications from different tenants could design complex workflows using multiple LLM requests to accomplish one task. However, they have to use the over-simplified request-level API provided by today's public LLM services, losing essential application-level information. Public LLM services have to blindly optimize individual LLM requests, leading to sub-optimal end-to-end performance of LLM applications. This paper introduces Parrot, an LLM service system that focuses on the end-to-end experience of LLM-based applications. Parrot proposes Semantic Variable, a unified abstraction to expose application-level knowledge to public LLM services. A Semantic Variable annotates an input/output variable in the prompt of a request, and creates the data pipeline when connecting multiple LLM requests, providing a natural way to program LLM applications. Exposing Semantic Variables to the public LLM service allows it to perform conventional data flow analysis to uncover the correlation across multiple LLM requests. This correlation opens a brand-new optimization space for the end-to-end performance of LLM-based applications. Extensive evaluations demonstrate that Parrot can achieve up to an order-of-magnitude improvement for popular and practical use cases of LLM applications.