AutoMix：自动混合语言模型

AutoMix: Automatically Mixing Language Models

October 19, 2023

作者: Aman Madaan, Pranjal Aggarwal, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Mausam, Manaal Faruqui

cs.AI

摘要

现在，大型语言模型(LLMs)以各种规模和配置从云API提供商处提供。虽然这种多样性提供了广泛的选择，但有效利用这些选项以优化计算成本和性能仍然具有挑战性。在这项工作中，我们提出了AutoMix，一种策略性地将查询路由到更大的LM的方法，这取决于从较小LM的输出的近似正确性。AutoMix的核心是一种少样本自我验证机制，它可以估计自身输出的可靠性，而无需进行训练。考虑到验证可能存在噪声，我们在AutoMix中使用一个元验证器来提高这些评估的准确性。我们在五个基于上下文推理数据集上使用LLAMA2-13/70B进行的实验表明，AutoMix超越了已建立的基准线，将每单位成本的增量收益提高了高达89%。我们的代码和数据可在https://github.com/automix-llm/automix 上获得。

English

Large language models (LLMs) are now available in various sizes and configurations from cloud API providers. While this diversity offers a broad spectrum of choices, effectively leveraging the options to optimize computational cost and performance remains challenging. In this work, we present AutoMix, an approach that strategically routes queries to larger LMs, based on the approximate correctness of outputs from a smaller LM. Central to AutoMix is a few-shot self-verification mechanism, which estimates the reliability of its own outputs without requiring training. Given that verifications can be noisy, we employ a meta verifier in AutoMix to refine the accuracy of these assessments. Our experiments using LLAMA2-13/70B, on five context-grounded reasoning datasets demonstrate that AutoMix surpasses established baselines, improving the incremental benefit per cost by up to 89%. Our code and data are available at https://github.com/automix-llm/automix.