AutoRAG-HP：用于检索增强生成的自动在线超参数调整

AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation

June 27, 2024

作者: Jia Fu, Xiaoting Qin, Fangkai Yang, Lu Wang, Jue Zhang, Qingwei Lin, Yubo Chen, Dongmei Zhang, Saravan Rajmohan, Qi Zhang

cs.AI

摘要

最近大型语言模型的进展已经改变了机器学习/人工智能的发展，需要重新评估用于检索增强生成（RAG）系统的AutoML原则。为了解决RAG中的超参数优化和在线适应的挑战，我们提出了AutoRAG-HP框架，将超参数调整构建为在线多臂老虎机（MAB）问题，并引入了一种新颖的两级层次MAB（Hier-MAB）方法，以有效探索大搜索空间。我们在调整超参数方面进行了大量实验，例如顶部k个检索文档、提示压缩比和嵌入方法，使用ALCE-ASQA和自然问题数据集。我们的评估结果显示，联合优化这三个超参数，基于MAB的在线学习方法可以在具有显著梯度的搜索空间中实现大约0.8的Recall@5，仅使用Grid Search方法所需的LLM API调用的约20%。此外，所提出的Hier-MAB方法在更具挑战性的优化场景中优于其他基线。代码将在https://aka.ms/autorag上提供。

English

Recent advancements in Large Language Models have transformed ML/AI development, necessitating a reevaluation of AutoML principles for the Retrieval-Augmented Generation (RAG) systems. To address the challenges of hyper-parameter optimization and online adaptation in RAG, we propose the AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical MAB (Hier-MAB) method for efficient exploration of large search spaces. We conduct extensive experiments on tuning hyper-parameters, such as top-k retrieved documents, prompt compression ratio, and embedding methods, using the ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly optimization all three hyper-parameters demonstrate that MAB-based online learning methods can achieve Recall@5 approx 0.8 for scenarios with prominent gradients in search space, using only sim20% of the LLM API calls required by the Grid Search approach. Additionally, the proposed Hier-MAB approach outperforms other baselines in more challenging optimization scenarios. The code will be made available at https://aka.ms/autorag.