AutoRAG-HP：用於檢索增強生成的自動在線超參數調整。

AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation

June 27, 2024

作者: Jia Fu, Xiaoting Qin, Fangkai Yang, Lu Wang, Jue Zhang, Qingwei Lin, Yubo Chen, Dongmei Zhang, Saravan Rajmohan, Qi Zhang

cs.AI

摘要

最近大型語言模型的進步已經改變了機器學習/人工智慧的發展，需要重新評估檢索增強生成（RAG）系統的AutoML原則。為應對RAG中的超參數優化和在線適應的挑戰，我們提出了AutoRAG-HP框架，將超參數調整定義為在線多臂擇機問題，並引入了一種新穎的兩級階層多臂擇機（Hier-MAB）方法，以有效探索大型搜索空間。我們在調整超參數方面進行了廣泛實驗，如頂部k檢索文檔、提示壓縮比和嵌入方法，使用ALCE-ASQA和自然問題數據集。我們的評估從聯合優化所有三個超參數中顯示，基於多臂擇機的在線學習方法可以實現對於搜索空間中明顯梯度的情況下，Recall@5約為0.8，僅使用Grid Search方法所需的LLM API調用的約20%。此外，所提出的Hier-MAB方法在更具挑戰性的優化情境中優於其他基準。代碼將在https://aka.ms/autorag提供。

English

Recent advancements in Large Language Models have transformed ML/AI development, necessitating a reevaluation of AutoML principles for the Retrieval-Augmented Generation (RAG) systems. To address the challenges of hyper-parameter optimization and online adaptation in RAG, we propose the AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical MAB (Hier-MAB) method for efficient exploration of large search spaces. We conduct extensive experiments on tuning hyper-parameters, such as top-k retrieved documents, prompt compression ratio, and embedding methods, using the ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly optimization all three hyper-parameters demonstrate that MAB-based online learning methods can achieve Recall@5 approx 0.8 for scenarios with prominent gradients in search space, using only sim20% of the LLM API calls required by the Grid Search approach. Additionally, the proposed Hier-MAB approach outperforms other baselines in more challenging optimization scenarios. The code will be made available at https://aka.ms/autorag.