API-BLEND:用於訓練和基準測試API LLMs 的全面語料庫
API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs
February 23, 2024
作者: Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, Luis A. Lastras
cs.AI
摘要
在有效地使用工具和外部應用程式介面(API)來規劃和完成任務的過程中,對於大型語言模型(LLMs)的需求正在增加。因此,對於能夠獲取涉及工具/API 調用的足夠量的訓練和測試數據的方法引起了極大興趣。出現了兩條主要研究路線來應對這一挑戰。第一條路線專注於合成數據生成技術,而第二條則涉及策劃與任務相關的數據集,這些數據集可以轉換為基於 API / 工具的任務。在本文中,我們專注於識別、策劃和轉換現有數據集的任務,並引入 API-BLEND,這是一個用於訓練和系統性測試工具增強型 LLMs 的大型語料庫。這些數據集模擬了涉及 API 任務的現實情境,例如 API / 工具檢測、槽填充以及檢測到的 API 的排序。我們展示了 API-BLEND 數據集在訓練和基準測試方面的實用性。
English
There is a growing need for Large Language Models (LLMs) to effectively use
tools and external Application Programming Interfaces (APIs) to plan and
complete tasks. As such, there is tremendous interest in methods that can
acquire sufficient quantities of train and test data that involve calls to
tools / APIs. Two lines of research have emerged as the predominant strategies
for addressing this challenge. The first has focused on synthetic data
generation techniques, while the second has involved curating task-adjacent
datasets which can be transformed into API / Tool-based tasks. In this paper,
we focus on the task of identifying, curating, and transforming existing
datasets and, in turn, introduce API-BLEND, a large corpora for training and
systematic testing of tool-augmented LLMs. The datasets mimic real-world
scenarios involving API-tasks such as API / tool detection, slot filling, and
sequencing of the detected APIs. We demonstrate the utility of the API-BLEND
dataset for both training and benchmarking purposes.