API-BLEND:用于训练和基准测试API LLMs 的综合语料库
API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs
February 23, 2024
作者: Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, Luis A. Lastras
cs.AI
摘要
随着对大型语言模型(LLMs)有效利用工具和外部应用程序接口(APIs)以规划和完成任务的需求不断增长。因此,人们对能够获取涉及工具/API调用的足够数量的训练和测试数据的方法表现出极大兴趣。针对解决这一挑战,出现了两条主要的研究方向。第一条侧重于合成数据生成技术,而第二条涉及策划与任务相关的数据集,这些数据集可以转化为基于API/工具的任务。本文聚焦于识别、策划和转化现有数据集的任务,并引入了API-BLEND,一个用于训练和系统测试工具增强型LLMs的大型语料库。这些数据集模拟涉及API任务的现实场景,如API/工具检测、槽填充以及检测到的API的排序。我们展示了API-BLEND数据集在训练和基准测试方面的实用性。
English
There is a growing need for Large Language Models (LLMs) to effectively use
tools and external Application Programming Interfaces (APIs) to plan and
complete tasks. As such, there is tremendous interest in methods that can
acquire sufficient quantities of train and test data that involve calls to
tools / APIs. Two lines of research have emerged as the predominant strategies
for addressing this challenge. The first has focused on synthetic data
generation techniques, while the second has involved curating task-adjacent
datasets which can be transformed into API / Tool-based tasks. In this paper,
we focus on the task of identifying, curating, and transforming existing
datasets and, in turn, introduce API-BLEND, a large corpora for training and
systematic testing of tool-augmented LLMs. The datasets mimic real-world
scenarios involving API-tasks such as API / tool detection, slot filling, and
sequencing of the detected APIs. We demonstrate the utility of the API-BLEND
dataset for both training and benchmarking purposes.