RuleChef: 将LLM任务知识锚定于人类可编辑规则

摘要

我们提出了RuleChef框架，它利用大型语言模型（LLMs）为文本分类、命名实体识别（NER）或关系抽取等自然语言处理任务生成可执行的规则。规则基于任务描述和一组标注示例生成，随后根据更多示例以及人类对现有规则的反馈进行迭代改进。RuleChef还可利用给定任务中任意现有模型观察到的输入-输出对来引导规则生成。LLMs仅在学习阶段使用，用于合成规则并根据保留样本上测量到的失败进行迭代修补。该过程最终生成一个快速、确定且可检查的规则系统。我们在分类和NER任务上进行了初步评估，并以Apache 2.0许可证将RuleChef作为开源软件发布。

English

We present RuleChef, a framework that uses large language models (LLMs) to generate executable rules for NLP tasks such as text classification, Named Entity Recognition (NER), or relation extraction. Rules are generated based on a task description and a set of labeled examples, then they are iteratively improved based both on additional examples and on human feedback overexisting rules. RuleChef can also be used to bootstrap rules using the observed input-output pairs from any existing model for a given task. LLMs are used only at learning time, synthesizing rules and iteratively patching them based on failures measured on a held-out split. The result of this process is a fast, deterministic, and inspectable rule system. Preliminary evaluation is performed on both classification and NER tasks. We release RuleChef as open-source software under an Apache 2.0