利用大型語言模型進行領域特定語言生成的語法提示

Grammar Prompting for Domain-Specific Language Generation with Large Language Models

May 30, 2023

作者: Bailin Wang, Zi Wang, Xuezhi Wang, Yuan Cao, Rif A. Saurous, Yoon Kim

cs.AI

摘要

大型語言模型（LLMs）可以從僅有少量上下文示例中學習執行各種自然語言任務。然而，對於從高度結構化語言（例如，從語義解析到複雜的特定領域語言）生成字符串，LLM從僅有少數示例中泛化是具有挑戰性的。我們探索了語法提示作為一種簡單方法，讓LLMs能夠在上下文學習期間使用外部知識和特定領域約束，透過以巴科斯-瑙爾范式（BNF）表示的語法來表達。語法提示通過將每個示範示例與一個專門的語法相結合，該語法最少程度上足以生成特定輸出示例，其中專門的語法是完整DSL語法的子集。對於推論，LLM首先根據測試輸入預測BNF語法，然後根據語法規則生成輸出。實驗表明，語法提示可以使LLMs在各種DSL生成任務上表現出競爭力，包括語義解析（SMCalFlow、Overnight、GeoQuery）、PDDL規劃，甚至分子生成（SMILES）。

English

Large language models (LLMs) can learn to perform a wide range of natural language tasks from just a handful of in-context examples. However, for generating strings from highly structured languages (e.g., semantic parsing to complex domain-specific languages), it is challenging for the LLM to generalize from just a few exemplars. We explore grammar prompting as a simple approach for enabling LLMs to use external knowledge and domain-specific constraints, expressed through a grammar expressed in Backus--Naur Form (BNF), during in-context learning. Grammar prompting augments each demonstration example with a specialized grammar that is minimally sufficient for generating the particular output example, where the specialized grammar is a subset of the full DSL grammar. For inference, the LLM first predicts a BNF grammar given a test input, and then generates the output according to the rules of the grammar. Experiments demonstrate that grammar prompting can enable LLMs to perform competitively on a diverse set of DSL generation tasks, including semantic parsing (SMCalFlow, Overnight, GeoQuery), PDDL planning, and even molecule generation (SMILES).