統一基於上下文集成之共形語言任務

摘要

許多自然語言處理任務，例如摘要和抽取式問答，皆可化約為在兩個約束之下從文件中檢索相關內容：其一是覆蓋率，即保留足夠的相關資訊以達成某目標；其二是簡潔性，即移除盡可能多的不相關資訊。共形預測方法已被用來保證覆蓋率，並須透過設計評分函數來最佳化簡潔性。當前最先進的評分函數使用人工設計的大型語言模型（LLM）提示詞，要求模型對內容的重要性進行評分；然而，人工提示工程既耗費人力，又具任務特異性。我們提出共形相關性框架（Conformal Relevance framework），利用上下文學習的範例篩選與集成方法來建構評分函數，使其能在最少人工投入下維持覆蓋率並增進簡潔性。我們在七個自然語言處理任務上展示此框架的應用，並在理論上研究多樣性對集成共形分數的影響：我們提出一個互補性條件，用以刻畫集成何時能改善最差情況下的句子分數，並給出一個針對集成改進的飽和界限。

English

Many NLP tasks, such as summarization and extractive question answering, reduce to retrieving relevant content from documents under two constraints: coverage, retaining enough pertinent information to achieve some goal, and conciseness, removing as much irrelevant information as possible. Conformal prediction methods have been used to guarantee coverage, and must be optimized for conciseness through design of a score function. State-of-the-art scoring functions use hand-engineered LLM prompts asking the model to rate the importance of content, but manual prompt engineering is labor-intensive and task-specific. We introduce the Conformal Relevance framework which uses in-context learning example curation and ensembling to create a score function which maintains coverage while improving conciseness with minimal manual input. We demonstrate this framework's application on seven NLP tasks, and also theoretically study the impact of diversity for ensembled conformal scores, giving a complementarity condition that characterizes when ensembling improves worst-case sentence scores, and a saturation bound on ensemble improvement.