回溯：檢索查詢的原因

Backtracing: Retrieving the Cause of the Query

March 6, 2024

作者: Rose E. Wang, Pawan Wirawarn, Omar Khattab, Noah Goodman, Dorottya Demszky

cs.AI

摘要

許多線上內容門戶允許用戶提問以補充他們的理解（例如，對講座的理解）。儘管信息檢索（IR）系統可能為此類用戶查詢提供答案，但它們並不直接幫助內容創作者——例如希望改進內容的講師——識別引發用戶提問的段落。我們引入了回溯任務，其中系統檢索最有可能引發用戶查詢的文本段落。我們對三個現實世界領域進行了形式化，這些領域中回溯對於改進內容傳遞和溝通至關重要：理解講座領域中學生困惑的原因、新聞文章領域中讀者好奇心的原因，以及對話領域中用戶情感的原因。我們評估了流行的信息檢索方法和語言建模方法的零-shot表現，包括雙編碼器、重新排序和基於概率的方法以及ChatGPT。儘管傳統的IR系統檢索語義相關信息（例如，對於查詢“多次投影是否仍然會導致相同點”的“投影矩陣”細節），但它們通常會遺漏因果相關的上下文（例如，講師表示“投影兩次得到的答案與一次投影相同”）。我們的結果顯示，在回溯方面還有改進的空間，並需要新的檢索方法。我們希望我們的基準測試有助於改進未來的回溯檢索系統，從而產生能夠完善內容生成並識別影響用戶查詢的語言觸發的系統。我們的代碼和數據是開源的：https://github.com/rosewang2008/backtracing。

English

Many online content portals allow users to ask questions to supplement their understanding (e.g., of lectures). While information retrieval (IR) systems may provide answers for such user queries, they do not directly assist content creators -- such as lecturers who want to improve their content -- identify segments that _caused_ a user to ask those questions. We introduce the task of backtracing, in which systems retrieve the text segment that most likely caused a user query. We formalize three real-world domains for which backtracing is important in improving content delivery and communication: understanding the cause of (a) student confusion in the Lecture domain, (b) reader curiosity in the News Article domain, and (c) user emotion in the Conversation domain. We evaluate the zero-shot performance of popular information retrieval methods and language modeling methods, including bi-encoder, re-ranking and likelihood-based methods and ChatGPT. While traditional IR systems retrieve semantically relevant information (e.g., details on "projection matrices" for a query "does projecting multiple times still lead to the same point?"), they often miss the causally relevant context (e.g., the lecturer states "projecting twice gets me the same answer as one projection"). Our results show that there is room for improvement on backtracing and it requires new retrieval approaches. We hope our benchmark serves to improve future retrieval systems for backtracing, spawning systems that refine content generation and identify linguistic triggers influencing user queries. Our code and data are open-sourced: https://github.com/rosewang2008/backtracing.