ObfusQAte：一個評估LLM在混淆事實問答中魯棒性的框架提案

摘要

大型語言模型（LLMs）的迅速普及，極大地促進了能夠進行事實問答（QA）的公平AI系統的發展。然而，目前尚無已知研究測試LLMs在面對模糊化問題時的穩健性。為系統性地評估這些限制，我們提出了一種新技術——ObfusQAte，並基於此引入了ObfusQA，這是一個首創的、全面的框架，具有多層次的模糊化級別，旨在從三個不同維度檢驗LLM的能力：（i）命名實體間接性，（ii）干擾項間接性，以及（iii）上下文過載。通過捕捉語言中的這些細微差別，ObfusQA為評估LLM的穩健性和適應性提供了一個全面的基準。我們的研究發現，當面對這些日益細膩的變體時，LLMs往往會失敗或產生虛構的回應。為推動這一方向的研究，我們將ObfusQAte公開提供。

English

The rapid proliferation of Large Language Models (LLMs) has significantly contributed to the development of equitable AI systems capable of factual question-answering (QA). However, no known study tests the LLMs' robustness when presented with obfuscated versions of questions. To systematically evaluate these limitations, we propose a novel technique, ObfusQAte and, leveraging the same, introduce ObfusQA, a comprehensive, first of its kind, framework with multi-tiered obfuscation levels designed to examine LLM capabilities across three distinct dimensions: (i) Named-Entity Indirection, (ii) Distractor Indirection, and (iii) Contextual Overload. By capturing these fine-grained distinctions in language, ObfusQA provides a comprehensive benchmark for evaluating LLM robustness and adaptability. Our study observes that LLMs exhibit a tendency to fail or generate hallucinated responses when confronted with these increasingly nuanced variations. To foster research in this direction, we make ObfusQAte publicly available.

ObfusQAte：一個評估LLM在混淆事實問答中魯棒性的框架提案

ObfusQAte: A Proposed Framework to Evaluate LLM Robustness on Obfuscated Factual Question Answering

摘要

Support