盲人摸象：探討大型語言模型在長尾分歧知識下的認知短視

摘要

事實問答（QA）通常假定存在單一標準答案，因而掩蓋了大型語言模型（LLMs）是否保留長尾事實分歧敘述的問題。為填補此缺口，我們提出 ElephantBench，這是一個包含 1,094 道問題的閉卷知識探針，透過可稽核的圖形化管線生成。該管線從低曝光度的網路語料庫中檢索相關文件，辨識自然發生的分歧，並將其轉化為多敘述問答紀錄。每個答案都會對照原始文件與權威的公開網路來源進行驗證，再由人工標註者審查。在 32 個模型中，即使是最強大的模型，也僅在 52.4% 的問題上能還原兩種敘述；而在其餘幾乎所有問題中，它只記得一種敘述，卻遺漏另一種。擴大模型規模與增加推論時的推理能提升召回率，但無法消除這種不完整性。語料庫分析進一步顯示，曝光不平衡有利於主流敘述，而少數方曝光度越高，則與更完整的召回表現相關。這些發現確立了 ElephantBench 作為可重現的知識探針，用於診斷參數化記憶中的認知近視。更廣泛而言，我們基於圖形的基準建構管線提供了一種高效且可擴展的方式，能將長尾語料庫轉化為可追溯來源的知識探針，以支持評估與提升下一代大型語言模型認知嚴謹性的相關工作。程式碼已公開於 https://github.com/Tencent/ElephantBench。

English

Factual question answering (QA) typically assumes a single canonical answer, obscuring whether large language models (LLMs) retain divergent accounts of long-tail facts. To address this gap, we introduce ElephantBench, a closed-book knowledge probe comprising 1,094 questions generated through an auditable graph-based pipeline. The pipeline retrieves related documents from a low-exposure web corpus, identifies naturally occurring disagreements, and converts them into multi-account QA records. Each answer is verified against the originating documents and authoritative public web sources and is then reviewed by human annotators. Across 32 models, even the strongest model recovers both accounts on only 52.4% of questions, while on nearly all remaining questions it recalls one account but omits the other. Scaling model size and inference-time reasoning improve recall but do not eliminate this incompleteness. Corpus analysis further shows that exposure imbalance favors the dominant account, whereas greater minority-side exposure is associated with more complete recall. These findings establish ElephantBench as a reproducible knowledge probe for diagnosing epistemic myopia in parametric memory. More broadly, our graph-based benchmark construction pipeline provides an efficient and scalable way to turn long-tail corpora into source-traceable knowledge probes, supporting efforts to evaluate and advance the epistemic rigour of next-generation LLMs. Code is available at https://github.com/Tencent/ElephantBench.