盲人摸象：探究大语言模型在长尾分歧知识下的认知短视

摘要

事实性问答（QA）通常假设存在单一标准答案，这掩盖了大语言模型（LLMs）是否保留长尾事实的多元叙述这一问题。为弥补这一空白，我们提出ElephantBench，一个包含1,094个问题的闭卷知识探针，所有问题均通过可审计的基于图结构的流水线生成。该流水线从低曝光网络语料中检索相关文档，识别其中自然存在的分歧，并将其转化为多版本问答记录。每条答案均对照原始文档及权威公开网络来源进行核验，并由人工标注者进行审查。在32个模型的测试中，即使最强的模型也仅在52.4%的问题上同时还原两个版本，而在几乎所有剩余问题上，模型仅回忆起其中一个版本而遗漏另一个。扩大模型规模与增强推理时思考虽能提升召回率，但无法消除这种不完整性。语料分析进一步表明，曝光不均衡偏向主导版本，而少数方曝光越多，越有助于完整召回。这些发现使ElephantBench成为一个可复现的知识探针，用于诊断参数化记忆中的认知短视。更广泛地说，我们基于图结构的基准构建流水线提供了一种高效且可扩展的方式，将长尾语料转化为可溯源的知识探针，为评估和提升下一代LLM的认知严谨性提供支持。代码已开源至 https://github.com/Tencent/ElephantBench。

English

Factual question answering (QA) typically assumes a single canonical answer, obscuring whether large language models (LLMs) retain divergent accounts of long-tail facts. To address this gap, we introduce ElephantBench, a closed-book knowledge probe comprising 1,094 questions generated through an auditable graph-based pipeline. The pipeline retrieves related documents from a low-exposure web corpus, identifies naturally occurring disagreements, and converts them into multi-account QA records. Each answer is verified against the originating documents and authoritative public web sources and is then reviewed by human annotators. Across 32 models, even the strongest model recovers both accounts on only 52.4% of questions, while on nearly all remaining questions it recalls one account but omits the other. Scaling model size and inference-time reasoning improve recall but do not eliminate this incompleteness. Corpus analysis further shows that exposure imbalance favors the dominant account, whereas greater minority-side exposure is associated with more complete recall. These findings establish ElephantBench as a reproducible knowledge probe for diagnosing epistemic myopia in parametric memory. More broadly, our graph-based benchmark construction pipeline provides an efficient and scalable way to turn long-tail corpora into source-traceable knowledge probes, supporting efforts to evaluate and advance the epistemic rigour of next-generation LLMs. Code is available at https://github.com/Tencent/ElephantBench.