CypherBench：走向在LLM时代对现代知识图谱的精确检索

CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era

December 24, 2024

作者: Yanlin Feng, Simone Papicchio, Sajjadur Rahman

cs.AI

摘要

从图数据中检索对于增强大型语言模型（LLM）具有至关重要的作用，可以为其提供开放领域知识和私人企业数据，也是最近GraphRAG系统（edge等，2024年）的关键组成部分。尽管在知识图谱和知识库问答方面进行了几十年的研究，但领先的LLM框架（如Langchain和LlamaIndex）对于从现代百科知识图谱（如Wikidata）中检索仅有最低限度的支持。在本文中，我们分析了根本原因，并认为现代RDF知识图谱（如Wikidata、Freebase）对于LLM而言效率较低，原因在于其过大的模式远远超出了典型的LLM上下文窗口，使用资源标识符、重叠的关系类型和缺乏规范化。作为解决方案，我们提出在底层RDF图之上的属性图视图，可以通过Cypher有效地查询。我们在Wikidata上实现了这一想法，并引入了CypherBench，这是第一个具有11个大规模、多领域属性图的基准测试，涵盖780万个实体和超过10,000个问题。为了实现这一目标，我们解决了几个关键挑战，包括开发了一个RDF到属性图转换引擎，创建了一个文本到Cypher任务生成的系统化流程，并设计了新的评估指标。

English

Retrieval from graph data is crucial for augmenting large language models (LLM) with both open-domain knowledge and private enterprise data, and it is also a key component in the recent GraphRAG system (edge et al., 2024). Despite decades of research on knowledge graphs and knowledge base question answering, leading LLM frameworks (e.g. Langchain and LlamaIndex) have only minimal support for retrieval from modern encyclopedic knowledge graphs like Wikidata. In this paper, we analyze the root cause and suggest that modern RDF knowledge graphs (e.g. Wikidata, Freebase) are less efficient for LLMs due to overly large schemas that far exceed the typical LLM context window, use of resource identifiers, overlapping relation types and lack of normalization. As a solution, we propose property graph views on top of the underlying RDF graph that can be efficiently queried by LLMs using Cypher. We instantiated this idea on Wikidata and introduced CypherBench, the first benchmark with 11 large-scale, multi-domain property graphs with 7.8 million entities and over 10,000 questions. To achieve this, we tackled several key challenges, including developing an RDF-to-property graph conversion engine, creating a systematic pipeline for text-to-Cypher task generation, and designing new evaluation metrics.