超越以相关性为中心的检索：基于评分标准的文档集选择与排序

摘要

随着大语言模型和AI智能体成为搜索结果的主要消费方，文档集质量决定了下游生成任务的上限。然而，现有评估系统仍局限于对文档进行独立评分并通过nDCG聚合，忽略了文档间的交互关系（冗余、冲突、互补性），也无法回答“为何某个文档集优于另一个”的问题。为解决这些问题，我们提出了一套完整的评估-诊断-优化框架。我们设计了SetwiseEvalKit，一个覆盖短文本与长文本场景的三级九维度文档集评估基准，包含约2.8万条高质量评估准则。我们系统评估了12种重排序方法：即使最优方法覆盖率也不超过45%，跨文档协调维度普遍薄弱，且没有一种方法能在两种场景下均保持顶尖性能。在此基础上，我们提出Rubric4Setwise——一种无训练方法，将基于准则的评估标准转化为文档集选择信号，以更少的文档和搜索轮次实现了最优的下游生成性能。它是唯一在两种场景下均保持最先进结果的方法，验证了从评估到优化的闭环有效性。

English

As large language models and AI agents become the primary consumers of search results, document set quality determines the upper bound of downstream generation. Yet existing evaluation systems remain confined to scoring documents independently and aggregating via nDCG, ignoring inter-document interactions (redundancy, conflict, complementarity) and unable to answer what makes one document set better than another. To address these issues, we propose a complete evaluate-diagnose-optimize framework. We design SetwiseEvalKit, a three-level, nine-dimension document set evaluation benchmark covering both short-form and long-form scenarios, comprising approximately 28K high-quality evaluation rubrics. We systematically evaluate 12 rerankers: even the best method achieves no more than 45% coverage, cross-document coordination dimensions are universally weak, and no single method maintains top performance across both settings. Building on this, we propose Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both scenarios, validating the effectiveness of closing the loop from evaluation to optimization.