超越以相關性為中心的檢索：以評分標準為導向的文檔集選取與排序

摘要

隨著大型語言模型與AI代理逐漸成為搜尋結果的主要使用者，文檔集的品質決定了下游生成任務的上限。然而，現有的評估系統仍局限於獨立評分各文檔並透過nDCG進行彙總，忽略了文檔間的交互作用（如冗餘、衝突、互補性），也無法回答為何某個文檔集優於另一個。為解決這些問題，我們提出了一套完整的「評估-診斷-優化」框架。我們設計了SetwiseEvalKit——一個三層次、九維度的文檔集評估基準，涵蓋短文本與長文本兩種場景，包含約28,000條高品質評估準則。我們系統性地評估了12種重排序方法：即使是最佳方法，其覆蓋率也不超過45%；跨文檔協調維度的表現普遍薄弱；且沒有任何一種方法能在兩種場景下同時維持頂尖表現。基於此，我們提出Rubric4Setwise——一種無需訓練的方法，能將基於準則的評估標準轉化為文檔集選擇信號，在減少文檔數量與搜尋輪次的情況下，達到最佳的下游生成表現。此方法是唯一能在兩種場景下皆維持最優成果的方法，驗證了從評估到優化的閉環有效性。

English

As large language models and AI agents become the primary consumers of search results, document set quality determines the upper bound of downstream generation. Yet existing evaluation systems remain confined to scoring documents independently and aggregating via nDCG, ignoring inter-document interactions (redundancy, conflict, complementarity) and unable to answer what makes one document set better than another. To address these issues, we propose a complete evaluate-diagnose-optimize framework. We design SetwiseEvalKit, a three-level, nine-dimension document set evaluation benchmark covering both short-form and long-form scenarios, comprising approximately 28K high-quality evaluation rubrics. We systematically evaluate 12 rerankers: even the best method achieves no more than 45% coverage, cross-document coordination dimensions are universally weak, and no single method maintains top performance across both settings. Building on this, we propose Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both scenarios, validating the effectiveness of closing the loop from evaluation to optimization.