RL-Index：用於檢索索引推理的強化學習

摘要

检索外部知识对于解决现实任务至关重要，然而当查询与其相关知识之间的关系涉及超越表层语义或词汇匹配的隐式和复杂推理时（例如，依赖相同定理的数学问题或需要深度推理的编码任务），这仍然具有挑战性。现有方法主要依赖查询端推理（如查询改写），这引入了显著的在线延迟，且未能充分利用对知识语料库本身进行推理的机会（即索引端推理）。在本文中，我们提出了RL-Index，一个将检索索引推理构建为强化学习问题的智能体索引框架。RL-Index并不在查询时执行推理，而是将推理转移至索引阶段，通过用LLM生成的推理依据对文档进行增强，这些推理依据显式编码了潜在的查询-知识关系。为了优化这些推理依据的质量，我们采用了群组相对策略优化（GRPO），并将检索相似度作为可验证的奖励信号，从而能够直接优化索引决策以提升检索效果。在BRIGHT基准上的大量实验表明，RL-Index能一致地提升检索性能和下游问答能力，同时显著降低在线推理延迟。此外，所学到的推理依据增强可泛化至多种检索器和生成器，突显其作为跨不同检索系统的即插即用索引策略的稳健性。

English

Retrieving external knowledge is essential for solving real-world tasks, yet it remains challenging when the relationship between a query and its relevant knowledge involves implicit and complex reasoning beyond surface-level semantic or lexical matching (e.g., mathematical problems relying on the same theorem or coding requiring deep reasoning). Existing approaches primarily rely on query-side reasoning (e.g., query rewriting), which introduces significant online latency and underutilizes the opportunity to perform reasoning over the knowledge corpus itself (i.e., index-side reasoning). In this paper, we propose RL-Index, an agentic indexing framework that formulates retrieval index reasoning as a reinforcement learning problem. Instead of performing reasoning at query time, RL-Index shifts reasoning to the indexing stage by augmenting documents with LLM-generated rationales that explicitly encode the latent query-knowledge relationship. To optimize the quality of these rationales, we employ Group Relative Policy Optimization (GRPO) and use retrieval similarity as a verifiable reward signal, enabling direct optimization of indexing decisions for retrieval effectiveness. Extensive experiments on the BRIGHT benchmark demonstrate that RL-Index consistently improves both retrieval and downstream question-answering performance, while significantly reducing online inference latency. Moreover, the learned rationale augmentation generalizes across diverse retrievers and generators, highlighting its robustness as a plug-and-play indexing strategy across different retrieval systems.