OpenBioRQ：面向智能體的未解決生物醫學研究問題

摘要

一个有效的引用看似证据，但链接能够解析并不意味着被引论文支持该主张。我发现，当前的智能体模型很少捏造引用（超过99%可解析），然而约有15.9%的链接指向了错误的论文。现有基准测试未能发现这一失效模式：当问题有固定答案密钥时，模型可以从该密钥中重现期望的源文献，而非独立验证该源文献是否支持其主张。我提出了\openbiorq{}，这是一个基于检索的智能体基准测试，包含12个领域的12,553个未解决的生物医学研究问题，将开放性问题视为忠实性与弃权探针。据我所知，这是首个将智能体设置（模型必须执行多次工具调用）与无答案密钥的未解决问题相结合的生物医学基准。开放性通过真实的后续证据进行验证，而非依赖模型的参数化知识。难度具有实证性：我以三个开放权重参考模型无法回答的问题作为难度锚点，而非依赖主观难度标签。在这个最难子集上，与难度锚点同系列但被排除在外的模型仅能解决约17%的问题，而三个独立的前沿智能体（Gemini-3-Pro、Opus-4.7、GPT-5.5）的表现则横跨29-60%的广阔区间。因此，该基准具有高难度、非饱和性（最佳智能体仍留下约33-40%未解决）以及在不同能力层级间具备区分度的特点。除了难度之外，我观察到在最难问题上出现了智能体崩溃现象——智能体停止使用工具。对于最容易崩溃的模型，完全阻止其使用工具几乎不会改变其得分——因此，工具恰恰在最需要它们的地方失去了效用。一个冻结的每问题检查清单将评判者间一致性从斯皮尔曼相关系数0.35提升至0.82。

English

A working citation looks like proof -- but the fact that a link resolves does not mean the cited paper supports the claim. I find that current agentic models rarely fabricate citations (over 99% resolve), yet roughly 15.9% link to the wrong paper. Existing benchmarks miss this failure mode: when a question has a fixed answer key, a model can reproduce the expected source from that key rather than independently verifying that the source supports the claim. I introduce \openbiorq{}, a retrieval-grounded agentic benchmark of 12{,}553 unsolved biomedical research questions across 12 domains that treats open questions as a faithfulness-and-abstention probe. To my knowledge, this is the first biomedical benchmark to combine an agentic setting -- where the model must issue multiple tool calls -- with unsolved questions that have no answer key. Openness is verified against real follow-up evidence rather than a model's parametric knowledge. Difficulty is empirical: I anchor it on questions that three open-weight reference models fail to answer, rather than on subjective hardness labels. On this hardest subset, held-out models from the same lineage as the difficulty anchors solve only ~17%, while three independent frontier agents (Gemini-3-Pro, Opus-4.7, GPT-5.5) span a wide 29-60% range. The benchmark is thus hard, non-saturating (the best agent still leaves ~33-40\% unsolved), and discriminating across capability tiers. Beyond difficulty, I observe agentic collapse on the hardest questions, where agents stop using their tools. For the most collapse-prone model, blocking tool access entirely barely changes its score -- so tools stop paying off exactly where they are needed most. A frozen per-question checklist raises inter-judge agreement from Spearman 0.35 to 0.82.