創新評估：將研究理念評估視為知識驅動的多視角推理問題

摘要

大型語言模型的快速演進已催生科學創意產出的激增，然而這種飛躍並未伴隨相應的創意評估機制進步。科學評估的本質需要知識基礎、集體審議與多準則決策，但現有評估方法常受制於狹隘的知識視野、扁平化的評估維度，以及LLM作為評判者固有的偏見。為解決這些問題，我們將創意評估視為基於知識的多視角推理任務，提出深度創新評估框架InnoEval，旨在模擬人類層級的創意評鑑。我們採用異質性深度知識搜尋引擎，從多元網路來源檢索並錨定動態證據，並透過匯聚不同學術背景評審的創新審查委員會達成審議共識，實現跨多項指標的多維度解耦評估。基於權威同儕評審資料構建的完整數據集顯示，InnoEval在點對點、配對比較及群組評估任務中均持續超越基準模型，其判斷模式與共識機制與人類專家高度契合。

English

The rapid evolution of Large Language Models has catalyzed a surge in scientific idea production, yet this leap has not been accompanied by a matching advance in idea evaluation. The fundamental nature of scientific evaluation needs knowledgeable grounding, collective deliberation, and multi-criteria decision-making. However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation dimensions, and the inherent bias in LLM-as-a-Judge. To address these, we regard idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduce InnoEval, a deep innovation evaluation framework designed to emulate human-level idea assessment. We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources. We further achieve review consensus with an innovation review board containing reviewers with distinct academic backgrounds, enabling a multi-dimensional decoupled evaluation across multiple metrics. We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval. Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.

創新評估：將研究理念評估視為知識驅動的多視角推理問題

InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

摘要

Support