创新评估：将研究思路评估视为基于知识的多元视角推理问题

摘要

大语言模型的快速发展催生了科学创意产出的激增，但这种飞跃并未伴随创意评估能力的同步提升。科学评估的本质需要知识基础作为支撑、集体审议机制以及多标准决策过程。然而，现有创意评估方法往往存在知识视野局限、评估维度扁平化以及LLM作为评判者固有偏见等问题。为此，我们将创意评估视为基于知识的多视角推理问题，提出深度创新评估框架InnoEval，旨在模拟人类水平的创意评估能力。该框架采用异构深度知识搜索引擎，从多元网络源动态检索并锚定证据；通过组建具有不同学术背景评审人员的创新评审委员会，实现跨多指标的多维解耦评估，最终达成评审共识。我们基于权威同行评审投稿构建了综合性数据集对InnoEval进行基准测试。实验表明，该框架在点对点、配对比较和群体评估任务中均能稳定超越基线模型，其判断模式与共识形成机制与人类专家高度吻合。

English

The rapid evolution of Large Language Models has catalyzed a surge in scientific idea production, yet this leap has not been accompanied by a matching advance in idea evaluation. The fundamental nature of scientific evaluation needs knowledgeable grounding, collective deliberation, and multi-criteria decision-making. However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation dimensions, and the inherent bias in LLM-as-a-Judge. To address these, we regard idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduce InnoEval, a deep innovation evaluation framework designed to emulate human-level idea assessment. We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources. We further achieve review consensus with an innovation review board containing reviewers with distinct academic backgrounds, enabling a multi-dimensional decoupled evaluation across multiple metrics. We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval. Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.

创新评估：将研究思路评估视为基于知识的多元视角推理问题

InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

摘要

Support