Colon-X：从多模态理解到临床推理的智能结肠镜技术新突破

摘要

本研究推出Colon-X开放计划，旨在推动结肠镜多模态智能发展。我们首先构建了ColonVQA——迄今最全面的结肠镜多模态数据集，涵盖76种临床发现和18项多模态任务，包含超过110万条视觉问答条目。除作为社区级数据基础外，我们进一步探索结肠镜领域关键但尚未充分研究的范式转变：从多模态理解向临床推理演进。（a）为评估当前多模态理解能力现状，我们系统测试了22个多模态大语言模型的泛化能力，并考察其在人为干扰下的可靠性。结果表明领先MLLMs的临床输出仍远未达到稳健可信的水平。（b）为缩小这一差距，我们深入探索面向结肠镜的推理核心智能：通过多专家辩论流程构建临床推理数据集ColonReason，并开发首个体现R1范式的新型模型ColonR1，该模型融合任务自适应奖励与梯度稳定优化技术。在数据稀缺条件下，ColonR1以56.61%的综合准确度超越监督微调25.22%，为多模态结肠镜分析建立了推理赋能的新基准。所有数据与模型资源已公开于https://github.com/ai4colonoscopy/Colon-X。

English

In this study, we present Colon-X, an open initiative aimed at advancing multimodal intelligence in colonoscopy. We begin by constructing ColonVQA, the most comprehensive multimodal dataset ever built for colonoscopy, featuring over 1.1M+ visual question answering entries across 76 clinical findings and 18 multimodal tasks. Beyond serving as a community-wide data foundation, we further investigate a critical yet underexplored transition in colonoscopy - evolving from multimodal understanding to clinical reasoning: (a) To capture the current landscape of multimodal understanding behaviors, we systematically assess the generalizability of 22 multimodal large language models and examine their reliability under human-induced perturbations. The results reveal that clinical outputs from leading MLLMs remain far from robust and trustworthy. (b) To narrow this gap, we further explore reasoning-centric intelligence tailored for colonoscopy. Specifically, we curate ColonReason, a clinically grounded reasoning dataset annotated through a multi-expert debating pipeline, and develop ColonR1, the first R1-styled model incorporating task-adaptive rewarding and gradient-stable optimization techniques. Under data-scarce conditions, our ColonR1 achieves 56.61% overall accuracy, outperforming supervised fine-tuning by 25.22%, and sets a new reasoning-enabled baseline for multimodal colonoscopy analysis. All data and model resources are publicly available at https://github.com/ai4colonoscopy/Colon-X.

Colon-X：从多模态理解到临床推理的智能结肠镜技术新突破

Colon-X: Advancing Intelligent Colonoscopy from Multimodal Understanding to Clinical Reasoning

摘要

Support