基于令牌的双视图融合与大型视觉模型自适应用于乳腺癌分类

摘要

准确的乳腺癌钼靶分类需要有效整合头尾位（CC）和内外斜位（MLO）的互补信息，这两类视图能更全面地表征乳腺异常。然而，现有的多视角学习方法通常依赖特征级聚合或单阶段交叉注意力，这可能混淆视角特定表征与共享表征，并将交互限制在有限的网络深度内。为解决这些局限，我们提出了一种以令牌为中心的双视角学习框架，该框架在冻结的视觉Transformer骨干中统一了基于提示的适应与跨视角融合。该框架将视角间交互重构为结构化的令牌级通信，其中专用融合令牌通过交叉注意力显式编码CC与MLO视图之间的双向信息交换，作为跨视角依赖关系的中间载体，而非依赖直接的特征融合。与在单层进行融合的传统方法不同，融合模块被插入多个Transformer深度，从而在编码器层级间实现渐进且重复的交互。融合令牌被重新整合到令牌序列中，并通过后续Transformer层进行精炼，在保留视角特定结构的同时促进互补信息的分层传播。在VinDr-Mammo和CMMD数据集上的实验表明，该框架相较于线性探测、仅提示适应以及传统融合基线均取得一致改进。在VinDr-Mammo的BI-RADS分类任务中，该框架实现了50.40%的F1分数和0.8090的AUC，其中在二分类设置下比双视角融合基线提升了0.10 AUC。消融研究进一步验证了基于令牌的融合与多深度交互设计的有效性。

English

Accurate breast cancer classification from mammography requires effective integration of complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views, which provide a more complete characterization of breast abnormalities. However, existing multi-view learning approaches typically rely on feature-level aggregation or single-stage cross-attention, which can entangle view-specific and shared representations and restrict interaction to limited network depths. To address these limitations, we propose a token-centric dual-view learning framework that unifies prompt-based adaptation and cross-view fusion within a frozen vision transformer backbone. The framework reformulates inter-view interaction as structured token-level communication, where dedicated fusion tokens explicitly encode bidirectional information exchange between CC and MLO views via cross-attention, serving as intermediate carriers of cross-view dependencies rather than relying on direct feature fusion. Unlike conventional methods that apply fusion at a single layer, fusion modules are inserted at multiple transformer depths, enabling progressive and repeated interaction across the encoder hierarchy. Fusion tokens are reintegrated into the token sequence and refined by subsequent transformer layers, facilitating hierarchical propagation of complementary information while preserving view-specific structure. Experiments on VinDr-Mammo and CMMD datasets demonstrate consistent improvements over linear probing, prompt-only adaptation, and conventional fusion baselines. On the VinDr-Mammo BI-RADS classification task, the framework achieves 50.40% F1-score and 0.8090 AUC, including a 0.10 AUC improvement over a dual-view fusion baseline in the binary setting. Ablation studies further validate the effectiveness of token-based fusion and multi-depth interaction design.