iFAN：朴素掩码Transformer的推理感知学习

摘要

基于查询的掩码变换器通过最终层查询预测之间的像素级竞争来组装分割输出，然而这一推理过程在训练中并未被显式优化。我们识别出两个关键的不匹配：具有最高概率-掩码分数的查询不一定产生最准确的掩码，且最终层解码可能丢弃来自中间层的更优预测。为解决这些问题，我们提出了推理感知学习（Inference-Aware Learning, iFAN），一种适用于普通掩码变换器的通用训练框架。iFAN引入调整概率-掩码排序（Adjusted Probability-Mask Ranking, APMR），将查询竞争与预测掩码质量对齐，并抑制高置信度但不准确的竞争者。我们进一步采用跨层自蒸馏（Cross-Layer Self-Distillation, CLSD）将更强的中间层预测迁移至最终层。排序和蒸馏目标仅用于训练，而推理仍保持高效的最终层解码。在COCO、ADE20K和Cityscapes上的实验表明，在全景、实例和语义分割任务中，以及不同架构、骨干网络规模和输入分辨率下均取得了一致的性能提升。总体而言，iFAN在平均性能上提升了1.20 PQ、1.30 AP和0.63 mIoU，而额外参数、FLOPs和推理延迟可忽略不计。

English

Query-based mask transformers assemble segmentation outputs through pixel-wise competition among query predictions of the final layer, yet this inference process is not explicitly optimized during training. We identify two key mismatches: the query with the highest probability-mask score does not necessarily produce the most accurate mask, and final-layer decoding may discard superior predictions from intermediate layers. To address these issues, we propose Inference-Aware Learning (iFAN), a general training framework for plain mask transformers. iFAN introduces Adjusted Probability-Mask Ranking (APMR), which aligns query competition with predicted mask quality and suppresses high-confidence but inaccurate competitors. We further employ Cross-Layer Self-Distillation (CLSD) to transfer stronger intermediate predictions to the final layer. The ranking and distillation objectives are training-only, while inference retains efficient final-layer decoding. Experiments on COCO, ADE20K, and Cityscapes demonstrate consistent improvements across panoptic, instance, and semantic segmentation, as well as across different architectures, backbone scales, and input resolutions. Overall, iFAN improves performance by an average of 1.20 PQ, 1.30 AP, and 0.63 mIoU, with negligible additional parameters, FLOPs and inference latency.