局部松弛，全局均衡：几何引导的视觉-语言混合专家负载均衡

摘要

视觉-语言MoE批次包含不同数量的图像和文本token。图像分辨率、图像数量、分块方式以及提示长度都会改变这种token混合比例。我们将标准的token级Switch辅助损失称为Std-Aux。Std-Aux仅平衡混合负载，因此在某种混合比例下，图像和文本负载的较大误差可能相互抵消。在我们的主模型上，经过训练的同一路由器在不同图像分辨率下的负载不均衡程度变化超过五倍。我们固定图像和文本的负载分布，并推导出随token混合比例变化的精确负载曲线。图像-文本负载差距控制了对token混合比例的敏感度。物理预处理也可能改变条件负载分布，而固定分布定律排除了这类变化。为了设计解决方案，我们检查了路由器的输入结构。图像和文本占据不同区域，而视觉token按源图像强烈聚集。模态边界促使我们分别设置图像和文本项；图像边界则促使每个图像使用一个等权重的路由实例。ReBA，即“内部松弛、跨域平衡”（Relax Within, Balance Across），实现了这两种选择。在四个拆分骨干网络上，ReBA在所有报告的基准输入上都降低了负载，同时保持平均任务准确率与Std-Aux相当。ReBA还降低了测试范围内的平均负载，并降低了分辨率和分块变化下的最差物理负载。代码可在 https://github.com/ZiangWu-77/ReBA 获取。

English

Vision-language MoE batches contain different numbers of image and text tokens. Image resolution, image count, tiling, and prompt length all change this token mix. We call the standard token-level Switch auxiliary loss Std-Aux. Std-Aux balances only the mixed load, so large image and text load errors can cancel at one mix. On our main model, the same trained router shows more than a fivefold change in load imbalance across image resolutions. We hold the image and text load profiles fixed and derive the exact load curve as the token mix varies. The image-text load gap controls sensitivity to the token mix. Physical preprocessing can also change the conditional profiles. The fixed-profile law excludes such changes. To design a remedy, we examine the router input structure. Image and text occupy distinct regions, while visual tokens group strongly by source image. The modality boundary motivates separate image and text terms. The image boundary motivates one equal-weight routing instance per image. ReBA, or Relax Within, Balance Across, implements both choices. Across four split backbones, ReBA lowers load on every reported benchmark input while keeping mean task accuracy comparable to Std-Aux. ReBA also lowers average load over the tested range and worst physical load under resolution and tiling shifts. Code is available at https://github.com/ZiangWu-77/ReBA.