MODUS：多樣模態的僅解碼器任意至任意建模

摘要

任意到任意模型能够在一个统一网络中，根据任意其他模态的组合预测任意模态，这种范式常用于多模态视觉与视觉-语言模型，并逐渐扩展到生态学、天文学等科学领域。现有的任意到任意模型通常采用编码器-解码器或扩散架构从零训练，这限制了其性能，且无法利用强大的预训练仅解码器模型作为先验。本研究探索了基于仅解码器的任意到任意多模态建模方法，该方法对称处理所有模态，支持任意模态作为输入和输出，无需特定模态的解码头、损失函数或任务流程。由于每种模态同时是该模型的输入和输出，所提出的模型命名为Modus，能够支持多种应用场景，例如通过中间模态实现链式生成，或通过利用另一生成模态对模型自身输出进行评分来实现跨模态自验证。Modus模型开箱即用性能强大，在多个基准测试中仅凭单一模型即可与专家模型和多任务基线模型相媲美。所有材料已开源至 https://modus-multimodal.epfl.ch/。

English

Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced at https://modus-multimodal.epfl.ch/.