面向多模态大语言模型的自适应推理学习

Learning to Inference Adaptively for Multimodal Large Language Models

March 13, 2025

作者: Zhuoyan Xu, Khoi Duc Nguyen, Preeti Mukherjee, Saurabh Bagchi, Somali Chaterji, Yingyu Liang, Yin Li

cs.AI

摘要

多模态大语言模型（MLLMs）在推理方面展现了卓越的能力，但其高昂的计算成本限制了其在资源受限环境中的部署。尽管近期已有提升MLLMs效率的研究，现有方案仍难以应对运行时条件的变化，尤其是资源可用性的波动（例如，设备上其他程序执行导致的资源竞争）。为填补这一空白，我们提出了AdaLLaVA，一种自适应推理框架，该框架能够在推理过程中根据输入数据及延迟预算，动态调整MLLM的操作配置。我们在涉及问答、推理及幻觉检测的多个基准测试上进行了广泛实验。结果表明，AdaLLaVA能有效遵循输入延迟预算，在运行时实现不同精度与延迟的权衡。此外，我们证明AdaLLaVA不仅能适应输入延迟与内容，还可与令牌选择技术结合以进一步提升效率，并且能够泛化应用于多种MLLMs。项目网页及代码发布地址为https://zhuoyan-xu.github.io/ada-llava/。

English

Multimodal Large Language Models (MLLMs) have shown impressive capabilities in reasoning, yet come with substantial computational cost, limiting their deployment in resource-constrained settings. Despite recent efforts on improving the efficiency of MLLMs, prior solutions fall short in responding to varying runtime conditions, in particular changing resource availability (e.g., contention due to the execution of other programs on the device). To bridge this gap, we introduce AdaLLaVA, an adaptive inference framework that learns to dynamically reconfigure operations in an MLLM during inference, accounting for the input data and a latency budget. We conduct extensive experiments across benchmarks involving question-answering, reasoning, and hallucination. Our results show that AdaLLaVA effectively adheres to input latency budget, achieving varying accuracy and latency tradeoffs at runtime. Further, we demonstrate that AdaLLaVA adapts to both input latency and content, can be integrated with token selection for enhanced efficiency, and generalizes across MLLMs. Our project webpage with code release is at https://zhuoyan-xu.github.io/ada-llava/.