OmniPack：面向高效全模态大语言模型的统一Token压缩

摘要

全模态大语言模型（Omni-LLMs）在视听理解任务上取得了显著性能，但处理冗长且高度冗余的视觉与音频词元序列会带来巨大的计算开销，亟需通过激进的词元压缩来实现高效部署。现有方法在低词元预算下往往性能下降：LLM前的压缩可能丢弃结构上重要且全局分布的证据，而LLM内的压缩则常常未能充分利用查询条件化的视听协作。为解决这些局限性，我们提出了OmniPack，一种无训练框架，它将LLM前的结构压缩与LLM内的任务相关语义精炼相协调。在进入LLM之前，OmniPack通过模态特定重要性、全局覆盖和相似性感知合并来消除结构冗余。在充分的多模态交互之后，它进一步通过文本引导和视听协作整合多样化、任务相关的表示。在三个Omni-LLM骨干和五个基准上的大量实验表明，OmniPack在不同保留比例下 consistently 实现了最佳的性能-效率权衡，优于所有现有方法。值得注意的是，在Qwen2.5-Omni-7B上，OmniPack保留了原始性能的98.0%，同时将FLOPs降低至16.7%；并且仅使用原始FLOPs的6.8%即可保持92.9%的原始性能。

English

Omni-modal large language models (Omni-LLMs) have achieved remarkable performance on audio-visual understanding tasks, but processing long and highly redundant visual and audio token sequences incurs substantial computational overhead, demanding aggressive token compression for efficient deployment. Existing methods often degrade at low token budgets: pre-LLM compression may discard structurally important and globally distributed evidence, whereas inner-LLM compression often underexploits query-conditioned audio-visual collaboration. To address these limitations, we propose OmniPack, a training-free framework that coordinates structural compression before the LLM with task-relevant semantic refinement within the LLM. Before the LLM, OmniPack removes structural redundancy through modality-specific importance, global coverage, and similarity-aware merging. After sufficient multimodal interaction, it further consolidates diverse, task-relevant representations through textual guidance and audio-visual collaboration. Extensive experiments on five benchmarks with three Omni-LLM backbones demonstrate that OmniPack consistently achieves the best performance-efficiency trade-off across diverse retention ratios, outperforming all existing methods. Notably, on Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while reducing FLOPs to 16.7%, and still retains 92.9% of the original performance with only 6.8% of the original FLOPs.