OmniDelta：面向全模态大语言模型中令牌压缩的技能驱动预算分配

摘要

新兴的全模态大语言模型（OmniLLMs）实现了文本、音频和视频的统一理解，但长序列的音频-视频标记带来了显著的存储和推理成本。现有的压缩方法主要关注在固定预算下选择重要标记，而对前置的预算分配问题探索不足。我们证明，查询与音频/视频之间的直接相似性对于模态间预算分配不可靠，而模态内统一预算可能在保留冗余内容的同时遗漏关键证据。为解决这些局限，我们提出OmniDelta——一种免训练、基于技能驱动的框架，将意图感知的模态间分配与内容感知的模态内分配相结合。OmniDelta首先构建音频和视频技能池，根据查询需求调整固定保留标记预算，随后利用局部复杂性和时间冗余度，在音频片段和视频帧之间重新分配模态预算。由此产生的局部预算可与现有剪枝策略结合，在保留总标记比率不变的情况下改变预算的分配位置。在四个音频-视频基准测试中使用两种Qwen2.5-Omni模型进行的实验表明，OmniDelta在不同剪枝比率下建立了新的精度-效率帕累托前沿。在Qwen2.5-Omni-7B模型上保留25%标记时，OmniDelta相比全标记推理减少22.0%的GPU内存，并实现1.64倍的端到端加速。

English

Emerging Omni-modal Large Language Models (OmniLLMs) enable unified understanding of text, audio, and video, but their long audio-video token sequences introduce substantial memory and inference costs. Existing compression methods mainly focus on selecting important tokens under fixed budgets, leaving the preceding budget-allocation problem underexplored. We show that direct query-to-audio/video similarity is unreliable for inter-modal budget allocation, and that uniform intra-modal budgets can miss key evidence while retaining redundant content. To address these limitations, we propose OmniDelta, a training-free, skill-driven framework that couples intent-aware inter-modal allocation with content-aware intra-modal allocation. OmniDelta first constructs audio and video skill pools to shift the fixed retained-token budget according to query demand, then reallocates modality budgets over audio segments and video frames using local complexity and temporal redundancy. The resulting local budgets can be combined with existing pruning strategies, preserving the total retained-token ratio while changing where the budget is spent. Experiments on four audio-video benchmarks with two Qwen2.5-Omni models show that OmniDelta establishes a new accuracy-efficiency Pareto frontier across pruning ratios. At 25% token retention on Qwen2.5-Omni-7B, OmniDelta reduces GPU memory by 22.0% and achieves a 1.64x end-to-end speedup over full-token inference.