OmniDelta：全模态大語言模型中技能驅動的令牌壓縮預算分配

摘要

新興的全模態大型語言模型（OmniLLMs）能實現對文字、音訊與視訊的統一理解，但其長長的音訊-視訊令牌序列帶來了巨大的記憶體與推理成本。現有的壓縮方法主要聚焦於在固定預算下選取重要令牌，而忽略了前置的預算分配問題。我們證明，直接以查詢與音訊/視訊之間的相似度進行跨模態預算分配並不可靠，且均勻的模態內預算可能遺漏關鍵證據，同時保留冗餘內容。為解決這些限制，我們提出 OmniDelta——一種免訓練、基於技能驅動的框架，將意圖感知的跨模態分配與內容感知的模態內分配相結合。OmniDelta 首先構建音訊與視訊技能池，根據查詢需求調整固定的保留令牌預算，再透過局部複雜度與時間冗餘性將模態預算重新分配至音訊片段與視訊幀。最終得到的局部預算可與現有剪枝策略結合，在改變預算花費位置的同時保留總保留令牌比例。在兩個 Qwen2.5-Omni 模型上的四項音訊-視訊基準測試結果顯示，OmniDelta 在各剪枝比例下建立了全新的準確率-效率帕累托前沿。在 Qwen2.5-Omni-7B 模型上保留 25% 令牌時，OmniDelta 相較於全令牌推理可減少 22.0% 的 GPU 記憶體，並實現 1.64 倍的端到端加速。

English

Emerging Omni-modal Large Language Models (OmniLLMs) enable unified understanding of text, audio, and video, but their long audio-video token sequences introduce substantial memory and inference costs. Existing compression methods mainly focus on selecting important tokens under fixed budgets, leaving the preceding budget-allocation problem underexplored. We show that direct query-to-audio/video similarity is unreliable for inter-modal budget allocation, and that uniform intra-modal budgets can miss key evidence while retaining redundant content. To address these limitations, we propose OmniDelta, a training-free, skill-driven framework that couples intent-aware inter-modal allocation with content-aware intra-modal allocation. OmniDelta first constructs audio and video skill pools to shift the fixed retained-token budget according to query demand, then reallocates modality budgets over audio segments and video frames using local complexity and temporal redundancy. The resulting local budgets can be combined with existing pruning strategies, preserving the total retained-token ratio while changing where the budget is spent. Experiments on four audio-video benchmarks with two Qwen2.5-Omni models show that OmniDelta establishes a new accuracy-efficiency Pareto frontier across pruning ratios. At 25% token retention on Qwen2.5-Omni-7B, OmniDelta reduces GPU memory by 22.0% and achieves a 1.64x end-to-end speedup over full-token inference.