用於視覺令牌剪枝的AI4AI框架

摘要

視覺 token 剪枝可以大幅降低多模態大型語言模型（MLLMs）的推論成本，然而現有方法大多依賴於固定、手工設計的啟發式規則，以及代價高昂的專家試誤。隨著剪枝目標、預算與模型架構日益多樣化，手動探索不斷擴大的設計空間變得越來越困難。本文旨在透過回答一個自然的問題來建構視覺 token 剪枝的 AI4AI 框架：大型語言模型能否自動設計出有效的視覺 token 縮減演算法？儘管 LLM 具備廣泛的演算法知識與強大的推理能力，但要將這類通用知識轉化為特定任務的有效解決方案仍非易事。我們認為，關鍵在於設計一種適當的搜尋狀態表示，將 LLM 的內部知識與視覺 token 剪枝的結構要求及約束聯繫起來。基於此洞察，我們提出 AutoPrune，一個免訓練的 LLM 驅動視覺 token 剪枝策略設計框架。AutoPrune 的核心在於引入了一種 Token 剪枝領域特定語言（TPDSL），其中包含 131 個可重用原子，涵蓋預算控制、token 評分、選擇約束與 token 重組。TPDSL 的一個關鍵特性是將每個搜尋狀態表示為一個強基礎策略的殘差修正。這種殘差形式縮小了搜尋空間，並將 LLM 的注意力引導至對性能最具影響力的策略組件。在 14 個多模態基準及三個 MLLM 骨幹上的實驗證明了 AutoPrune 的有效性、效率與可遷移性。即使移除 94.4% 的視覺 token，AutoPrune 仍能保留全 token 性能的 99% 以上，同時將 FLOPs 降低 9.9 倍、預填充延遲降低 6.4 倍。

English

Visual-token pruning can substantially reduce the inference cost of multimodal large language models (MLLMs), yet existing methods largely rely on fixed, handcrafted heuristics and costly expert trial and error. As pruning objectives, budgets, and model architectures diversify, manually navigating the expanding design space becomes increasingly difficult. This paper aims to build an AI4AI framework for visual-token pruning by addressing a natural question: Can large language models automatically design effective visual-token reduction algorithms? Although LLMs possess broad algorithmic knowledge and strong reasoning capabilities, translating such general knowledge into effective solutions for a specialized task remains nontrivial. We argue that the key lies in designing an appropriate search-state representation that connects the internal knowledge of LLMs with the structural requirements and constraints of visual-token pruning. Based on this insight, we propose AutoPrune, a training-free framework for LLM-driven visual-token pruning policy design. At its core, AutoPrune introduces a Token Pruning Domain-Specific Language (TPDSL) comprising 131 reusable atoms for budget control, token scoring, selection constraints, and token reassembly. A key property of TPDSL is that it represents each search state as a residual modification of a strong base policy. This residual formulation narrows the search space and directs the LLM's attention toward the policy components that are most consequential for performance. Experiments on 14 multimodal benchmarks and three MLLM backbones demonstrate the effectiveness, efficiency, and transferability of AutoPrune. Even when removing 94.4% of visual tokens, AutoPrune preserves more than 99% of full-token performance while reducing FLOPs by 9.9x and prefill latency by 6.4x.