面向视觉Token剪枝的AI4AI框架

摘要

视觉标记剪枝能够显著降低多模态大语言模型（MLLMs）的推理成本，但现有方法主要依赖固定的手工启发式规则和代价高昂的专家试错。随着剪枝目标、预算和模型架构的多样化，人工在日益扩展的设计空间中探索变得越来越困难。本文旨在通过回答一个自然问题来构建面向视觉标记剪枝的AI4AI框架：大语言模型能否自动设计有效的视觉标记缩减算法？尽管大语言模型具备广泛的算法知识和强大的推理能力，但将这些通用知识转化为针对特定任务的有效解决方案仍然并非易事。我们认为，关键在于设计一种恰当的搜索状态表示，将大语言模型的内部知识与视觉标记剪枝的结构要求和约束联系起来。基于这一洞察，我们提出了AutoPrune，一个免训练的大语言模型驱动视觉标记剪枝策略设计框架。其核心是引入了一种标记剪枝领域特定语言（TPDSL），包含131个可复用原子组件，用于预算控制、标记评分、选择约束和标记重组。TPDSL的一个关键特性是，它将每个搜索状态表示为对强基础策略的残差修改。这种残差形式缩小了搜索空间，并将大语言模型的注意力引导到对性能影响最大的策略组件上。在14个多模态基准和三个MLLM骨干网络上的实验证明了AutoPrune的有效性、高效性和可迁移性。即使移除94.4%的视觉标记，AutoPrune仍能保留超过99%的全标记性能，同时将FLOPs降低9.9倍，预填充延迟降低6.4倍。

English

Visual-token pruning can substantially reduce the inference cost of multimodal large language models (MLLMs), yet existing methods largely rely on fixed, handcrafted heuristics and costly expert trial and error. As pruning objectives, budgets, and model architectures diversify, manually navigating the expanding design space becomes increasingly difficult. This paper aims to build an AI4AI framework for visual-token pruning by addressing a natural question: Can large language models automatically design effective visual-token reduction algorithms? Although LLMs possess broad algorithmic knowledge and strong reasoning capabilities, translating such general knowledge into effective solutions for a specialized task remains nontrivial. We argue that the key lies in designing an appropriate search-state representation that connects the internal knowledge of LLMs with the structural requirements and constraints of visual-token pruning. Based on this insight, we propose AutoPrune, a training-free framework for LLM-driven visual-token pruning policy design. At its core, AutoPrune introduces a Token Pruning Domain-Specific Language (TPDSL) comprising 131 reusable atoms for budget control, token scoring, selection constraints, and token reassembly. A key property of TPDSL is that it represents each search state as a residual modification of a strong base policy. This residual formulation narrows the search space and directs the LLM's attention toward the policy components that are most consequential for performance. Experiments on 14 multimodal benchmarks and three MLLM backbones demonstrate the effectiveness, efficiency, and transferability of AutoPrune. Even when removing 94.4% of visual tokens, AutoPrune preserves more than 99% of full-token performance while reducing FLOPs by 9.9x and prefill latency by 6.4x.