VisCo: 利用大型语言模型作为视觉标记压缩的固有编码器

摘要

视觉语言模型（VLM）需要处理大量视觉令牌，导致显著的推理延迟和内存开销。这促使了针对视觉令牌压缩的广泛研究。尽管免训练策略依赖启发式指标，在高压缩比下性能严重下降，而许多基于训练的方法引入外部压缩模块，迫使VLM主干网络进行适配，这带来了大量重训练成本并损害了VLM的先前知识。有效的视觉令牌压缩依赖于强大的信息编码能力——预训练VLM已具备这一能力，但现有方法未能充分利用。受此启发，我们提出VisCo，一种训练高效的自我压缩框架，将预训练VLM本身复用为内在压缩器。VisCo是一种参数共享的自编码器，通过少量记忆令牌压缩视觉信息，并将层级化信息从编码阶段传递至解码阶段。实验表明，VisCo在所有评估压缩比下均超越先前方法，在更高压缩比下优势更大，甚至在极端单令牌设定下也能保持稳定。此外，当与原始视觉令牌结合使用时，学得的记忆令牌甚至能提升基础模型性能，这表明VisCo捕捉到了超出压缩范围的互补表示。

English

Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference latency and memory overhead. This has motivated extensive research on visual token compression. While training-free strategies rely on heuristic metrics and suffer significant performance degradation under high compression ratios, many training-based methods introduce external compression modules that force the VLM backbone to adapt, incurring substantial retraining cost and compromising VLMs' priors. Effective visual token compression hinges on strong information encoding, a capability already present in pretrained VLMs but underutilized by existing approaches. Motivated by this, we propose VisCo, a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor. VisCo is a parameter-sharing autoencoder that compresses visual information using a small set of memory tokens and transfers hierarchical information from encoding to decoding. Experiments show that VisCo surpasses prior methods across all evaluated compression ratios, with larger gains under more aggressive compression, and remains stable even in the extreme single-token setting. Moreover, when combined with the original visual tokens, the learned memory tokens can even improve the base model, suggesting that VisCo captures complementary representations beyond compression.