VisCo：利用大型語言模型作為內建編碼器進行視覺標記壓縮

摘要

視覺-語言模型（VLM）處理大量視覺令牌，導致顯著的推理延遲與記憶體開銷，因而促成視覺令牌壓縮的廣泛研究。無需訓練的策略依賴啟發式度量，在高壓縮比下效能大幅衰減；而許多基於訓練的方法引入外部壓縮模組，迫使VLM主幹網路進行適應，不僅帶來龐大的再訓練成本，也損害VLM的既有先驗知識。有效的視覺令牌壓縮仰賴強大的資訊編碼能力，此能力已內建於預訓練VLM中，卻未被現有方法充分利用。受此啟發，我們提出VisCo，一種訓練高效的自我壓縮框架，將預訓練VLM本身重複利用為內建壓縮器。VisCo是一個參數共享的自編碼器，透過少量記憶令牌壓縮視覺資訊，並將編碼階段的階層式資訊傳遞至解碼階段。實驗結果顯示，在所有評估的壓縮比下，VisCo均超越先前方法，在更激進的壓縮設定中優勢更大，甚至在極端的單一令牌設定下仍保持穩定。此外，當與原始視覺令牌結合時，所學習的記憶令牌甚至能提升基礎模型表現，顯示VisCo捕捉到超越壓縮層面的互補表徵。

English

Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference latency and memory overhead. This has motivated extensive research on visual token compression. While training-free strategies rely on heuristic metrics and suffer significant performance degradation under high compression ratios, many training-based methods introduce external compression modules that force the VLM backbone to adapt, incurring substantial retraining cost and compromising VLMs' priors. Effective visual token compression hinges on strong information encoding, a capability already present in pretrained VLMs but underutilized by existing approaches. Motivated by this, we propose VisCo, a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor. VisCo is a parameter-sharing autoencoder that compresses visual information using a small set of memory tokens and transfers hierarchical information from encoding to decoding. Experiments show that VisCo surpasses prior methods across all evaluated compression ratios, with larger gains under more aggressive compression, and remains stable even in the extreme single-token setting. Moreover, when combined with the original visual tokens, the learned memory tokens can even improve the base model, suggesting that VisCo captures complementary representations beyond compression.