测试时频谱感知潜在导向：实现视觉语言模型的零样本泛化

摘要

视觉语言模型（VLM）在零样本推理方面表现出色，但在测试时域偏移下性能常会下降。因此，基于情景的测试时自适应策略近期成为将VLM适配至单张未标注图像的有效技术。然而现有适配策略（如测试时提示调优）通常需要对大型编码器权重进行反向传播，或需改动核心模型组件。本研究提出频谱感知测试时导向（STS）框架，该轻量级适配方法从文本嵌入中提取频谱子空间以定义主语义方向，通过调整少量样本偏移参数来最小化增强视图间的熵值，从而实现频谱感知的隐空间表征导向。STS完全在推理阶段于隐空间运行，无需对冻结编码器进行反向传播或结构修改。基于标准评估协议的综合实验表明，STS在绝大多数情况下显著超越或优于最先进的测试时自适应方法，同时仅引入极少额外参数，推理速度较传统测试时提示调优提升最高达8倍，内存占用减少12倍。代码已开源：https://github.com/kdafnis/STS。

English

Vision-Language Models (VLMs) excel at zero-shot inference but often degrade under test-time domain shifts. For this reason, episodic test-time adaptation strategies have recently emerged as powerful techniques for adapting VLMs to a single unlabeled image. However, existing adaptation strategies, such as test-time prompt tuning, typically require backpropagating through large encoder weights or altering core model components. In this work, we introduce Spectrum-Aware Test-Time Steering (STS), a lightweight adaptation framework that extracts a spectral subspace from the textual embeddings to define principal semantic directions and learns to steer latent representations in a spectrum-aware manner by adapting a small number of per-sample shift parameters to minimize entropy across augmented views. STS operates entirely at inference in the latent space, without backpropagation through or modification of the frozen encoders. Building on standard evaluation protocols, our comprehensive experiments demonstrate that STS largely surpasses or compares favorably against state-of-the-art test-time adaptation methods, while introducing only a handful of additional parameters and achieving inference speeds up to 8x faster with a 12x smaller memory footprint than conventional test-time prompt tuning. The code is available at https://github.com/kdafnis/STS.

测试时频谱感知潜在导向：实现视觉语言模型的零样本泛化

Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models

摘要

Support