隐形的捷径：为何视觉编码器能识别你的相机

摘要

深度视觉模型会利用捷径，依赖于与监督信号相关的线索。先前工作主要关注可见偏差，例如物体-背景或纹理相关性。我们识别出捷径学习的另一个来源：嵌入在像素级别的不可见元数据痕迹，这些元数据涉及图像处理和照片采集。我们假设大规模语义监督——无论是通过类别标签（ImageNet）还是十亿规模描述文本（LAION）——在预训练期间自然诱导出元数据-语义相关性，从而使模型将低层信号转化为预测性特征。通过引入受控的元数据-语义相关性，我们证明更强的相关性会导致对元数据痕迹的系统性更高敏感性，并在元数据分布偏移下带来更大的性能下降。我们进一步探索了在预训练期间和之后应用的缓解策略，这些策略不仅降低了对目标元数据的敏感性，也降低了对未见元数据的敏感性，同时不牺牲下游任务的性能。元数据敏感性也有积极的一面：它部分解释了一些编码器强大的生成图像检测能力，而缓解这种敏感性可以改善分布外泛化。代码：https://github.com/ryan-caesar-ramos/visual-encoder-traces

English

Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces