隱形捷徑：為何視覺編碼器知道你的相機

摘要

深度視覺模型會利用捷徑，依賴與監督信號相關的線索。先前的研究聚焦於可見的偏誤，例如物體-背景或紋理相關性。我們識別出捷徑學習的另一種來源：嵌入於像素層級的不可見元數據痕跡，這些元數據涵蓋影像處理與照片獲取等資訊。我們假設大規模語義監督——無論是透過類別標籤（ImageNet）還是十億規模的標題描述（LAION）——在預訓練過程中自然地誘發元數據與語義之間的相關性，促使模型將低階信號轉化為預測性特徵。透過引入受控的元數據-語義相關性，我們證明較強的相關性會產生系統性更高的元數據痕跡敏感度，並在元數據分佈偏移下造成更大的效能衰退。我們進一步探討在預訓練期間及之後應用的緩解策略，這些策略不僅能降低對目標元數據的敏感度，也能降低對未見元數據的敏感度，且不犧牲下游任務的效能。元數據敏感度也有其正面意義：它部分解釋了某些編碼器強大的生成影像偵測能力，而緩解此敏感度則可改善分佈外泛化。程式碼：https://github.com/ryan-caesar-ramos/visual-encoder-traces

English

Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces