自然圖像的對比學習理論

摘要

為何使用簡單圖像與資料增強的對比學習能夠為下游任務產出有用的表徵？我們透過解析計算針對一系列基礎資料增強方法與任何具備平穩統計特性的圖像資料集，在對比損失函數下的最優表徵來探討此問題。我們證明，對於某些資料增強方法，最優解可由一類卷積神經網路達成：其第一層濾波器為正弦波，接續逐點非線性函數、全局平均池化，以及執行部分白化的最終線性層。我們亦顯示，針對更複雜的資料增強方法，此類卷積神經網路中的最優權重仍為正弦波。這些正弦波的頻率及其權重可透過簡單的注水演算法，依據資料集的期望功率譜計算得出。透過在不同圖像資料集與資料增強方法上進行實驗，我們發現此類以隨機梯度下降訓練的卷積神經網路，在經驗上會於第一層學習到正弦波，並執行部分白化。

English

Why does contrastive learning with simple images and augmentations yield useful representations for downstream tasks? We address this question by analytically computing the optimal representation in terms of a contrastive loss for a range of basic augmentations and any image dataset with stationary statistics. We show that for certain augmentations the optimum can be attained by a CNN whose first layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer that performs partial whitening. We also show that the optimal weights in such CNNs for more complicated augmentations are still sinusoids. The frequencies of the sinusoids and their weights can be computed using a simple waterfilling algorithm given the dataset's expected power spectrum. Experiments with different image datasets and augmentations show that such CNNs trained with SGD empirically learn sinusoids in their first layer and to perform partial whitening