自然图像对比学习理论

摘要

为何基于简单图像与数据增强的对比学习能为下游任务生成有用的表征？我们通过分析计算针对一系列基础增强操作和任意具有平稳统计特性的图像数据集的对比损失最优表征来探讨这一问题。研究表明，对于某些增强操作，最优解可通过第一层滤波器为正弦函数的CNN实现，该网络随后依次接入逐点非线性变换、全局平均池化以及执行部分白化的最终线性层。我们还发现，对于更复杂的增强操作，此类CNN中的最优权重仍保持正弦形式。给定数据集的预期功率谱后，正弦函数的频率及其权重可通过简单的水注算法计算得出。针对不同图像数据集与增强操作的实验表明，经SGD训练后的此类CNN确实在其第一层学习到正弦函数并实现了部分白化。

English

Why does contrastive learning with simple images and augmentations yield useful representations for downstream tasks? We address this question by analytically computing the optimal representation in terms of a contrastive loss for a range of basic augmentations and any image dataset with stationary statistics. We show that for certain augmentations the optimum can be attained by a CNN whose first layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer that performs partial whitening. We also show that the optimal weights in such CNNs for more complicated augmentations are still sinusoids. The frequencies of the sinusoids and their weights can be computed using a simple waterfilling algorithm given the dataset's expected power spectrum. Experiments with different image datasets and augmentations show that such CNNs trained with SGD empirically learn sinusoids in their first layer and to perform partial whitening