無需強假設：透過時間差異的視覺表徵學習

摘要

人工智能的进步，很大程度上是由那些假设更少的方法所推动的。随着计算能力和数据量的增加，归纳偏置较弱的方法通常优于假设更强的那些。这在视觉表征学习领域尤为典型——该方法已从监督学习主导，经历弱监督学习，发展到如今无需人工标注的自监督学习的广泛成功。然而，即便是现代的自监督学习方法，仍依赖于诸如数据增强、掩码或裁剪等强烈的归纳偏置。若此趋势持续，即使这些残余的偏置在大规模场景下也将成为瓶颈——我们的实验证实了这一点：归纳偏置的最优强度随数据增长而减弱。这促使我们寻找依赖更少假设的方法。为此，我们提出视觉中的时间差分（Temporal Difference in Vision, TDV），一种从视频中进行自监督学习的新范式，它避开了现有的归纳偏置，仅依赖于一个因果假设：过去导致未来。TDV通过联合训练图像编码器与运动编码器，使得当前帧的表征加上编码后的运动，等于下一帧的表征。尽管未利用任何强烈的归纳偏置，TDV在密集空间任务上仍能达到最先进方法的水平，为无需强假设的表征学习奠定了基础。

English

Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly characteristic of the field of Visual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success of Self-Supervised Learning without human labels. Yet, even modern Self-Supervised Learning approaches still depend on strong inductive biases such as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale -- and our experiments confirm this: the optimal strength of inductive biases decreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video that avoids existing inductive biases, relying instead on a causal assumption that the past causes the future. TDV functions by jointly training an image encoder and a motion encoder so that the current frame's representation plus the encoded motion equals the next frame's representation. Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks, laying the foundation for representation learning without strong assumptions.