SiamJEPA：论孪生学生编码器在JEPA中的作用

摘要

近期，联合嵌入预测架构（JEPAs）作为一种自监督表示学习的有效框架，在计算机视觉和机器学习领域引起了广泛关注。与重建像素的掩码自编码器不同，JEPA模型通过预测掩码区域的潜在嵌入来学习表示。现有的基于JEPA的方法（如I-JEPA和V-JEPA）通常在学生网络中使用单一编码器。相比之下，采用孪生编码器作为学生网络更自然地契合了脑启发表示学习框架，但孪生编码器在JEPA模型中的作用仍鲜有探索。本文研究了孪生学生编码器在基于JEPA的表示学习中的影响。为此，我们提出了SiamJEPA——一种配备指数移动平均（EMA）教师网络的掩码孪生学生编码器。SiamJEPA也可视为脑启发表示学习模型PhiNet的JEPA形式化表述。通过在ImageNet线性探测上的大量实验，我们证明孪生编码器能够有效作为JEPA目标的正则化手段，提升表示的可分离性，并加速训练初期的学习进程。此外，在有限训练预算下，SiamJEPA持续优于同类单编码器JEPA变体，且其线性探测准确率超越需要更长训练时间的掩码自编码器（MAE）。我们的发现表明，孪生学生编码器不仅是架构选择，更构成了预测性表示学习的重要归纳偏置。这些结果为基于JEPA的模型设计提供了新见解，并揭示引入孪生学生架构是一种提升自监督表示学习的简洁而有效的方法。

English

Recently, Joint Embedding Predictive Architectures (JEPAs) have attracted significant attention in the computer vision and machine learning communities as a promising framework for self-supervised representation learning. Unlike masked autoencoders that reconstruct pixels, JEPA models learn representations by predicting latent embeddings of masked regions. Existing JEPA-based methods, such as I-JEPA and V-JEPA, typically employ a single encoder in the student network. In contrast, using Siamese encoders for student network is more naturally aligned with brain-inspired representation learning frameworks, yet their role in JEPA models remains largely unexplored. In this paper, we investigate the effect of Siamese student encoders in JEPA-based representation learning. To this end, we propose SiamJEPA, masked Siamese student encoders equipped with an exponential moving average (EMA) teacher network. SiamJEPA can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. Through extensive experiments on ImageNet linear probing, we demonstrate that Siamese encoders act as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training. Furthermore, SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets and achieves higher linear probing accuracy than Masked Autoencoders (MAE) which requires longer training. Our findings reveal that Siamese student encoders are not merely an architectural choice but constitute an important inductive bias for predictive representation learning. These results provide new insights into the design of JEPA-based models and suggest that incorporating Siamese student architectures offers a simple yet effective approach for improving self-supervised representation learning.