SiamJEPA：論孿生學生編碼器在JEPA中的作用

摘要

近期，聯合嵌入預測架構（JEPAs）作為自監督表徵學習的重要框架，引起了電腦視覺與機器學習領域的廣泛關注。與重建像素的遮罩自編碼器不同，JEPA 模型透過預測遮罩區域的潛在嵌入來學習表徵。現有的基於 JEPA 的方法（如 I-JEPA 和 V-JEPA）通常採用學生網路中的單一編碼器。相比之下，在學生網路中使用孿生編碼器更自然地符合腦啟發表徵學習框架，然而其在 JEPA 模型中的作用仍未得到充分探討。本文研究了孿生學生編碼器對基於 JEPA 的表徵學習的影響。為此，我們提出了 SiamJEPA，即配備指數移動平均（EMA）教師網路的遮罩孿生學生編碼器。SiamJEPA 也可被視為腦啟發表徵學習模型 PhiNet 的 JEPA 形式。透過在 ImageNet 線性探測上的大量實驗，我們證明孿生編碼器作為 JEPA 目標的有效正則化項，能夠提升表徵的可分離性，並加速訓練初期的學習進程。此外，在有限的訓練預算下，SiamJEPA 始終優於可比的單一編碼器 JEPA 變體，並達到比需要更長訓練時間的遮罩自編碼器（MAE）更高的線性探測準確率。我們的研究發現，孿生學生編碼器不僅僅是架構選擇，更構成了預測表徵學習的重要歸納偏置。這些結果為基於 JEPA 的模型設計提供了新視角，並表明引入孿生學生架構是改進自監督表徵學習的一種簡單而有效的方法。

English

Recently, Joint Embedding Predictive Architectures (JEPAs) have attracted significant attention in the computer vision and machine learning communities as a promising framework for self-supervised representation learning. Unlike masked autoencoders that reconstruct pixels, JEPA models learn representations by predicting latent embeddings of masked regions. Existing JEPA-based methods, such as I-JEPA and V-JEPA, typically employ a single encoder in the student network. In contrast, using Siamese encoders for student network is more naturally aligned with brain-inspired representation learning frameworks, yet their role in JEPA models remains largely unexplored. In this paper, we investigate the effect of Siamese student encoders in JEPA-based representation learning. To this end, we propose SiamJEPA, masked Siamese student encoders equipped with an exponential moving average (EMA) teacher network. SiamJEPA can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. Through extensive experiments on ImageNet linear probing, we demonstrate that Siamese encoders act as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training. Furthermore, SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets and achieves higher linear probing accuracy than Masked Autoencoders (MAE) which requires longer training. Our findings reveal that Siamese student encoders are not merely an architectural choice but constitute an important inductive bias for predictive representation learning. These results provide new insights into the design of JEPA-based models and suggest that incorporating Siamese student architectures offers a simple yet effective approach for improving self-supervised representation learning.