壓縮還是不壓縮- 自監督學習與資訊理論：一個評論

摘要

深度神經網絡在監督式學習任務中展現出卓越的性能，但需要大量標記數據。自監督學習提供了一種替代範式，使模型能夠從數據中學習而無需明確標籤。信息理論在理解和優化深度神經網絡方面發揮了重要作用。具體來說，信息瓶頸原則已被應用於優化在監督式設置中壓縮和相關信息保存之間的折衷。然而，在自監督學習中的最佳信息目標仍不清楚。本文從信息理論的角度回顧了各種自監督學習方法，並提出了一個統一的框架，形式化了自監督信息理論學習問題。我們將現有研究整合到一個連貫的框架中，檢視最近的自監督方法，並確定研究機會和挑戰。此外，我們討論了信息理論量的實證測量及其估計器。本文全面評估了信息理論、自監督學習和深度神經網絡之間的交集。

English

Deep neural networks have demonstrated remarkable performance in supervised learning tasks but require large amounts of labeled data. Self-supervised learning offers an alternative paradigm, enabling the model to learn from data without explicit labels. Information theory has been instrumental in understanding and optimizing deep neural networks. Specifically, the information bottleneck principle has been applied to optimize the trade-off between compression and relevant information preservation in supervised settings. However, the optimal information objective in self-supervised learning remains unclear. In this paper, we review various approaches to self-supervised learning from an information-theoretic standpoint and present a unified framework that formalizes the self-supervised information-theoretic learning problem. We integrate existing research into a coherent framework, examine recent self-supervised methods, and identify research opportunities and challenges. Moreover, we discuss empirical measurement of information-theoretic quantities and their estimators. This paper offers a comprehensive review of the intersection between information theory, self-supervised learning, and deep neural networks.

壓縮還是不壓縮- 自監督學習與資訊理論：一個評論

To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review

摘要

Support