压缩还是不压缩-自监督学习和信息论：一项综述

摘要

深度神经网络在监督学习任务中表现出色，但需要大量标记数据。自监督学习提供了一种替代范式，使模型能够从数据中学习而无需明确标签。信息论在理解和优化深度神经网络方面发挥了重要作用。具体而言，信息瓶颈原理已被应用于优化在监督设置中压缩和相关信息保留之间的权衡。然而，在自监督学习中的最佳信息目标仍不清楚。本文从信息论的角度审视各种自监督学习方法，并提出一个统一的框架，形式化自监督信息论学习问题。我们将现有研究整合到一个连贯的框架中，审视最近的自监督方法，并确定研究机会和挑战。此外，我们讨论信息论量及其估计器的经验测量。本文全面审视了信息论、自监督学习和深度神经网络之间的交叉领域。

English

Deep neural networks have demonstrated remarkable performance in supervised learning tasks but require large amounts of labeled data. Self-supervised learning offers an alternative paradigm, enabling the model to learn from data without explicit labels. Information theory has been instrumental in understanding and optimizing deep neural networks. Specifically, the information bottleneck principle has been applied to optimize the trade-off between compression and relevant information preservation in supervised settings. However, the optimal information objective in self-supervised learning remains unclear. In this paper, we review various approaches to self-supervised learning from an information-theoretic standpoint and present a unified framework that formalizes the self-supervised information-theoretic learning problem. We integrate existing research into a coherent framework, examine recent self-supervised methods, and identify research opportunities and challenges. Moreover, we discuss empirical measurement of information-theoretic quantities and their estimators. This paper offers a comprehensive review of the intersection between information theory, self-supervised learning, and deep neural networks.

压缩还是不压缩-自监督学习和信息论：一项综述

To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review

摘要

Support