学习通用预测器

摘要

元学习已经成为一种强大的方法，用于训练神经网络从有限数据中快速学习新任务。对不同任务的广泛接触导致多才多艺的表示，从而实现了通用问题解决能力。但是，元学习的局限性在哪里？在这项工作中，我们探讨了将最强大的通用预测器——Solomonoff归纳（SI）——通过充分利用元学习的潜力，嵌入神经网络的可能性。我们使用通用图灵机（UTMs）生成训练数据，用于让网络接触各种模式。我们对UTM数据生成过程和元训练协议进行了理论分析。我们使用神经架构（如LSTM、Transformer）和不同复杂性和通用性的算法数据生成器进行了全面实验。我们的结果表明，UTM数据是元学习的宝贵资源，可以用来训练能够学习通用预测策略的神经网络。

English

Meta-learning has emerged as a powerful approach to train neural networks to learn new tasks quickly from limited data. Broad exposure to different tasks leads to versatile representations enabling general problem solving. But, what are the limits of meta-learning? In this work, we explore the potential of amortizing the most powerful universal predictor, namely Solomonoff Induction (SI), into neural networks via leveraging meta-learning to its limits. We use Universal Turing Machines (UTMs) to generate training data used to expose networks to a broad range of patterns. We provide theoretical analysis of the UTM data generation processes and meta-training protocols. We conduct comprehensive experiments with neural architectures (e.g. LSTMs, Transformers) and algorithmic data generators of varying complexity and universality. Our results suggest that UTM data is a valuable resource for meta-learning, and that it can be used to train neural networks capable of learning universal prediction strategies.

学习通用预测器

Learning Universal Predictors

摘要

Support