學習通用預測器

摘要

元學習已成為一種強大的方法，用於訓練神經網絡從有限數據中快速學習新任務。對不同任務的廣泛暴露導致多功能表示，從而實現通用問題解決。但是，元學習的極限在哪裡？在這項工作中，我們探索了將最強大的通用預測器——Solomonoff歸納（SI）——通過利用元學習的極限嵌入神經網絡的潛力。我們使用通用圖靈機（UTM）生成用於讓網絡暴露於各種模式的訓練數據。我們對UTM數據生成過程和元訓練協議進行了理論分析。我們使用各種複雜性和通用性的算法數據生成器，對神經結構（例如LSTM、Transformer）進行了全面實驗。我們的結果表明，UTM數據是元學習的寶貴資源，可用於訓練能夠學習通用預測策略的神經網絡。

English

Meta-learning has emerged as a powerful approach to train neural networks to learn new tasks quickly from limited data. Broad exposure to different tasks leads to versatile representations enabling general problem solving. But, what are the limits of meta-learning? In this work, we explore the potential of amortizing the most powerful universal predictor, namely Solomonoff Induction (SI), into neural networks via leveraging meta-learning to its limits. We use Universal Turing Machines (UTMs) to generate training data used to expose networks to a broad range of patterns. We provide theoretical analysis of the UTM data generation processes and meta-training protocols. We conduct comprehensive experiments with neural architectures (e.g. LSTMs, Transformers) and algorithmic data generators of varying complexity and universality. Our results suggest that UTM data is a valuable resource for meta-learning, and that it can be used to train neural networks capable of learning universal prediction strategies.

學習通用預測器

Learning Universal Predictors

摘要

Support