流等變循環神經網路

摘要

數據以連續流的形式抵達我們的感官，從一個瞬間平滑地轉變到下一個瞬間。這些平滑的轉變可以被視為我們所處環境的連續對稱性，定義了刺激隨時間變化的等價關係。在機器學習中，尊重數據對稱性的神經網絡架構被稱為等變網絡，並在泛化能力和樣本效率方面具有可證明的優勢。然而，迄今為止，等變性僅被考慮用於靜態變換和前饋網絡，限制了其在序列模型（如循環神經網絡，RNNs）及相應的時間參數化序列變換中的應用。在本研究中，我們將等變網絡理論擴展到“流”這一領域——即捕捉自然隨時間變化的單參數李子群，如視覺運動。我們首先展示標準RNNs通常不具備流等變性：其隱藏狀態無法以幾何結構化的方式對移動刺激進行轉換。然後，我們展示了如何引入流等變性，並證明這些模型在訓練速度、長度泛化和速度泛化方面顯著優於非等變模型，無論是在下一步預測還是序列分類任務中。我們將這項工作視為構建尊重支配我們周圍世界的時間參數化對稱性的序列模型的第一步。

English

Data arrives at our senses as a continuous stream, smoothly transforming from one instant to the next. These smooth transformations can be viewed as continuous symmetries of the environment that we inhabit, defining equivalence relations between stimuli over time. In machine learning, neural network architectures that respect symmetries of their data are called equivariant and have provable benefits in terms of generalization ability and sample efficiency. To date, however, equivariance has been considered only for static transformations and feed-forward networks, limiting its applicability to sequence models, such as recurrent neural networks (RNNs), and corresponding time-parameterized sequence transformations. In this work, we extend equivariant network theory to this regime of `flows' -- one-parameter Lie subgroups capturing natural transformations over time, such as visual motion. We begin by showing that standard RNNs are generally not flow equivariant: their hidden states fail to transform in a geometrically structured manner for moving stimuli. We then show how flow equivariance can be introduced, and demonstrate that these models significantly outperform their non-equivariant counterparts in terms of training speed, length generalization, and velocity generalization, on both next step prediction and sequence classification. We present this work as a first step towards building sequence models that respect the time-parameterized symmetries which govern the world around us.

流等變循環神經網路

Flow Equivariant Recurrent Neural Networks

摘要

Support