BlackGoose Rimer：運用RWKV-7作為大規模時間序列建模中Transformer的簡潔而卓越替代方案

BlackGoose Rimer: Harnessing RWKV-7 as a Simple yet Superior Replacement for Transformers in Large-Scale Time Series Modeling

March 8, 2025

作者: Li weile, Liu Xiao

cs.AI

摘要

時間序列模型在擴展以處理大型且複雜的數據集方面面臨著重大挑戰，這與大型語言模型（LLMs）所實現的擴展能力相似。時間序列數據的獨特特性以及模型擴展的計算需求，促使我們需要創新的方法。儘管研究人員已經探索了多種架構，如Transformer、LSTM和GRU來應對這些挑戰，但我們提出了一種使用RWKV-7的新穎解決方案，該方案將元學習整合到其狀態更新機制中。通過將RWKV-7的時間混合和通道混合組件整合到基於Transformer的時間序列模型Timer中，我們實現了約1.13至43.3倍的性能提升，並在訓練時間上減少了4.5倍，同時僅使用了1/23的參數。我們的代碼和模型權重已公開，供進一步研究和開發使用，詳見https://github.com/Alic-Li/BlackGoose_Rimer。

English

Time series models face significant challenges in scaling to handle large and complex datasets, akin to the scaling achieved by large language models (LLMs). The unique characteristics of time series data and the computational demands of model scaling necessitate innovative approaches. While researchers have explored various architectures such as Transformers, LSTMs, and GRUs to address these challenges, we propose a novel solution using RWKV-7, which incorporates meta-learning into its state update mechanism. By integrating RWKV-7's time mix and channel mix components into the transformer-based time series model Timer, we achieve a substantial performance improvement of approximately 1.13 to 43.3x and a 4.5x reduction in training time with 1/23 parameters, all while utilizing fewer parameters. Our code and model weights are publicly available for further research and development at https://github.com/Alic-Li/BlackGoose_Rimer.