KAN或MLP：一個更公平的比較

KAN or MLP: A Fairer Comparison

July 23, 2024

作者: Runpeng Yu, Weihao Yu, Xinchao Wang

cs.AI

摘要

本文並未介紹新穎的方法，而是提供了對於 KAN 和 MLP 模型在各種任務中（包括機器學習、計算機視覺、音頻處理、自然語言處理和符號公式表示）更公正和全面的比較。具體而言，我們控制參數數量和 FLOPs 來比較 KAN 和 MLP 的表現。我們的主要觀察是，除了符號公式表示任務外，MLP 通常優於 KAN。我們還對 KAN 進行消融研究，發現其在符號公式表示中的優勢主要來自其 B-spline 激活函數。當將 B-spline 應用於 MLP 時，符號公式表示的性能顯著提高，超越或與 KAN 相匹配。然而，在 MLP 已經優於 KAN 的其他任務中，B-spline 並未顯著提升 MLP 的性能。此外，我們發現在標準的類增量持續學習設置中，KAN 的遺忘問題比 MLP 更嚴重，這與 KAN 論文中報告的結果不同。我們希望這些結果能為未來有關 KAN 和其他 MLP 替代方案的研究提供洞見。項目鏈接：https://github.com/yu-rp/KANbeFair

English

This paper does not introduce a novel method. Instead, it offers a fairer and more comprehensive comparison of KAN and MLP models across various tasks, including machine learning, computer vision, audio processing, natural language processing, and symbolic formula representation. Specifically, we control the number of parameters and FLOPs to compare the performance of KAN and MLP. Our main observation is that, except for symbolic formula representation tasks, MLP generally outperforms KAN. We also conduct ablation studies on KAN and find that its advantage in symbolic formula representation mainly stems from its B-spline activation function. When B-spline is applied to MLP, performance in symbolic formula representation significantly improves, surpassing or matching that of KAN. However, in other tasks where MLP already excels over KAN, B-spline does not substantially enhance MLP's performance. Furthermore, we find that KAN's forgetting issue is more severe than that of MLP in a standard class-incremental continual learning setting, which differs from the findings reported in the KAN paper. We hope these results provide insights for future research on KAN and other MLP alternatives. Project link: https://github.com/yu-rp/KANbeFair