KAN还是MLP：一个更公平的比较
KAN or MLP: A Fairer Comparison
July 23, 2024
作者: Runpeng Yu, Weihao Yu, Xinchao Wang
cs.AI
摘要
本文并未介绍新颖的方法。相反，它提供了对KAN和MLP模型在各种任务中的更公平、更全面的比较，包括机器学习、计算机视觉、音频处理、自然语言处理和符号公式表示。具体而言，我们控制参数数量和FLOPs来比较KAN和MLP的性能。我们的主要观察是，除了符号公式表示任务外，MLP通常优于KAN。我们还对KAN进行消融研究，发现其在符号公式表示中的优势主要来自其B样条激活函数。当B样条应用于MLP时，在符号公式表示中的性能显著提高，超过或与KAN相匹敌。然而，在MLP已经优于KAN的其他任务中，B样条并不能显著提升MLP的性能。此外，我们发现在标准的类增量连续学习环境中，KAN的遗忘问题比MLP更严重，这与KAN论文中报道的结果不同。我们希望这些结果能为未来对KAN和其他MLP替代方案的研究提供启示。项目链接：https://github.com/yu-rp/KANbeFair
English
This paper does not introduce a novel method. Instead, it offers a fairer and
more comprehensive comparison of KAN and MLP models across various tasks,
including machine learning, computer vision, audio processing, natural language
processing, and symbolic formula representation. Specifically, we control the
number of parameters and FLOPs to compare the performance of KAN and MLP. Our
main observation is that, except for symbolic formula representation tasks, MLP
generally outperforms KAN. We also conduct ablation studies on KAN and find
that its advantage in symbolic formula representation mainly stems from its
B-spline activation function. When B-spline is applied to MLP, performance in
symbolic formula representation significantly improves, surpassing or matching
that of KAN. However, in other tasks where MLP already excels over KAN,
B-spline does not substantially enhance MLP's performance. Furthermore, we find
that KAN's forgetting issue is more severe than that of MLP in a standard
class-incremental continual learning setting, which differs from the findings
reported in the KAN paper. We hope these results provide insights for future
research on KAN and other MLP alternatives. Project link:
https://github.com/yu-rp/KANbeFairSummary
AI-Generated Summary