演化和機器學習的奈特盲點

Evolution and The Knightian Blindspot of Machine Learning

January 22, 2025

作者: Joel Lehman, Elliot Meyerson, Tarek El-Gaaly, Kenneth O. Stanley, Tarin Ziyaee

cs.AI

摘要

本文指出機器學習（ML）在很大程度上忽略了普遍智能的一個重要方面：對未知未來的魯棒性在開放世界中。這種魯棒性與經濟學中的Knightian不確定性（KU）有關，即無法量化的不確定性，在ML的關鍵形式主義中被排除在考慮之外。本文旨在識別這個盲點，論證其重要性，並促進研究以應對這一挑戰，我們認為這對於創建真正魯棒的開放世界AI是必要的。為了幫助闡明這個盲點，我們將機器學習的一個領域，強化學習（RL），與生物進化過程進行對比。儘管RL取得了驚人的持續進展，但在開放世界的情況下仍然面臨困難，通常在意想不到的情況下失敗。例如，將僅在美國接受培訓的自駕車政策零-shot轉移到英國的想法目前看來非常雄心勃勃。戲劇性對比的是，生物進化經常產生在開放世界中茁壯成長的代理人，有時甚至適應了非常不同的情況（例如入侵物種；或者人類，他們確實進行這種零-shot國際駕駛）。有趣的是，進化實現這種魯棒性而無需明確的理論、形式主義或數學梯度。我們探討了支撐RL典型形式主義的假設，展示了它們如何限制RL與不斷變化的複雜世界所特有的未知未知的互動。此外，我們確定了進化過程促進對新奇和不可預測挑戰的魯棒性的機制，並討論了將這些機制算法化的潛在途徑。結論是，ML仍存在引人入勝的脆弱性可能是由於其形式主義中的盲點，直接面對KU挑戰可能會帶來顯著收益。

English

This paper claims that machine learning (ML) largely overlooks an important facet of general intelligence: robustness to a qualitatively unknown future in an open world. Such robustness relates to Knightian uncertainty (KU) in economics, i.e. uncertainty that cannot be quantified, which is excluded from consideration in ML's key formalisms. This paper aims to identify this blind spot, argue its importance, and catalyze research into addressing it, which we believe is necessary to create truly robust open-world AI. To help illuminate the blind spot, we contrast one area of ML, reinforcement learning (RL), with the process of biological evolution. Despite staggering ongoing progress, RL still struggles in open-world situations, often failing under unforeseen situations. For example, the idea of zero-shot transferring a self-driving car policy trained only in the US to the UK currently seems exceedingly ambitious. In dramatic contrast, biological evolution routinely produces agents that thrive within an open world, sometimes even to situations that are remarkably out-of-distribution (e.g. invasive species; or humans, who do undertake such zero-shot international driving). Interestingly, evolution achieves such robustness without explicit theory, formalisms, or mathematical gradients. We explore the assumptions underlying RL's typical formalisms, showing how they limit RL's engagement with the unknown unknowns characteristic of an ever-changing complex world. Further, we identify mechanisms through which evolutionary processes foster robustness to novel and unpredictable challenges, and discuss potential pathways to algorithmically embody them. The conclusion is that the intriguing remaining fragility of ML may result from blind spots in its formalisms, and that significant gains may result from direct confrontation with the challenge of KU.