龍之幼雛：Transformer與大腦模型之間缺失的連結

摘要

自約翰·馮·諾伊曼與艾倫·圖靈以來，計算系統與大腦之間的關係一直是理論先驅們的靈感源泉。均勻且無標度的生物網絡，如大腦，具備強大的特性，包括隨時間推移的泛化能力，這正是機器學習邁向通用推理模型之路上的主要障礙。我們推出“龍之雛”（BDH），這是一種基於無標度生物啟發網絡的新型大型語言模型架構，該網絡由局部交互的神經元粒子構成。BDH在保持類似Transformer性能的同時，結合了堅實的理論基礎與內在的可解釋性。 BDH是一種實用且性能卓越的、基於注意力機制的狀態空間序列學習架構，處於技術前沿。除了作為圖模型外，BDH還具備GPU友好的實現形式。它展現出與Transformer相似的規模法則：在相同參數量（從1000萬到10億）及相同訓練數據下，BDH在語言與翻譯任務上的表現可與GPT2匹敵。 BDH可被視為一種大腦模型。在推理過程中，BDH的工作記憶完全依賴於使用尖峰神經元的赫布學習所實現的突觸可塑性。我們通過實驗證實，當BDH在處理語言輸入時聽到或推理特定概念時，特定的個別突觸會增強連接。BDH的神經元交互網絡是一個高度模塊化且具有重尾度分佈的圖。BDH模型在生物學上是合理的，它解釋了人類神經元可能用於實現言語的一種機制。 BDH專為可解釋性而設計。其激活向量稀疏且為正。我們在語言任務上展示了BDH的單義性。狀態的可解釋性，超越了神經元與模型參數的可解釋性，是BDH架構的固有特性。

English

The relationship between computing systems and the brain has served as motivation for pioneering theoreticians since John von Neumann and Alan Turing. Uniform, scale-free biological networks, such as the brain, have powerful properties, including generalizing over time, which is the main barrier for Machine Learning on the path to Universal Reasoning Models. We introduce `Dragon Hatchling' (BDH), a new Large Language Model architecture based on a scale-free biologically inspired network of \n locally-interacting neuron particles. BDH couples strong theoretical foundations and inherent interpretability without sacrificing Transformer-like performance. BDH is a practical, performant state-of-the-art attention-based state space sequence learning architecture. In addition to being a graph model, BDH admits a GPU-friendly formulation. It exhibits Transformer-like scaling laws: empirically BDH rivals GPT2 performance on language and translation tasks, at the same number of parameters (10M to 1B), for the same training data. BDH can be represented as a brain model. The working memory of BDH during inference entirely relies on synaptic plasticity with Hebbian learning using spiking neurons. We confirm empirically that specific, individual synapses strengthen connection whenever BDH hears or reasons about a specific concept while processing language inputs. The neuron interaction network of BDH is a graph of high modularity with heavy-tailed degree distribution. The BDH model is biologically plausible, explaining one possible mechanism which human neurons could use to achieve speech. BDH is designed for interpretability. Activation vectors of BDH are sparse and positive. We demonstrate monosemanticity in BDH on language tasks. Interpretability of state, which goes beyond interpretability of neurons and model parameters, is an inherent feature of the BDH architecture.

龍之幼雛：Transformer與大腦模型之間缺失的連結

The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain

摘要

Support