龙之幼雏：Transformer与大脑模型间的缺失纽带

摘要

自约翰·冯·诺依曼与艾伦·图灵以来，计算系统与大脑之间的关系一直是理论先驱们探索的驱动力。像大脑这样均匀、无标度的生物网络，具备强大的特性，包括随时间推移的泛化能力，而这正是机器学习迈向通用推理模型道路上的主要障碍。我们推出“龙雏”（BDH），一种基于无标度生物启发网络的新型大语言模型架构，该网络由局部交互的神经元粒子构成。BDH在保持类似Transformer性能的同时，结合了坚实的理论基础与内在的可解释性。 BDH是一种实用、高效的最先进注意力机制状态空间序列学习架构。除了作为图模型外，BDH还具备GPU友好的实现形式。它展现出与Transformer相似的扩展规律：实证表明，在相同参数规模（从1000万到10亿）及相同训练数据下，BDH在语言和翻译任务上的表现可与GPT2相媲美。 BDH可视为一种大脑模型。在推理过程中，BDH的工作记忆完全依赖于使用尖峰神经元的赫布学习突触可塑性。我们通过实验证实，在处理语言输入时，每当BDH“听到”或“思考”特定概念，特定的单个突触会增强连接。BDH的神经元交互网络是一个具有重尾度分布的高模块化图。BDH模型在生物学上具有合理性，解释了人类神经元可能用于实现言语的一种机制。 BDH设计注重可解释性。其激活向量稀疏且为正。我们在语言任务中展示了BDH的单义性。状态的可解释性，超越了神经元和模型参数的可解释性，是BDH架构的固有特性。

English

The relationship between computing systems and the brain has served as motivation for pioneering theoreticians since John von Neumann and Alan Turing. Uniform, scale-free biological networks, such as the brain, have powerful properties, including generalizing over time, which is the main barrier for Machine Learning on the path to Universal Reasoning Models. We introduce `Dragon Hatchling' (BDH), a new Large Language Model architecture based on a scale-free biologically inspired network of \n locally-interacting neuron particles. BDH couples strong theoretical foundations and inherent interpretability without sacrificing Transformer-like performance. BDH is a practical, performant state-of-the-art attention-based state space sequence learning architecture. In addition to being a graph model, BDH admits a GPU-friendly formulation. It exhibits Transformer-like scaling laws: empirically BDH rivals GPT2 performance on language and translation tasks, at the same number of parameters (10M to 1B), for the same training data. BDH can be represented as a brain model. The working memory of BDH during inference entirely relies on synaptic plasticity with Hebbian learning using spiking neurons. We confirm empirically that specific, individual synapses strengthen connection whenever BDH hears or reasons about a specific concept while processing language inputs. The neuron interaction network of BDH is a graph of high modularity with heavy-tailed degree distribution. The BDH model is biologically plausible, explaining one possible mechanism which human neurons could use to achieve speech. BDH is designed for interpretability. Activation vectors of BDH are sparse and positive. We demonstrate monosemanticity in BDH on language tasks. Interpretability of state, which goes beyond interpretability of neurons and model parameters, is an inherent feature of the BDH architecture.

龙之幼雏：Transformer与大脑模型间的缺失纽带

The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain

摘要

Support