Brainstacks：基于冻结MoE-LoRA堆栈的跨领域认知能力实现持续大语言模型学习

Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learning

April 1, 2026

作者: Mohammad R. Abu Ayyash

cs.AI

摘要

我们提出Brainstacks——一种面向大语言模型持续多领域微调的模块化架构，该架构将领域专业知识封装为冻结的适配器堆栈，在推理时以叠加组合方式作用于共享的冻结基座模型。其包含五个核心组件：（1）采用QLoRA 4比特量化与rsLoRA缩放的MoE-LoRA，通过Shazeer式带噪声的Top-2路由机制覆盖全部七个Transformer投影层；（2）内部循环通过冻结已训练堆栈并叠加新堆栈实现残差提升；（3）外部循环按课程依赖顺序训练序列化领域专用堆栈；（4）基于随机SVD的零空间投影技术将新堆栈约束至与既往方向正交的子空间，实现完全隔离下的零遗忘；（5）基于经验发现的领域组合目标训练出的Sigmoid元路由器，可选择性加权堆栈以实现跨领域组合。两项边界实验：（6）在随机初始化模型上进行PSN预训练；（7）逐领域强化学习（DPO/GRPO）验证与SFT后对齐技术的兼容性。在TinyLlama-1.1B（4领域9堆栈）和Gemma 3 12B IT（5领域10堆栈）上的验证表明：MoE-LoRA收敛速度比参数量匹配的单LoRA快2.5倍，残差提升突破单堆栈性能天花板，路由系统可恢复因无门控堆栈累积而受损的生成质量。核心发现：基于结果的路由器揭示领域堆栈编码的是可迁移的认知基元（指令遵循清晰度、数值推理、程序逻辑、思维链结构）而非领域特定知识——医疗提示词在97%情况下被路由至聊天+数学堆栈，尽管这些堆栈未包含任何医疗数据。

English

We present Brainstacks, a modular architecture for continual multi-domain fine-tuning of large language models that packages domain expertise as frozen adapter stacks composing additively on a shared frozen base at inference. Five interlocking components: (1) MoE-LoRA with Shazeer-style noisy top-2 routing across all seven transformer projections under QLoRA 4-bit quantization with rsLoRA scaling; (2) an inner loop performing residual boosting by freezing trained stacks and adding new ones; (3) an outer loop training sequential domain-specific stacks with curriculum-ordered dependencies; (4) null-space projection via randomized SVD constraining new stacks to subspaces orthogonal to prior directions, achieving zero forgetting in isolation; (5) an outcome-based sigmoid meta-router trained on empirically discovered domain-combination targets that selectively weights stacks, enabling cross-domain composition. Two boundary experiments: (6) PSN pretraining on a randomly initialized model; (7) per-domain RL (DPO/GRPO) validating compatibility with post-SFT alignment. Validated on TinyLlama-1.1B (4 domains, 9 stacks) and Gemma 3 12B IT (5 domains, 10 stacks), MoE-LoRA achieves 2.5x faster convergence than parameter-matched single LoRA, residual boosting breaks through the single-stack ceiling, and the routed system recovers generation quality destroyed by ungated stack accumulation. The central finding: the outcome-based router discovers that domain stacks encode transferable cognitive primitives (instruction-following clarity, numerical reasoning, procedural logic, chain-of-thought structure) rather than domain-specific knowledge, with medical prompts routing to chat+math stacks in 97% of cases despite zero medical data in those stacks.