大型语言模型中基于超网络的知识注入的缩放定律

摘要

将事实知识大规模且可靠地注入大型语言模型仍是一个开放挑战。超网络为大规模知识注入提供了有前景的解决方案。尽管超网络通常用于测试时自适应，但我们探索其在训练时知识注入中的应用——即给定大规模事实语料库，训练超网络生成固定的LoRA适配器，当该适配器插入目标模型后，使模型能够回答关于这些事实的问题。本研究探究超网络能否用于执行训练时知识注入，以及这种能力如何随规模变化。超网络的缩放行为此前鲜有研究。我们的设计将超网络的注入能力与目标模型的通用能力解耦，从而首次能够严格研究超网络架构的缩放定律。我们刻画了损失、推理准确性及分布外泛化能力如何随超网络深度、宽度及目标网络规模变化。我们构建了大规模数据集MegaWikiQA，其中包含来自Wikidata5M样本的39个领域、数千万个多跳问答示例。研究结果揭示：(i) 基于超网络的注入在所有架构维度上展现出广泛的预测性幂律缩放规律；(ii) 超网络在规模增大时能够实现可靠的分布外泛化，表明超网络为LoRA微调和全微调等其他训练时自适应方法提供了有前景的替代方案，在所有分布外评估中表现出更陡峭的缩放指数。综合这些结果，超网络被确立为一种原理清晰且可扩展的训练时自适应基底，并首次提供了基于经验实证的缩放定律，以指导超网络在大型语言模型中的事实推理应用。

English

Injecting factual knowledge into large language models (LLMs) reliably and at scale remains an open challenge. Hypernetworks provide a promising solution to large-scale knowledge injection. Although hypernetworks are typically applied for test-time adaptation, we explore their use in train-time knowledge injection, where, given a large corpus of facts, we train a hypernetwork to generate a fixed LoRA adapter that, when inserted into the target model, enable the model to answer questions about those facts. In this work, we investigate whether hypernetworks can be used to perform train-time knowledge injection and how this ability varies with scale. The scaling behavior of hypernetworks remains largely unstudied. Our design decouples the hypernetwork's injection capacity from the target model's general capability, enabling, for the first time, a rigorous study of scaling laws for hypernetwork architectures. We characterize how loss, reasoning accuracy, and out-of-distribution (OOD) generalization vary with hypernetwork depth, width, and target network size. We construct a large-scale dataset, called MegaWikiQA, containing tens of millions of multi-hop question-answer examples across 39 domains constructed from examples in Wikidata5M. Our results reveal: (i) hypernetwork-based injection exhibits broadly predictive power law scaling along all architecture axes; and (ii) hypernetworks are capable of reliable OOD generalization at increasing scales, suggesting that hypernetwork provides a promising alternative to other train-time adaptation methods such as LoRA finetuning and full fine-tuning, exhibiting steeper scaling exponents in all OOD evaluations. Together, these results establish hypernetworks as a principled and scalable substrate for train-time adaptation, and provide the first empirically grounded scaling laws to guide hypernetworks for factual reasoning in large language models.