VersatileFFN：通过自适应宽深度复用实现大语言模型的参数高效化

VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse

December 16, 2025

作者: Ying Nie, Kai Han, Hongguang Li, Hang Zhou, Tianyu Guo, Enhua Wu, Xinghao Chen, Yunhe Wang

cs.AI

摘要

大型语言模型（LLM）的快速扩展虽取得了显著性能，但也带来了高昂的内存成本。现有的参数高效方法（如剪枝和量化）主要对预训练模型进行压缩，并未增强架构能力，因此会触及基础模型的表征能力上限。本文提出VersatileFFN——一种新颖的前馈网络（FFN），能够在固定参数预算下实现参数在宽度和深度维度上的灵活复用。受认知双过程理论启发，VersatileFFN包含两条自适应路径：宽度自适应路径通过单一共享FFN生成混合子专家，在不增加参数的情况下模拟稀疏专家路由机制；深度自适应路径通过递归应用同一FFN模拟对复杂标记的深层处理。难度感知门控机制动态平衡两条路径，引导"简单"标记通过高效的宽度路径处理，同时为"困难"标记分配更深层的迭代优化。关键在于两条路径复用相同参数，所有额外能力均来自计算而非内存开销。跨多基准测试和模型规模的实验验证了该方法的有效性。代码发布于https://github.com/huawei-noah/noah-research/tree/master/VersatileFFN。

English

The rapid scaling of Large Language Models (LLMs) has achieved remarkable performance, but it also leads to prohibitive memory costs. Existing parameter-efficient approaches such as pruning and quantization mainly compress pretrained models without enhancing architectural capacity, thereby hitting the representational ceiling of the base model. In this work, we propose VersatileFFN, a novel feed-forward network (FFN) that enables flexible reuse of parameters in both width and depth dimensions within a fixed parameter budget. Inspired by the dual-process theory of cognition, VersatileFFN comprises two adaptive pathways: a width-versatile path that generates a mixture of sub-experts from a single shared FFN, mimicking sparse expert routing without increasing parameters, and a depth-versatile path that recursively applies the same FFN to emulate deeper processing for complex tokens. A difficulty-aware gating dynamically balances the two pathways, steering "easy" tokens through the efficient width-wise route and allocating deeper iterative refinement to "hard" tokens. Crucially, both pathways reuse the same parameters, so all additional capacity comes from computation rather than memory. Experiments across diverse benchmarks and model scales demonstrate the effectiveness of the method. The code will be available at https://github.com/huawei-noah/noah-research/tree/master/VersatileFFN.