SFT冲突，RL共存：大语言模型多任务学习的理论与实证分析

摘要

监督微调（SFT）与强化学习（RL）在提升大语言模型（LLM）的多任务推理能力时，表现出截然不同的行为特性。我们的初步实验揭示了一个现象：SFT 在多阶段训练下会遭受严重的任务冲突，而 RL 则能在不同任务之间实现稳定共存。在经验层面，我们将其追溯到参数级，观察到 RL 在不同任务上引起的更新是稀疏且近似正交的。我们通过分析多任务梯度干扰，为该机制提供了理论解释。结果表明二者存在本质区别：SFT 中的干扰受范数约束，随绝对梯度幅值缩放；而 RL 中的干扰受方差约束，其界限由优势归一化和在线策略优化所引入的梯度方差决定。这种较小的方差界限使得跨任务的优化方向近似正交。基于这一洞察，我们提出了 Parallel-RL，一种将多任务训练解耦的范式，显著提升了训练效率与灵活性。

English

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.