SFT 衝突，RL 共存：大型語言模型多任務學習的理論與實證分析

摘要

監督式微調（SFT）與強化學習（RL）在提升大型語言模型（LLMs）多任務推理能力時，表現出根本不同的行為。我們的初步實驗揭示了一項現象：SFT 在多階段訓練下會遭遇嚴重的任務衝突，而 RL 則能使多樣任務穩定共存。在經驗上，我們將其追溯到參數層面，觀察到 RL 在不同任務間會產生稀疏且近似正交的更新。我們透過分析多任務梯度干擾，為此機制提供了理論解釋。我們的結果揭示了一項區別：SFT 中的干擾是範數受限的，隨絕對梯度大小而擴增；而 RL 中的干擾則是變異數受限的，由優勢正規化與同策略最佳化所誘發的梯度變異數界定。此一微小變異數界限導致跨任務的最佳化方向近乎正交。基於此洞察，我們提出了 Parallel-RL，一種將多任務訓練解耦的範式，顯著提升了效率與靈活性。

English

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.