MoTE：面向多任務影片理解的任務專家混合模型

摘要

程序性視頻語言模型必須從相同的視覺證據中解決異構任務，包括動作識別、未來預測和程序預測。密集Transformer解碼器在不同任務之間共享相同的前饋網絡，這可能糾纏任務行為，並使受控的能力擴展變得困難。稀疏混合專家（MoE）解碼器提供條件計算，但token級學習路由與任務級程序目標並不能自然對齊。我們提出MoTE（任務專家混合），這是一種解碼器架構，將大語言模型的前饋網絡轉換為任務特定的專家，同時保持多模態骨幹共享。每個樣本遵循一條樣本級任務路由，因此活躍的任務專家計算量與所存儲的任務專家數量無關。我們將此設計實例化為VideoLLM-MoTE，並在五個COIN基準上使用顯式任務路由對其進行評估。該五專家模型每個樣本激活約20億個LLM參數，並實現了比近期VideoLLM基線更高的平均top-1準確率。在相同的專家拓撲下，它優於密集的全專家激活和學習的稀疏路由控制。這些結果表明，任務結構化路由為多任務視頻語言學習提供了一種可解釋且計算高效的解碼器替代方案。

English

Procedural video-language models must solve heterogeneous tasks from the same visual evidence, including action recognition, forecasting, and procedure prediction. Dense transformer decoders share the same feed-forward networks across tasks, which can entangle task behavior and make controlled capability expansion difficult. Sparse Mixture-of-Experts (MoE) decoders provide conditional computation, but token-level learned routing is not naturally aligned with task-level procedural objectives. We propose MoTE (Mixture of Task Experts), a decoder architecture that converts large language model feed-forward networks into task-specific experts while keeping the multimodal backbone shared. Each example follows one sample-level task route, so active task-expert computation remains independent of the number of stored task experts. We instantiate this design as VideoLLM-MoTE and evaluate it on five COIN benchmarks using explicit task routes. The five-expert model activates ~2B LLM parameters per sample and achieves higher average top-1 accuracy than recent VideoLLM baselines. Under the same expert topology, it improves over dense all-expert activation and learned sparse-routing controls. These results show that task-structured routing provides an interpretable and compute-efficient decoder alternative for multi-task video-language learning.