MoTE：面向多任务视频理解的任务专家混合模型

摘要

程序性视频语言模型必须从相同的视觉证据中解决异构任务，包括动作识别、预测和程序预测。密集Transformer解码器在跨任务间共享相同的前馈网络，这可能导致任务行为相互纠缠，并使受控能力扩展变得困难。稀疏专家混合（MoE）解码器提供了条件计算能力，但基于token的学习路由与任务级程序性目标并不自然对齐。我们提出MoTE（Mixture of Task Experts），一种将大语言模型前馈网络转换为任务特定专家、同时保持多模态骨干网络共享的解码器架构。每个样本遵循一条样本级任务路由，因此活跃的任务专家计算量独立于所存储的任务专家数量。我们将这一设计实例化为VideoLLM-MoTE，并使用显式任务路由在五个COIN基准上对其进行评估。该五专家模型每个样本激活约2B的LLM参数，其平均top-1准确率高于近期VideoLLM基线方法。在相同专家拓扑下，它优于密集的全专家激活和学习的稀疏路由控制方法。这些结果表明，任务结构化路由为多任务视频语言学习提供了一种可解释且计算高效解码器替代方案。

English

Procedural video-language models must solve heterogeneous tasks from the same visual evidence, including action recognition, forecasting, and procedure prediction. Dense transformer decoders share the same feed-forward networks across tasks, which can entangle task behavior and make controlled capability expansion difficult. Sparse Mixture-of-Experts (MoE) decoders provide conditional computation, but token-level learned routing is not naturally aligned with task-level procedural objectives. We propose MoTE (Mixture of Task Experts), a decoder architecture that converts large language model feed-forward networks into task-specific experts while keeping the multimodal backbone shared. Each example follows one sample-level task route, so active task-expert computation remains independent of the number of stored task experts. We instantiate this design as VideoLLM-MoTE and evaluate it on five COIN benchmarks using explicit task routes. The five-expert model activates ~2B LLM parameters per sample and achieves higher average top-1 accuracy than recent VideoLLM baselines. Under the same expert topology, it improves over dense all-expert activation and learned sparse-routing controls. These results show that task-structured routing provides an interpretable and compute-efficient decoder alternative for multi-task video-language learning.