OmniCap-IF：全视频字幕生成中指令跟随能力的基准测试与改进

摘要

尽管全模态大语言模型（OLLMs）在联合处理音频和视频流方面展现了令人印象深刻的能力，但它们严格遵循复杂、多层面用户指令的能力仍尚未得到充分探索。现有基准主要关注整体视频理解或纯文本指令遵循，未能捕捉模态与用户约束之间的复杂交互。为填补这一空白，我们提出了OmniCap-IF，这是首个专门设计用于评估全模态字幕生成中指令遵循能力的综合性基准。OmniCap-IF采用系统化框架，从格式正确性和内容正确性两个维度评估字幕。我们的基准涵盖纯视觉、纯音频和音视频模态中的50种不同约束类型，同时集成时间定位以评估时空精度。对1920个高质量样本上代表性模型的广泛评估揭示了显著的性能差异。此外，我们的分析发现了一个关键的“格式-内容权衡”，表明增加格式复杂度会直接降低模型的全模态推理能力。最后，为推动该领域发展，我们整理了一个54K指令调优数据集OmniCap-IF-54K，并提出了OmniCaptioner-IF模型，该模型在复杂指令遵循和通用全模态字幕生成性能上均取得了显著提升。

English

While Omni-modal Large Language Models (OLLMs) have demonstrated impressive capabilities in jointly processing audio and visual streams, their ability to strictly adhere to complex, multi-faceted user instructions remains largely unexplored. Existing benchmarks primarily focus on holistic video understanding or text-only instruction following, failing to capture the intricate interplay between modalities and user constraints. To bridge this gap, we introduce OmniCap-IF, the first comprehensive benchmark specifically designed to evaluate instruction-following capabilities in omni-modal captioning. OmniCap-IF incorporates a systematic framework that assesses captions on two dimensions: format correctness and content correctness. Our benchmark encompasses 50 distinct constraint types across pure visual, pure audio, and audio-visual modalities, while integrating Temporal Grounding to assess spatio-temporal precision. Extensive evaluations of prominent models on 1,920 high-quality samples reveal significant performance disparities. Furthermore, our analysis uncovers a critical "format-content tradeoff", demonstrating that increasing formatting complexity directly degrades models' omni-modal reasoning abilities. Finally, to advance the field, we curate a 54K instruction-tuning dataset, OmniCap-IF-54K and present OmniCaptioner-IF, which achieves notable improvements in both complex instruction adherence and general omni-modal captioning performance.