MuSViT：一种用于乐谱表征的基础视觉模型

摘要

基础模型通过提供丰富且可复用的表征，可在不同任务间迁移，从而彻底改变了视觉和语言处理领域。然而，作为音乐语言的视觉编码形式，乐谱却缺乏此类强大的领域专用基础模型。我们提出MuSViT（乐谱视觉变换器）：首个面向乐谱表征的基础视觉模型——该模型采用基于掩码自编码器预训练的ViT编码器，训练数据来自IMSLP的970万页乐谱。为应对真实世界乐谱的复杂性，我们采用两阶段课程学习策略：先在排版乐谱上进行合成数据预热训练，再基于完整IMSLP语料库进行大规模训练。我们在四个下游任务（全页面及谱线级乐谱识别、音乐符号检测、乐谱难度分类）上评估MuSViT，并设置两种场景：线性探测（冻结编码器）与微调。在线性探测中，MuSViT持续优于现代视觉编码器，表明无论规模大小，通用表征在音乐记谱的结构化符号特性上均存在系统性不足。在微调场景下，MuSViT通常能改进任务特定的现有最优方法。此外，嵌入-转录一致性分析揭示：MuSViT能直接在表征空间中编码符号化的音乐结构——而其他编码器的嵌入与音乐记谱内容并无关联。这些结果确立了MuSViT作为乐谱理解领域基础骨干模型的地位。

English

Foundation models have transformed vision and language processing by providing rich, reusable representations that transfer across diverse tasks. Sheet music, as a visual encoding of musical language, lacks such a strong domain-specific backbone. We introduce MuSViT (Music Score Vision Transformer): the first foundation vision model for sheet music representation -- a ViT encoder pre-trained via Masked Autoencoders on 9.7 million pages from the IMSLP. To handle the complexity of real-world scores, we adopt a two-stage curriculum: a synthetic warm-up on typeset scores followed by large-scale training on the full IMSLP corpus. We evaluate MuSViT on four downstream tasks -- full-page and staff-level music score recognition, music symbol detection, and score difficulty classification -- under two scenarios: linear probing (frozen encoder) and fine-tuning. Under linear probing, MuSViT consistently outperforms modern vision encoders, revealing that general-purpose representations, regardless of scale, fall systematically short on the structured symbolic properties of musical notation. Under fine-tuning, MuSViT generally improves upon task-specific state-of-the-art methods. An additional embedding-transcription consistency analysis reveals that MuSViT encodes symbolic musical structure directly in its representation space -- unlike other encoders, whose embeddings do not correlate with music notation content. These results establish MuSViT as a foundation backbone for sheet music understanding.