MuSViT：用於樂譜表示的基礎視覺模型

摘要

基礎模型透過提供豐富且可重複使用的表徵，能跨不同任務遷移，從而徹底改變了視覺與語言處理。然而，作為音樂語言視覺編碼的樂譜，卻缺乏如此強大的領域專用基礎骨幹。我們提出 MuSViT（Music Score Vision Transformer）：第一個針對樂譜表徵的基礎視覺模型——一個透過遮罩自編碼器，在來自 IMSLP 的 970 萬頁資料上預訓練的 ViT 編碼器。為應對真實世界樂譜的複雜性，我們採用兩階段課程學習：先在排版樂譜上進行合成暖身，再於完整的 IMSLP 語料庫上進行大規模訓練。我們在四項下游任務上評估 MuSViT——全頁與五線譜層級的樂譜辨識、音樂符號偵測及樂譜難度分類——並在兩種情境下進行：線性探測（凍結編碼器）與微調。在線性探測下，MuSViT 始終優於現代視覺編碼器，顯示通用表徵無論規模大小，在音樂記譜的結構性符號屬性上均系統性地存在不足。在微調下，MuSViT 普遍改進了特定任務的現有最佳方法。一項額外的嵌入-轉錄一致性分析揭示，MuSViT 在其表徵空間中直接編碼了音樂符號結構——而其他編碼器的嵌入則與樂譜記譜內容無關。這些結果確立 MuSViT 作為樂譜理解的基礎骨幹。

English

Foundation models have transformed vision and language processing by providing rich, reusable representations that transfer across diverse tasks. Sheet music, as a visual encoding of musical language, lacks such a strong domain-specific backbone. We introduce MuSViT (Music Score Vision Transformer): the first foundation vision model for sheet music representation -- a ViT encoder pre-trained via Masked Autoencoders on 9.7 million pages from the IMSLP. To handle the complexity of real-world scores, we adopt a two-stage curriculum: a synthetic warm-up on typeset scores followed by large-scale training on the full IMSLP corpus. We evaluate MuSViT on four downstream tasks -- full-page and staff-level music score recognition, music symbol detection, and score difficulty classification -- under two scenarios: linear probing (frozen encoder) and fine-tuning. Under linear probing, MuSViT consistently outperforms modern vision encoders, revealing that general-purpose representations, regardless of scale, fall systematically short on the structured symbolic properties of musical notation. Under fine-tuning, MuSViT generally improves upon task-specific state-of-the-art methods. An additional embedding-transcription consistency analysis reveals that MuSViT encodes symbolic musical structure directly in its representation space -- unlike other encoders, whose embeddings do not correlate with music notation content. These results establish MuSViT as a foundation backbone for sheet music understanding.