前瞻聆聽：下一區塊嵌入預測賦予音訊學習模型可擴展性

摘要

自監督學習（SSL）已在音頻表徵學習中推動了實質性進展，儘管現有方法日益依賴繁複的預訓練方案，才能達到具有競爭力的性能。然而，一種截然不同的預訓練理念支撐了語言建模以及近期視覺表徵學習中最具影響力的進展：不是將編碼器訓練成靜態特徵提取器，而是訓練模型從先前上下文預測下一個元素——無論是離散詞元或連續嵌入。如此一來，自迴歸預測提供了一個統一的預訓練介面，可跨模態遷移，促使模型學習潛在的數據分佈。我們想探討：鑑於音頻的時間結構使區塊嵌入的自迴歸預測成為自然契合，這種簡單的因果範式能否產生強大的音頻學習器？我們提出了 NAPE（下一音頻區塊嵌入預測，Next-Audio-Patch-Embedding prediction），這是一個自監督框架，其中因果 Transformer 以因果遮罩和停止梯度作為唯一的訓練信號，根據先前的區塊嵌入逐個預測對數梅爾頻譜圖的下一個區塊嵌入。該設計刻意保持極簡，避免使用重建解碼器、聲學詞元器、師生架構以及輔助正則化損失。在六個音頻與語音基準上，NAPE 在多項任務上達到了最先進的微調性能，在不同編碼器大小下均一致地擴展，並在線性探測中展現了強勁的結果。此外，NAPE 在沒有顯式監督的情況下，也能產生結構化的注意力模式。

English

Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.