聆听前瞻：下一补丁嵌入预测实现可扩展音频学习器

摘要

自监督学习（SSL）推动了音频表示学习的重大进展，尽管现有方法日益依赖精心设计的预训练方案以达到具有竞争力的性能。语言建模及近期视觉表示学习中最具影响力的进展背后，是一种截然不同的预训练理念：模型并非被训练为静态特征提取器，而是学习从先前上下文预测下一个元素——无论是离散词元还是连续嵌入。由此，自回归预测提供了一种跨模态迁移的统一预训练接口，迫使模型学习底层数据分布。鉴于音频的时间结构使得patch嵌入的自回归预测成为天然适配，我们探究这种简单的因果范式是否能够催生强大的音频学习器。我们提出NAPE（Next-Audio-Patch-Embedding prediction，下一音频patch嵌入预测），一种自监督框架，其中因果Transformer从先前的对数梅尔频谱图patch嵌入中预测下一个patch嵌入，仅以因果掩码和停止梯度作为训练信号。该设计刻意保持极简，避免了重建解码器、声学词元器、师生架构和辅助正则化损失。在六个音频和语音基准测试中，NAPE在多项任务上取得了最先进的微调性能，在不同编码器规模上表现出一致的可扩展性，并产生了强大的线性探测结果。NAPE还在无需显式监督的情况下形成了结构化的注意力模式。

English

Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.