多任务多帧视觉钢琴转写

摘要

基于音频的钢琴转写在起音、音高和力度上表现良好，但延音踏板使声音在琴键释放后仍持续很久，因此音频系统预测的是踏板延长的音符结束偏移，而非物理上的琴键释放。然而，现有的视觉钢琴转写（VPT）系统侧重于从短视频窗口中进行起音检测，其偏移准确度远落后于起音，且尚未见音符级力度预测的相关报道。为解决上述不足，我们提出了V2N（Video to Notes），这是首个完整的VPT系统：一个共享的时间骨干网络为起音、偏移、按键保持和力度等任务特定头部提供特征，并通过逐帧监督而非仅在窗口中心进行联合训练。消融实验表明，多任务监督能够在提升起音准确度的同时实现偏移和力度预测；更长的时序上下文可带来进一步改进。V2N在PianoVAM和R3数据集上取得了新的最先进结果。

English

Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.