MPJudge：面向音乐诱导绘画的感知评估框架

摘要

音乐诱导绘画是一种独特的艺术实践，指在音乐影响下创作视觉艺术作品。评估画作是否忠实反映其灵感来源的音乐，是一项具有挑战性的感知评价任务。现有方法主要依赖情感识别模型来评估音乐与绘画的相似性，但这类模型会引入显著噪声，且忽略了情感之外的更广泛感知线索。为突破这些局限，我们提出了一种新颖的音乐诱导绘画评估框架，直接建模音乐与视觉艺术之间的感知一致性。我们发布了MPD数据集——首个由领域专家基于感知一致性标注的大规模音乐-绘画配对数据集。为更好地处理模糊案例，我们进一步收集了成对偏好标注数据。基于该数据集，我们开发了MPJudge模型，通过基于调制的融合机制将音乐特征整合到视觉编码器中。为有效学习模糊案例，我们采用直接偏好优化方法进行训练。大量实验表明，本方法优于现有方案。定性结果进一步显示，我们的模型能更精准地识别绘画中与音乐相关的区域。

English

Music induced painting is a unique artistic practice, where visual artworks are created under the influence of music. Evaluating whether a painting faithfully reflects the music that inspired it poses a challenging perceptual assessment task. Existing methods primarily rely on emotion recognition models to assess the similarity between music and painting, but such models introduce considerable noise and overlook broader perceptual cues beyond emotion. To address these limitations, we propose a novel framework for music induced painting assessment that directly models perceptual coherence between music and visual art. We introduce MPD, the first large scale dataset of music painting pairs annotated by domain experts based on perceptual coherence. To better handle ambiguous cases, we further collect pairwise preference annotations. Building on this dataset, we present MPJudge, a model that integrates music features into a visual encoder via a modulation based fusion mechanism. To effectively learn from ambiguous cases, we adopt Direct Preference Optimization for training. Extensive experiments demonstrate that our method outperforms existing approaches. Qualitative results further show that our model more accurately identifies music relevant regions in paintings.

MPJudge：面向音乐诱导绘画的感知评估框架

MPJudge: Towards Perceptual Assessment of Music-Induced Paintings

摘要

Support