基于音频嵌入的品味感知音乐检索

摘要

声音与味觉之间的跨模态对应在心理学和神经科学领域已得到充分证实，但在基于内容的多媒体检索中却鲜有涉及。本文将"从音频预测味觉"形式化为一个基于内容音乐信息检索的基准测试，基于经过感知验证的多源语料库，在共享多任务回归头下比较了来自四个HEAR家族的十种冻结音频编码器，并以门控晚期融合作为可配置变体。为评估模型有效性，我们计算了绝对误差与秩相关系数。最强系统对五种味觉的预测宏观均方根误差（RMSE）为0.134；在未参与训练的真实音乐数据上，其误差低于单个评分者偏离共识的一半（RMSE 0.13对比0.28），即模型追踪群体共识的精准度超越平均人类评分者，且远低于先前最先进基线（0.219）。在绝对误差上各编码器统计表现持平，单个VGGish即可媲美最优融合效果，但门控晚期融合的优势仅限于秩相关（宏观皮尔逊r 0.724对比0.666）。作为基于内容的检索索引运行时，预测味觉空间对309项曲目池的排序忠实度远高于CLAP文本基线（后者仅达随机水平）；脊回归探针与音频带阻消音实验揭示了最强表征与已有声音-味觉对应关系的匹配情况。

English

Crossmodal correspondences between sound and taste are well established in psychology and neuroscience, but largely absent from content-based multimedia retrieval. We formalise taste-from-audio prediction as a content-based music information retrieval benchmark over a perceptually validated multi-source corpus, comparing ten frozen audio encoders from the four HEAR families under a shared multi-task regression head, with gated late-fusion as a configurable variant. In order to assess the effectiveness of the models, we compute absolute error and rank correlation. The strongest systems predict the five tastes within a macro RMSE of 0.134; on held-out real music their error is less than half a single rater's deviation from the consensus (RMSE 0.13 vs. 0.28), so the model tracks the group consensus more closely than an average human rater, and well below the previous state of the art baseline (0.219). On absolute error the encoders are statistically flat, with a single VGGish matching the best fusion, but gated late-fusion's advantage is confined to rank correlation (macro Pearson r 0.724 vs. 0.666). Operationalised as a content-based retrieval index, the predicted taste space ranks a 309-item pool far more faithfully than a CLAP-text baseline, which sits at chance; ridge probes and an audio-bandstop knockout read the strongest representations against documented sound-taste correspondences.