探究大型音频语言模型在说话者情绪变化下的安全漏洞

摘要

大型音頻語言模型（LALMs）在基於文本的大語言模型基礎上擴展了聽覺理解能力，為多模態應用開闢了新途徑。儘管其感知、推理和任務執行能力已得到廣泛研究，但副語言變異下的安全對齊問題仍待深入探索。本研究系統性探討了說話者情緒的影響：我們構建了包含多種情緒及強度表達的惡意語音指令數據集，並評估了多個前沿LALMs。結果顯示顯著的安全不一致性——不同情緒會引發不同程度的非安全回應，且強度影響呈非單調性，中等情緒表達往往構成最大風險。這些發現揭示了LALMs中被忽視的脆弱性，呼籲需要專門設計的對齊策略來確保情緒變異下的穩健性，這是實現現實場景中可信部署的必要前提。

English

Large audio-language models (LALMs) extend text-based LLMs with auditory understanding, offering new opportunities for multimodal applications. While their perception, reasoning, and task performance have been widely studied, their safety alignment under paralinguistic variation remains underexplored. This work systematically investigates the role of speaker emotion. We construct a dataset of malicious speech instructions expressed across multiple emotions and intensities, and evaluate several state-of-the-art LALMs. Our results reveal substantial safety inconsistencies: different emotions elicit varying levels of unsafe responses, and the effect of intensity is non-monotonic, with medium expressions often posing the greatest risk. These findings highlight an overlooked vulnerability in LALMs and call for alignment strategies explicitly designed to ensure robustness under emotional variation, a prerequisite for trustworthy deployment in real-world settings.

探究大型音频语言模型在说话者情绪变化下的安全漏洞

Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variations

摘要

Support