HumanSense：從多模態感知到基於推理的多模態大語言模型之共情情境感知回應
HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs
August 14, 2025
作者: Zheng Qin, Ruobing Zheng, Yabing Wang, Tianqi Li, Yi Yuan, Jingdong Chen, Le Wang
cs.AI
摘要
儘管多模態大型語言模型（MLLMs）在實現真正類人互動方面展現出巨大潛力，但由於缺乏針對以人為本場景的細粒度評估框架，進展受到阻礙。這些框架需涵蓋對複雜人類意圖的理解以及提供富有同理心、情境感知的回應。為此，我們引入了HumanSense，這是一個全面的基準測試，旨在評估MLLMs在以人為本的感知與互動能力，特別關注對擴展多模態語境的深度理解及理性反饋的構建。我們的評估顯示，領先的MLLMs在面向高級互動任務上仍有顯著提升空間。通過補充視覺輸入與音頻及文本信息，可帶來實質性改進，而全模態模型在這些任務上展現出優勢。此外，我們認為恰當的反饋源於對對話者需求與情感的語境分析，推理能力則是解鎖這一能力的關鍵。因此，我們採用多階段、模態漸進的強化學習來增強全模態模型的推理能力，在評估結果上取得了顯著提升。同時，我們觀察到成功的推理過程呈現出高度一致的思維模式。通過設計相應的提示，我們也以無需訓練的方式提升了非推理模型的表現。項目頁面：brightpinkhttps://digital-avatar.github.io/ai/HumanSense/
English
While Multimodal Large Language Models (MLLMs) show immense promise for
achieving truly human-like interactions, progress is hindered by the lack of
fine-grained evaluation frameworks for human-centered scenarios, encompassing
both the understanding of complex human intentions and the provision of
empathetic, context-aware responses. Here we introduce HumanSense, a
comprehensive benchmark designed to evaluate the human-centered perception and
interaction capabilities of MLLMs, with a particular focus on deep
understanding of extended multimodal contexts and the formulation of rational
feedback. Our evaluation reveals that leading MLLMs still have considerable
room for improvement, particularly for advanced interaction-oriented tasks.
Supplementing visual input with audio and text information yields substantial
improvements, and Omni-modal models show advantages on these tasks.
Furthermore, we argue that appropriate feedback stems from a contextual
analysis of the interlocutor's needs and emotions, with reasoning ability
serving as the key to unlocking it. Accordingly, we employ a multi-stage,
modality-progressive reinforcement learning to enhance the reasoning abilities
of an Omni model, achieving substantial gains on evaluation results.
Additionally, we observe that successful reasoning processes exhibit highly
consistent thought patterns. By designing corresponding prompts, we also
enhance the performance of non-reasoning models in a training-free manner.
Project page:
brightpinkhttps://digital-avatar.github.io/ai/HumanSense/