MTR-DuplexBench：面向全双工语音语言模型多轮对话能力的综合评估基准构建

MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models

April 17, 2026

作者: He Zhang, Wenqian Cui, Haoning Xu, Xiaohui Li, Lei Zhu, Haoli Bai, Shaohua Ma, Irwin King

cs.AI

摘要

全双工语音语言模型（FD-SLMs）能够实现实时重叠的对话交互，相比传统半双工模型提供更具动态性的用户体验。然而，现有基准主要关注单轮交互评估，忽略了多轮对话的复杂性。在多轮场景下评估FD-SLMs面临重大挑战，包括对话轮次边界模糊和模型推理中的上下文不一致问题。同时，现有基准往往仅聚焦对话特征评估，忽略了其他关键维度。为弥补这些不足，我们提出MTR-DuplexBench——一个专为FD-SLMs多轮综合评估设计的新型基准。该基准不仅将连续全双工对话分割为离散轮次进行逐轮评估，还整合了对话特征、对话质量、指令遵循能力和安全性等多维评价指标。实验结果表明，现有FD-SLMs在维持多轮次、多维度性能一致性方面存在困难，这验证了本基准的必要性与有效性。代码与数据详见：https://github.com/ZhangHe0918/MTR-DuplexBench

English

Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experience compared to traditional half-duplex models. However, existing benchmarks primarily focus on evaluating single-round interactions, neglecting the complexities of multi-round communication. Evaluating FD-SLMs in multi-round settings poses significant challenges, including blurred turn boundaries in communication and context inconsistency during model inference. Also, existing benchmarks often focus solely on evaluating conversational features, neglecting other critical aspects. To address these gaps, we introduce MTR-DuplexBench, a novel benchmark designed for a comprehensive multi-round evaluation of FD-SLMs. MTR-DuplexBench not only segments continuous full-duplex dialogues into discrete turns for turn-by-turn assessment but also incorporates various evaluation aspects, including conversational features, dialogue quality, instruction following, and safety. Experimental results reveal that current FD-SLMs face difficulties in maintaining consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our benchmark. Code and data are available at: https://github.com/ZhangHe0918/MTR-DuplexBench