DialSim：一個用於評估對話代理人長期對話理解能力的即時模擬器

DialSim: A Real-Time Simulator for Evaluating Long-Term Dialogue Understanding of Conversational Agents

June 19, 2024

作者: Jiho Kim, Woosog Chay, Hyeonji Hwang, Daeun Kyung, Hyunseung Chung, Eunbyeol Cho, Yohan Jo, Edward Choi

cs.AI

摘要

最近大型語言模型（LLMs）的進步顯著提升了對話代理的能力，使其適用於各個領域（例如教育）。儘管取得了進展，對代理的評估通常忽略了真實世界對話的複雜性，例如實時互動、多方對話和延伸的語境依賴。為彌合這一差距，我們引入了 DialSim，一個實時對話模擬器。在這個模擬器中，一個代理被指派扮演流行電視節目中的角色，需要利用過去的對話信息回答即興問題，並區分已知和未知信息。DialSim 的關鍵特點包括評估代理在合理時間限制內回應的能力、處理長期多方對話，以及管理對抗環境（例如交換角色名稱）以挑戰代理對預訓練知識的依賴。我們利用這個模擬器來評估最新的對話代理並分析它們的限制。我們的實驗突顯了這些代理的優勢和劣勢，為未來改進對話人工智慧領域提供了有價值的見解。DialSim 可在 https://github.com/jiho283/Simulator 下載。

English

Recent advancements in Large Language Models (LLMs) have significantly enhanced the capabilities of conversational agents, making them applicable to various fields (e.g., education). Despite their progress, the evaluation of the agents often overlooks the complexities of real-world conversations, such as real-time interactions, multi-party dialogues, and extended contextual dependencies. To bridge this gap, we introduce DialSim, a real-time dialogue simulator. In this simulator, an agent is assigned the role of a character from popular TV shows, requiring it to respond to spontaneous questions using past dialogue information and to distinguish between known and unknown information. Key features of DialSim include evaluating the agent's ability to respond within a reasonable time limit, handling long-term multi-party dialogues, and managing adversarial settings (e.g., swap character names) to challenge the agent's reliance on pre-trained knowledge. We utilized this simulator to evaluate the latest conversational agents and analyze their limitations. Our experiments highlight both the strengths and weaknesses of these agents, providing valuable insights for future improvements in the field of conversational AI. DialSim is available at https://github.com/jiho283/Simulator.