通过LLM代理实现端到端同时语音翻译的人类水平

Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent

July 31, 2024

作者: Shanbo Cheng, Zhichao Huang, Tom Ko, Hang Li, Ningxin Peng, Lu Xu, Qini Zhang

cs.AI

摘要

本文介绍了Cross Language Agent - Simultaneous Interpretation（CLASI），这是一个高质量且类人化的Simultaneous Speech Translation（SiST）系统。受专业人类口译员启发，我们采用了一种新颖的数据驱动的读写策略来平衡翻译质量和延迟。为了解决领域术语翻译的挑战，CLASI采用了多模态检索模块来获取相关信息以增强翻译。在LLMs的支持下，我们的方法可以通过考虑输入音频、历史上下文和检索到的信息来生成容错翻译。实验结果显示，我们的系统在性能上明显优于其他系统。与专业人类口译员一致，我们使用更好的人类评估指标VIP（valid information proportion）来评估CLASI，该指标衡量了成功传达给听众的信息量。在现实场景中，演讲通常不流畅、非正式且不清晰，在中译英和英译中方向上，CLASI分别实现了81.3%和78.0%的VIP。相比之下，最先进的商业或开源系统仅实现了35.4%和41.6%。在其他系统仅实现不到13% VIP的极具挑战的数据集上，CLASI仍然可以实现70%的VIP。

English

In this paper, we present Cross Language Agent -- Simultaneous Interpretation, CLASI, a high-quality and human-like Simultaneous Speech Translation (SiST) System. Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency. To address the challenge of translating in-domain terminologies, CLASI employs a multi-modal retrieving module to obtain relevant information to augment the translation. Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information. Experimental results show that our system outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate CLASI with a better human evaluation metric, valid information proportion (VIP), which measures the amount of information that can be successfully conveyed to the listeners. In the real-world scenarios, where the speeches are often disfluent, informal, and unclear, CLASI achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese translation directions, respectively. In contrast, state-of-the-art commercial or open-source systems only achieve 35.4% and 41.6%. On the extremely hard dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70% VIP.