朝向透過LLM代理實現端到端同步語音翻譯的人類水準

摘要

本文介紹了跨語言代理 - 同時口譯（CLASI），一個高質量且類似人類的同步語音翻譯（SiST）系統。受專業人類口譯員的啟發，我們利用一種新型的數據驅動讀寫策略來平衡翻譯質量和延遲時間。為應對在領域術語翻譯方面的挑戰，CLASI採用多模檢索模塊來獲取相關信息以增強翻譯。在LLMs的支持下，我們的方法可以通過考慮輸入音頻、歷史上下文和檢索信息來生成容錯翻譯。實驗結果表明，我們的系統在性能上明顯優於其他系統。與專業人類口譯員一致，我們使用更好的人類評估指標 - 有效信息比例（VIP），該指標衡量了成功傳達給聽眾的信息量。在現實情境中，演講通常不流暢、非正式且不清晰，CLASI在中英文和英中文翻譯方向分別實現了81.3%和78.0%的VIP。相比之下，最先進的商業或開源系統僅實現了35.4%和41.6%。在其他系統僅實現不到13% VIP的極其困難的數據集上，CLASI仍然可以實現70%的VIP。

English

In this paper, we present Cross Language Agent -- Simultaneous Interpretation, CLASI, a high-quality and human-like Simultaneous Speech Translation (SiST) System. Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency. To address the challenge of translating in-domain terminologies, CLASI employs a multi-modal retrieving module to obtain relevant information to augment the translation. Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information. Experimental results show that our system outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate CLASI with a better human evaluation metric, valid information proportion (VIP), which measures the amount of information that can be successfully conveyed to the listeners. In the real-world scenarios, where the speeches are often disfluent, informal, and unclear, CLASI achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese translation directions, respectively. In contrast, state-of-the-art commercial or open-source systems only achieve 35.4% and 41.6%. On the extremely hard dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70% VIP.