转换器工作记忆中符号表示的复杂性与任务的复杂性相关。

Complexity of Symbolic Representation in Working Memory of Transformer Correlates with the Complexity of a Task

June 20, 2024

作者: Alsu Sagirova, Mikhail Burtsev

cs.AI

摘要

尽管Transformers被广泛用于自然语言处理任务，尤其是机器翻译，但它们缺乏明确的记忆来存储已处理文本的关键概念。本文探讨了在Transformer模型解码器中添加符号工作记忆内容的特性。这种工作记忆提升了模型在机器翻译任务中的预测质量，并作为神经符号化信息的表示，对模型进行正确翻译至关重要。记忆内容的研究揭示了翻译文本关键词被存储在工作记忆中，指向记忆内容与已处理文本相关性的重要性。此外，存储在记忆中的标记和词性的多样性与用于机器翻译任务的语料库复杂性相关。

English

Even though Transformers are extensively used for Natural Language Processing tasks, especially for machine translation, they lack an explicit memory to store key concepts of processed texts. This paper explores the properties of the content of symbolic working memory added to the Transformer model decoder. Such working memory enhances the quality of model predictions in machine translation task and works as a neural-symbolic representation of information that is important for the model to make correct translations. The study of memory content revealed that translated text keywords are stored in the working memory, pointing to the relevance of memory content to the processed text. Also, the diversity of tokens and parts of speech stored in memory correlates with the complexity of the corpora for machine translation task.